[Kde-hardware-devel] UPnP Kioslave and Amarok Collection notes

[Bart Cerneels]

On Sat, May 15, 2010 at 04:38, Nikhil Marathe <nsm.nikhil at gmail.com> wrote:
> Hi all,
>
> (This is CCed to hardware-devel since they are working on Solid
> integration which will eventually be used, and also because of
> Friedrich's Cagibi. Tuomo is the author of HUpnp, a Qt UPnP library.)
>
> This is a mind dump of what I have done/plan to do, implementation
> overview and so on. Comments are requested.
>
> Also I have a few questions about the generic Amarok Collection,
> written below, which I hope someone ( jefferai? ) can answer.
>
> The kioslave
> ==========
>
> MediaServer kioslave: http://gitorious.org/kio-upnp-ms/mainline
>
> 1) The slave uses the upnp-ms:// scheme.
> 2) The slave *expects* a UDN. upnp-ms:// is not valid. upnp-ms://<udn> IS.
> 3) It uses Cagibi to retrieve device details. This would include
> checking if it is indeed a MediaServer and getting the IP of the
> device.
> 4) [NOT DONE YET] UPnP supports unicast. HUpnp will implement it soon.
> The IP from (3) is used to get the service list. From this point
>    onwards HUpnp is used for all interaction with the device.
> 5) The Kioslave caches the device interface obtained from UPnP, so
> that it doesn't do discovery until the <udn> changes.
>
> The kioslave's features will mostly depend on what the underlying
> ContentDirectory (CDS) provides. Not all of them allow writing, even
> renaming etc. That said, retrieving files and browsing is definitely
> supported.
>
> upnp 'container' maps to our directories
> upnp 'item' maps to files
>
> The slave is based on a single directory. At any point it is only
> 'aware' of that node in the heirarchy. The slave keeps a reverse cache
> from item name -> ObjectID so that
> upnp-ms://<udn>/this/awesome/song.mp3 maps to the right object id and
> the slave knows what to do. Whenever the remote resource presents a
> URL from where it can be fetched, it is usually HTTP or some common
> protocol, the UDS_TARGET_URL property will be set, so that the
> appropriate kioslave automatically does the transfer. For uploads or
> copying to the server, the CreateObject+ImportResource actions are
> used if supported, otherwise it is not supported.
>
> If the server supports UpdateObject, then rename's and meta-data
> changes are allowed. Obviously DeleteResource decides whether deletion
> is possible.

I wouldn't even bother with Create, Import, Update, etc during this
GSoC. It's virtually unimplemented in any server. Where it is you
would need to be very strict w.r.t. the DLNA requirements. All in all,
it's not worth the effort ATM.

>
> The SystemUpdateID and ContainerUpdateIDs state variables can be used
> to watch out for changes in the current 'directory'.
>
> Q: How does a slave notify an application of these changes?
>  I don't think it is possible. the local changes are noticed by
> gamin/fam and the slave seems to be refreshed. So I think refreshing
> seems to be the only way to notify the user, in which case it may not
> be necessary to watch out for state variable changes.

There has to be a way to do this. Just try the nepomuk kio-slave and
notice how it lazy loads query results. That means the slave updates
right?

>
> About searching and sorting
> ------------------------------------
> The kioslave API seems to do these jobs, atleast in Dolphin/Konqueror.
> Basically you just set various UDS_* properties for each entry and the
> filter bar and date/name sorting just work. So the search query API
> does not seem to be required for the kioslave. The UDS_* filling will
> be done by parsing the DIDL output received from the MediaServer.

That is great to hear. Does this mean the nepomuk search box in
dolphin can be reused by the upnp slave?
(hint: kfm-devel in CC)

>
>
> The Amarok Collection
> ==============
>
> Currently puts in a new Collection entry with name/udn for each
> MediaServer: http://gitorious.org/~nikhilm/amarok/nikhilms-amarok/commits/upnp-collection
>
> Since this is more concerned about the CDS as a *whole* since the
> directory structure and the Meta-data heirarchies need not match. So
> my current plan is to implement the collection *directly in HUpnp*
> rather than use the kioslave, where each of the QueryMaker 'segments'
> like beginAnd() can be encoded in a buffer and then passed on to the
> CDS when run() is called.

I'm not a fan of this. That would mean having 2 implementations, each
with own bugs and state of maintenance.
If the server supports search, you can just use the search UDS_*
properties starting from from root (ObjectId=0). If no search you'll
need to build the MemoryQuerymaker maps by directory walking, which
will be easier via KIO. Even more benifit for KIO: it's out-of-process
(at least on desktop) and so will never block the GUI.

Can you explain me your reasons for implementing the CDS paring in
Amarok directly?

>
> To that end I was thinking of factoring out the DIDL parser into a
> reusable SAX parser like thing, since both components will have to
> parse the same output.

QXmlStreamReader is your friend. Allows you to parse directly to the
internal objects and even allows incremental parsing, so you don't
need to way for the entire XML to be downloaded. The result of an
unlimited query can be huge, so absolutely DON'T use a DOM tree.
An example of an incremental parser can be found in PodcastReader.cpp
(Amarok). Start with a non-incremental first.

>
> My other idea ( and I may be totally wrong here ) is to use the SQL
> database used by SQL collection to catalog the UPnP devices too. Since
> the UDN is unique that can act as an identifier. When a device is not
> present, that entire table or rows in the database can be ignored. The
> Collection can fullScan() the UPnP media server once and insert
> entries into the database. After that only when the SystemUpdateID
> changes, the incremental scan can update the database as necessary. In
> the situation that Amarok is shutdown, then the CDS contents are
> significantly changed and later Amarok is running, it might require a
> full scan.

Yup, totally wrong. How huge is the database going to get when you
find 20 server with several gigabytes of music (let's say a college
dormatory). Not even beginning to think about keeping database and
server in sync.

>
> The other way is to use a MemoryQueryMaker and MemoryCollection. The
> MemoryQueryMaker is also more important in my timeline, since all
> CDS's do not support the full UPnP Search() API.
> That said, does a MemoryCollection hold the entire contents in memory
> like a mmap(), or does it only hold the Meta-data in memory?
> And how do you get persistence in such a situation?
> Am I correct in saying that a Collection's job is to gather up the
> data from the underlying system, and it is the QueryMaker's job to
> actually do something useful with this data?

You don't need persistence since the tracks are not available when the
server is not available.
The querymaker is an interface to the tracks in the collection. Since
the CollectionBrowser only uses QM's it's required for all
Collections. (A Collection  without a (full) querymaker can be used as
a Service)

You'll just need to make your directory walker very fast (i.e.
incremental) with non-blocking behavior (the querymaker is usable
while scanning). And of course, this is only an issue for
non-searchable servers.

Bart