Mostly done with Szczecin

In the last two months I spent quite a lot of time working on a data import into OpenStreetMap that has been a little different from most of the imports happening in the project.  I’m quite happy with the result but I’m also really happy to be done with in.

Most of Poland administrative institutions today are behind in the aspect of sharing information with others for common good, and information re-use (and in many other aspects).  This includes geospatial data and clinging on to it and always assuming there might be something secret in it, such as… I don’t know, really.  Obviously at some level there can be information protected by the privacy laws intermixed with geospatial data (parcel owners etc.), the other common case of spatial data that cannot be shared is the locations of protected species.  But most of the time none of this is concerned.

Then there are laws in place that mandate those institutions to take money for disclosing particular types of information, and only under rather restrictive licensing conditions (nothing that remotely resembles CC-By-SA).  But according to some people in the know, there are other laws in Poland that make most of this information classify as public information.  Theoretically those former laws take precedence over the public information related ones, but last I heard there’s some other legal complication, way above my level of understanding law (unfortunately), that in effect means there’s a conflict/inconsistency in that system.  What this means is that the institutions can assume either interpretation and they should be safe under the law.  But they will always assume the “closed” interpretation.

So looking at all the other places and “battles” that people in OpenStreetMap have with their local administrations, it seems that this is a common trace in Europe, with a slowly progressing change in the direction of openness.  But perhaps if you drew a little map of how “open” the institutions in different places are based on the number of data releases that happened, the area covered by Poland would mostly range between black and dark grey.  So it was a lucky strike that the city of Szczecin was happy to let us use all the information available through their GIS website, including for automatic processing.

Their website has bitmap layers with some pretty high quality data, and no vector data available directly.  This meant that it could be manually copied or some complex and rather hacky vectorisation could be attempted (obviously talking only about the data layers that were lacking in OpenStreetMap, not just everything — if you’ve done any OSMing, you know that some types of data are unlikely to be crowdsourced).  French mappers are trying to manually copy the national cadastre bitmap layer made available by their administration, but it seems like a very tedious work, which is unlikely to be finished soon.  So I tried to automate as much as possible of the vectorisation and I think in a big part it was a success.  Still quite a lot of manual work was left to be done.  Not a very interesting job, but not one that you can let some monkeys do for you either, because everything that could be automated has already been automated.  So I’m really happy that it’s mostly done now (import status page firefox only, and takes a while to load).  More details in Polish available in this post, but check out the mapsurfer screenshots and the tree density heat-map there.  Maybe I’ll have a lightning talk about it at the upcoming State Of the Map 2010.

We’ve contacted some other municipalities trying hard not to scare them with the modern terms like share-alike etc., and as expected they are reluctant, but there seem to be two more candidates right now, and the import process is better streamlined now, and it really could be quite straight forward if it was not for some little annoying properties of the way the Szczecin data was shared.

For the moment I have a long backlog of information surveyed in my own neighbourhood to put on the map.

Javascript to D-bus, can you hear me

(I guess I better post something here because it’s been a while.)  So over the last two weekends I made a simple oFono client in javascript, meaning that it’s browser-based or “web-based”.  To do that I needed a way to talk to D-bus over HTTP.  I’ll try to set up a demo instance of the client later but now I’ll just mention the HTTP to D-bus gateway.  Even though the whole thing is a hack, maybe the gateway will be useful to someone.  It’s also possible that there are already fifteen similar programs there, I’ve not really checked.

The idea is rather simple, it’s a 10 kilobyte python script called gateway.py and you can run it in some directory and it will run a primitive web server on a given port using python’s built-in http library, and will serve the files from the current directory and its subdirectories.  It also understands a couple of fake requests that enable web applications to talk to D-bus.  It connects to the system bus and relays messages to and from D-bus services using the following three types of (GET) requests:

• /path/to/object/Interface.Method(parameters) – This makes a regular D-bus call to a given method on a given interface of an object.  It’s synchronous and the HTTP response will contain the D-bus response written as JSON.  The D-bus types correspond very neatly to JSON types so the response is easy to use in javascript on the web.
• /path/to/object/Interface.Signal/subscribe/<n> – This subscribes the application to a given D-bus signal.  The applications identify themselves with a number (<n>), this can be any integer but it should be (reasonably) unique, for example it can be a random number generated when the application loads.
• /idle/<n> – This just waits for any signal that application <n> is interested in, to arrive.  The signal arguments are then sent to the client as JSON again, in the HTTP response.  This way the browser keeps a socket open to the server and signals are sent over it.

Here are some example calls to make it clearer, along with their return values:

• $ wget 'http://localhost:8000/modem01/VoiceCallManager.Dial("5555")' -O -
  '/modem01/voicecall01'

• $ wget 'http://localhost:8000/modem01/voicecall01/VoiceCall.GetProperties()' -O -
  { 'State': 'active', 'StartTime': '2010-06-01T02:16:34+0200', 'LineIdentification': '5555' }

• $ wget 'http://localhost:8000/modem01/Modem.PropertyChanged/subscribe/500' -O -
  null

It’s easy enough to make a little javascript class in your code to hide the http stuff away so you can make plain js calls and get the return values and have handlers called for the signals.  Also, obviously ajax doesn’t just sit waiting for a http response so your application doesn’t become synchronous in any way.

You’ll notice that the interface names are shortened to just the last part of the name.  Since the part before the dot is usually same as the service name, you can skip it and it’ll be added automatically.  So you can write either /org.ofono.ModemManager or just /ModemManager

To check out the repository do,

git clone http://openstreetmap.pl/balrog/webfono.git

It’s python 3 and uses the D-bus and glib bindings, so getting these dependencies installed may be a little challenge at this point.