OSMQuery: Easily use OSM Data in ArcGIS

Together with Riccardo Klinger (formerly at Esri Germany, now at Allego), I developed the tool OSMQuery, which allows users to easily retrieve and use free data from OpenStreetMap in the ArcGIS environment.

This text is a more detailed version of an article that Riccardo Klinger and I wrote for the arcAKTUELL magazine. It contains more pictures and links and has been further edited. The arcAKTUELL article itself is available in a reader app here (albeit in German) and can be downloaded as a PDF here.

Header image: Copyright OpenStreetMap contributors, cartography CC BY-SA OpenStreetMap.org

OSM: a very rich data source

OpenStreetMap, or OSM for short, is much more than just a base map and thus an alternative to providers and frameworks such as Google Maps. OSM is first and foremost a free project, that collects freely usable geodata, structures it and makes it available for use by everyone in a database as open data.

OSM thus represents a diverse “treasure trove” of data with an open license and, although of varying quality, in principle global coverage. Since the launch of OSM in 2004, several million members have collected data worldwide: Roads, paths, rivers, administrative boundaries and much more.

Until now, the easiest way to access OSM data from ArcGIS was the “ArcGIS Editor for OSM”. This toolbox supports the import as well as the conversion and editing of data from OSM. A targeted query of OSM – for example for points of interest such as bakeries, buildings or hiking trails – is only possible after importing an unfiltered and therefore usually (very) large regional or even global OSM dataset.

Outside of desktop GIS, the Overpass API offers a possibility to query OSM data in the browser using a special syntax and to download the results in GeoJSON format, for example. However, this way requires knowledge in the query language Overpass QL, the categories contained in OSM and last but not least in data formats like GeoJSON, GPX or KML which are not necessarily common for desktop GIS.

Collaboration thanks to social networks

The afore-mentioned situation inspired Riccardo Klinger to build OSMQuery as a simple-to-use Python-Toolbox for querying and downloading OSM data for ArcGIS. He quickly developed a minimum viable product. Then he published his code on GitHub and tweeted about it.

This made me aware of OSMQuery, and without Riccardo and I knowing each other better than “by sight” on Twitter before, we initiated a collaboration. Together, we were able to develop OSMQuery in an agile way across countries. This was made possible by git and GitHub, by filing issues, ideas and feature requests and by jointly (further) developing the functionality in Python. Over a period of only eight days, we created a ready-to-use Python toolbox for OSM data queries in our spare-time.

Implementation

Querying

To create the backend for OSMQuery, web calls from queries in the overpass portal overpass-turbo.eu were analyzed: This showed that POST requests are sent, each containing a JSON object with the query formulated by the user in the form of an overpass QL string – for example:

( node ["shop"="bakery"](49.378,8.638,49.435,8.746); );

for querying for bakeries in the area of Heidelberg.

Key-value tags in OSM

The OSM database is not a classically structured database with predefined data models for different object categories. Instead, OSM relies on a bottom-up defined and highly variable data structure (sometimes called ”folksonomy” from ”folk taxonomy”). The data structure describes objects with so-called tags, or more precisely, key-value tags, such as ”shop”=”bakery” in the example above. Tags consist of keys (in the example: ”shop”) and values (in the example: ”bakery”).

This structure renders the OSM database highly flexible, but also requires some prior knowledge of valid or commonly used keys and values for queries. For OSMQuery, a list of frequently occurring tags (key-value combinations) was created and stored as a configuration file. This configuration file is read by OSMQuery and is displayed to the users in dropdown menus for “OSM tag key” and “OSM tag value”.

In each query, users can specify one key and one or several tags or a wildcard.

Spatial filter

In addition to the thematic delimitation using OSM tags, OSMQuery allows the specification of spatial and temporal analysis windows: The spatial extent of the query can be specified, as in other ArcGIS tools, for example as the currently visible extent or the extent of a feature class. By integrating the OSM-based geocoding service Nominatim, toponyms such as “Zurich, Switzerland” can also be geocoded in the background and then used directly as area of interest for OSMQuery.

Note: The Overpass API has a timeout and a memory limit. It is therefore not advisable to formulate queries for frequently occurring objects for entire countries or for entire federal states. OSM extracts provided by various organizations are better suited for this use-case.

Temporal filter

Because the data in OSM is stored complete with historical information, OSMQuery can also query data for specific points in time defined by the user.

Resultats

The geodata extracted from OSM is inserted into the Table of Contents of ArcMap or ArcGIS Pro as a temporary feature layer (one per geometry type that yielded features for the query). From there, they can be saved into common data formats and further processed .

Transforming OSM data into a GIS data model

To prepare the results in the form of feature layers, OSMQuery must convert OSM’s flexible key-value structure into a more rigid GIS data model. To that end, OSMQuery transforms all keys found into feature attributes, to which the respective values are then mapped as attribute values. This becomes clearer with an example: If e.g., a user queries banks (“amenity”=”bank”) and ATMs (“amenity”=”atm”) from OSM, the resulting data set contains the union of all OSM keys as attributes. In general, it contains both the attribute “opening_times” (originating from the banks and filled only for them) and the attribute “currencies” (available currencies at an ATM; filled only for ATMs).

Application examples

Leisure activities and bicycle traffic in Zurich

With OSMQuery, you can get a quick overview of the location of points- (or lines- and areas-)-of-interest in any given city or region. In the example, OSMQuery was used to download data for four amenity categories of OSM – namely cafés, restaurants and rental stations or bicycle/bicycle parking spaces – and to visualize them on a map.

The lost island of Schmöckwitz

On 16 August 2018, the Berliner Zeitung reported that an investor in the harbour basin of Schmöckwitz in Berlin had illegally dredged an island. Unfortunately, there was no up-to-date satellite imagery of the situation at that time. However, the community had already updated OSM. With OSMQuery, the relevant data could be easily downloaded and processed in order to show the situation before and after the incident. Thus, a graphical comparison and a visualization of the change became possible. The analysis based on this data showed that approximately 800m² of land had been removed during the period in question.