About Layer tree embedded widgets and have your WM(T)S always crispy sharp

Around 2014/2015 Martin updated the whole Legend / Layermanager code in QGIS. He wrote some nice blogs about this new “Layer Tree API”: Part 1, Part 2 and Part 3 so people would better understand how “to talk PyQGIS to the Legend”.
In 2016 Martin merged some code on top of this, which would make it possible to create so called ‘Layer Tree embedded widgets’.
In the image below you see an example of this: a little opacity slider which can be used there to change the opacity of the Layer to which it is connected visible in the Layer Manager.
Such embedded widgets are inserted when you move then in the Layer Properties/Legend tab from ‘Available widgets’ to ‘Used widgets’, ONLY for the layer you are viewing the properties.
Read More


This WCS investigation started off because my PDOKServicesPlugin was not working for the PDOK WCS services anymore. I wanted to see all fired network requests, so am creating a future plugin for that: https://github.com/rduivenvoorde/qgisnetworklogger. But I hit all kind of issues when trying out the service(s). This is some write up of my findings and hopefully is interesting enough for some people interested in WCS’s….
WCS is the WebCoverageService, an OGC api to serve Coverage data. In OGC terms a Coverage is “raster-data”: like geo tiff’s, ecw’s being rasterized, DEM’s (Digital Elevation Models) etc.
Our national opendata service: ‘PDOK’ has an WCS endpoint to the AHN (our national DEM available at max 50cm, though governmental agencies can go up to 10cm resolution).
One of the drawbacks of WCS is that there are already so many different versions: 1.0.0, 1.0.1, 1.1.2, 2.0.0, 2.1.0 etc etc
QGIS WCS provider supports the versions: 1.0.0, 1.1.0, 1.1.2.
WCS is part of the OWS-family of api’s: the ‘OGC Web Services’.
To determine the capabilities of a service you use (just like every other OWS service) a ‘GetCapabilities’-request, like:
This shows you the general information of the service, AND which datasets are available.
To fetch more information about one dataset, we use the ‘DescribeCoverage’-request, like:
(in newer versions of the WCS you also have to add the CoverageId to the request)
After you received this information (well, actually nothing more then some crs, layernames and dimension information, I’ve never understood why you would not add data statistic in this also…) you are ready to actually GET the data with a so called ‘GetCoverage’-request. In this request you set the desired data type.
And example to retrieve a part of our 5m National DEM around Haarlem as a tiff raster image:
Save this to disk and you can automagically load it in QGIS, because this is besides a true tiff-image also a geotiff, meaning it has some metadata in it keeping information about the crs (Coordinate Reference System/Projection) and it’s scale and place on earth:
As you can see, for example by using the identify tool, in this image there is just one ‘band’: Band 1. Where I clicked the data shows 43, note that this is NOT the height there, but the RGB value of the pixel in the position, going from 0 to 255. So this is NOT the actual data?
You can also load the following url in your browser to ask the service to provide a png image:
Saving THAT one to disk and loading it in QGIS shows you that it will NOT be placed nicely in the right position of the map. That is because a png image does NOT contain the mentioned metadata (you can provide it as a separate worldfile next to it though).
When you use QGIS to look into the png image, you will find 3 bands: 1,2 and 3 (actually RGB) all holding the same values.
So the service encoded the (actual) raster data into some kind of image where the values are spread over the range 0 till 255…. It is still possible to have some general view of the height variations in this area (actually not that much, besides the dunes in the west 🙂 ) by providing some styling to the png or tiff:
But what are the real data values of the cells of the raster?
To find that you can (either use the capabilities url and load the service via QGIS, OR use the following request, which should get exact the same data but encoded as 32 bit floats in a tiff (GEOTIFF_FLOAT32):
NOW you receive actual data, data in every cell has it’s (modelled) height in meters, ranging from -4.485 meter below seelevel (in the lower right corner) upto 55.5212 meter: I think some ‘high’ dunes in the west or maybe some tower 🙂
Mmm, not sure what is going on here, because measuring the cellen, they are not 0.5 meter x 0.5 meter ???
Let’s try the actual service then: use the GetCapabilities service url from above, create a WCS connection and request the ahn3_05m_dsm (raw data) in the format FLOAT32. This takes some time but NOW you will have 0.5×0.5m cells (though it takes some time to retrieve the data….
Open the Layer props, and from the ‘Render type’ dropdown just select ‘Hillshade’. Zoomed in at Haarlem, you will see:
Nice and easy he? The ‘mountain’ in the middle is an old ‘landfill’, now ‘recreation area’, 17 meters high.
What are the issues I hit with WCS?
WCS as Standard
– There are just too many variables and options in the ‘standard’. Because of that both WCS client-implementations AND WCS server-implementations often do not play well together. In my case the server change at PDOK from Geoserver -> Mapserver made the Formats totally different. NOT so much a problem if you have a highly configurable client like QGIS. But if you have a more simple client (as my plugin, or some OpenLayers or Leaflet plugin) it would be good if those formats stay more or like the same.
– All the capability documents are (very) different in the different versions
– So: please OGC: Keep It Simple and backward compatible please!
WCS and Server and Services
– the idea is that a WCS is able to server actual data. Why then are most server implementations also serving those image/rgb format (why a PNG with values of 0 – 255 if you want to know the height???). IF you want something like that, use a WMS to serve the data. So PDOK, maybe just only server the FLOAT32 version? As the height/data is in meters so having integers makes the height-resolution rather low.
– why do you not get back the real data if you request an area in float32 in a browser?
– Not sure how realistic it is to server a DEM from a small country like NL via WCS. As you can imagine the actual data is HUGE, so I was thinking: why did we not come up with WCTC (a Tiled version of the WCS) ???
WCS as client
– looking at all the requests QGIS is firing to the server, it seems like it tries to find out some stats first (for the whole dataset), killing the WCS or at least timing out pretty often. Not sure how valid the stats are from the whole extent (but created from the pyramid by gdal?). Anyway: in my view QGIS fires way to much requests to the server. In case of a tiny tiff serving as wcs that is not a problem, but the AHN/DEM of NL is just to big for that. WHY is that raster-statistics info not available in the GetCapabilities or DescribeCoverage response?
PDOK serving WCS’s
– why is PDOK (or it’s data-providers) using different values for the NODATA value for every different service. Can we try to use some standard for that (at least between those services of the same kin)? Like -9999 (as we know nowhere in NL we go that low…).
– should the AHN not being served as float32 only (and a WMS next to it if you want images)?
Serving the AHN as WCS is doable, but you really have to be patient, raise your network timeout value, hope that PDOK internally does not time out, find out about the data first by a lot of trial and error and THEN you can have fun.

QGIS en het KLIC

Just released a new version of the KLIC viewer plugin for QGIS.
This was neccesary because the format of the information received has changed a lot! Before it only included the information on pipelines in raster format. Now the information on pipelines delivered in XML can include information in vector format .
The KLIC viewer plugin shows the result that you receive when you do a call before you dig request in the Netherlands at the National Land Registration and Mapping office named Kadaster.
klicbericht_in_vectorLoaded in QGIS the result look like this: A map with network information of different types of cables and pipes.
Before you plan to dig mechanically, you have to register online at the kadaster where you want to dig. Kadaster collects all information from all net owners that have pipes or cables in the digging area. Next you will receive an email with the result where you can download you Zip-file that contains information like shown above.
klicviewer_enThe plugin looks pretty simple, some buttons to load the Zip-file and a list that can be used to open included documents as wel.
klicviewer_themes_enThe plugin also has a tab with themes, you can use it to quickly turn on or of many layers with a particular theme.
All best wishes for 2019 and have lot of  fun with QGIS!
Regards, Diethard

Locatieserver: hectometerpaaltjes en percelen

For those interested in dutch OpenData: our national OpenData service PDOK has a Solr based geocoding service available.
Since this week it is possible to search for ‘parcel’ codes to find cadastral parcels, but also to search on so called ‘hectometer-paaltjes’: the little green number signs you see when you drive the dutch highways.
So in QGIS with the ‘PDOK services plugin’ you can use different filters to search for such objects.
Below a screenie with both a search and one of the reference maps from PDOK:
Happy QGISsing 🙂

(English) AmsterdamTimeMachine

Via twitter: AmsterdamTimeMachine.nl.
Jan Hartman’s and WebMappers hard work of georeferencing a set of Old Amsterdam maps: http://amsterdamtimemachine.nl.
6 XYZ-Map services with maps old as 1625 to have a look into history, off course also to be loaded in QGIS 🙂
Wanna see ‘the red light district’ in 1625?
Or see Dutch 17th century glory on a map (well, 135 degrees rotated 🙂 ):
Or see that there was also a ‘Waag’ (“weigh house”) on Dam Square?
Go grab the xyz-urls from http://amsterdamtimemachine.nl or download this zip containing a QGIS project file.
The zip contains a .qgs file which you can load as project with both QGIS 2.18 and QGIS 3.x NOW. But also a file ‘amsterdamtimemachine.nl.xml’ which you can load as a set of XYZ-services IF you have a QGIS newer then 3.1 (so you either run a nighly, or wait for 3.2 to come out)..
Happy QGISsing…

Maak een QgsLocator (Plugin) met PyQGIS

What is a Locator (plugin)
Some months ago, Nyall Dawson silently dropped a nice widget into the lower left corner of your QGIS screen:
People familiar with QtCreator (the Qt-development environment) should recognize it as a QtCreator Locator look-a-like: a way to (very) quickly search in your project for words, classes, bookmarks, help topics, files etc etc.. It is actually a multipurpose tool to ‘quickly’ search for something, or ‘start’ something. See the original Qt documentation.
Nyall had the brilliant idea to add something like that to QGIS too… and he did. In the code it is called a QgsLocator and it can load/register all kind of so called ‘QgsLocatorFilters’.
Out of the box by starting to type in the search field, you search for: Actions, Processing Algorithms, Spatial Bookmarks, Active Layer Features, Project Layers and Project Layouts.
Just start to type ‘bel…’ and you will see there are a lot of buff(er) related Processing Algorithms available.
Then by clicking on one of the algorithms, you actually start it in processing, super fast, super cool.
Or if you have a world map loaded like in the image and click on ‘Belgium’ you zoom to that ‘feature’.
BUT the coolest thing is, that as a (python) developer you can ‘easily’ implement such a QgsLocatorFilter yourself.
The thing that came up with me first is: ‘Wow, what about using this for a fast searching geocoder…’
So, let’s try…
How to create one
The crux is, to implement/code a so called ‘QgsLocatorFilter’ (see the QgsLocatorFilter api docs, the PyQGIS api or the cpp headerfile).
YOUR implementation of this piece of code is actually the work horse of your Locator Filter. It will take the user input, do something with it (in our case: throw it to an Online Geocoder Api), get back the results, create ‘QgsLocatorResult’s from that, which then will be shown to the user AND describe what happens when a user double clicks on the result.
Off course you need some ‘glue’ to put it in QGIS after this:
– a plugin which you can put in the QGIS plugin repository so a user can download your filter
– the ‘registering’ of your filter to QGIS so it is ‘picked up’ and shown in the Locator search widget
(see the nominatim locator plugin code)
Some geocoders (notably the Google one), need a personal api key to work.
When you left click on the little magnifier-icon a menu will appear in which you can select ‘Configure’
That will bring you to the Locator tab in the Options dialog:
There you can choose to always enable a Locator Filter, or fully disable it.
As you can see in the right, there is also an (now inactive) configure button. If your locator returns True as a result of calling hasConfigWidget(), then that button becomes active and you can for example implement your own Geocoder configuration window.
An example geocoder: Nominatim_Locator_Plugin
Nominatim (from the Latin, ‘by name’) is “a tool to search OSM data by name and address and to generate synthetic addresses of OSM points (reverse geocoding)”. You can read the OSM wiki or the OSM api information.
I have created a plugin called the ‘Nominatim_Locator_Plugin’ which implements this QgsLocatorFilter interface. Install it in QGIS by enabling the experimental plugins, search for ‘Nominatim’ and install the ‘Nominatim Locator Filter’ plugin. Or … have a look into the python code.
As you can see in the ‘fetchResults’ we wait until we have at least 2 characters, and then we start searching.
One special case with the Nominatim service is that it is a free service, but you are NOT allowed to use it as a ‘suggest’ service. That is: only search a full place name. To handle this, I added the gesture that you have to finish your place- or address-term with a ‘space’, and THEN the request is fired.
The cool thing with Nominatim and OpenStreetMap is that it handles all kind of languages. If you type the dutch word of Statue of Liberty: ‘vrijheidsbeeld’, it will just show up:
Some technical details
Which methods you have to define you can find in qgslocatorfilter.h.
One thing to be aware of is that the searching/retrieving/showing of the results is all done in multiple threads! I thought to be clever and tried to use asynchronous GET requests to the services, but ended up fighting ‘thread troubles’. So my tip: just use synchronous HTTP for your requests.
Which brings me to which library to use for the HTTP traffic. In QGIS 2 plugins you see people use the Requests-module pretty often, or plain httplib2 or implement their own NetworkAccessManagers.
Disadvantage of using external modules or home-brew solutions is that often QGIS-configurations like Proxy Settings or Network Timeout values are ignored…
For my experiments I used this module: networkaccessmanager.py.
The nice thing is that it tries to be a thin Python wrapper around QGIS’s own QgsNetworkAccessManager (which in this case is actually a thin wrapper around Qt’s QnetworkAccessManager).
It uses the Proxy settings that a User configured and if needed it can use QGIS’s authorisation module so the HTTP traffix is authenticated against the users own systems.
This boundless solution is a great one, but it would be even cooler if the (rather naive) QgsNetworkAccessManager Cpp implementation could be extended to be more Python/User friendly. So it is easier to handle redirects, timouts, headers etc. And like in the boundless module you could choose between synchronous (blocking) or asynchronous (non-blocking) calls.
I created a QEP (QGIS Enhancement Proposal) for it, and hope others are interested and chime in.
Future Ideas
Off course FOSS is never ‘finished’, as in the good sense of being ‘never finished’ 🙂
New ideas always pop up when you are coding:
– use a Nominatim configuration screen, to for example only search for certain osm tags (what about: “find all pubs in current mapcanvas view”, best to be done with a map of your current surroundings 🙂 )
– use the accept-language option of Nominatim (use current QGIS language) in the query, so you can use ‘Эйфелева башня’ or ‘eiffeltoren’ and get your results back in your preferred language.
– a Locator for Google Maps api searching. You will need a key for that one, so create a configuration Window for it too (actually: done 🙂 )
– the PDOK locatieserver interface, our national geocoder service (actually: done 🙂 )
– YOUR national geocoder service
So, I hope we get you interested in Locators and will see your plugins appear at plugins.qgis.org
Ah, and some last tips for Python QGIS / PyQGIS programmers:
Latest Python API docs (thanks Denis): https://qgis.org/pyqgis/master/
https://qgis.org/api/api_break.html (all QGIS2.x – QGIS3.x api breaks and fixes)