Features not filtered by mapnik style

[rouen-sk]

I am trying mapnik-vector-tile (1.2.0) with Mapnik 3.0.11. And there is one curious problem - in the resulting vector tile, there are ALL the features returned from datasource. I am not sure if I understand this correctly, but I thought they would be filtered the same way as in raster rendering - respecting maximum-scale-denominator from Layer and MaxScaleDenominator from Style from XML style loaded in map object. Now my tiles on zoom 9 are 20 MB in size, because they contain every housenumber and unimportant road in huge area...

I am trying to figure it out from sources - I am looking at create_geom_layer in vector_tile_processor.ipp and it seems to me, that it is iterating all features from layer and encoding them - I dont see any filtering anywhere, but since I am not c++ programmer, I am not really sure :) Thanks for help!

[pnorman]

The filtering happens in your datasource definition, just like with Mapnik stylesheets.

An example layer definition for a vector tile source (in Mapnik XML) is

<Layer name="landuse"
  buffer-size="8"
  srs="+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0.0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs +over">
    
    <Datasource>
       <Parameter name="dbname"><![CDATA[gis]]></Parameter>
       <Parameter name="extent"><![CDATA[-20037508.34,-20037508.34,20037508.34,20037508.34]]></Parameter>
       <Parameter name="geometry_field"><![CDATA[]]></Parameter>
       <Parameter name="geometry_table"><![CDATA[]]></Parameter>
       <Parameter name="host"><![CDATA[]]></Parameter>
       <Parameter name="key_field"><![CDATA[]]></Parameter>
       <Parameter name="key_field_as_attribute"><![CDATA[]]></Parameter>
       <Parameter name="max_size"><![CDATA[512]]></Parameter>
       <Parameter name="password"><![CDATA[]]></Parameter>
       <Parameter name="port"><![CDATA[]]></Parameter>
       <Parameter name="srid"><![CDATA[]]></Parameter>
       <Parameter name="table"><![CDATA[(
SELECT
    osm_id,
    way,
    CASE
      WHEN "natural" = 'wood' OR landuse IN ('wood', 'forest') THEN 'wood'
      WHEN leisure IN ('national_reserve', 'nature_reserve', 'golf_course') OR boundary = 'national_park' THEN 'park'
      WHEN landuse IN ('cemetery', 'industrial') THEN landuse
      WHEN aeroway IS NOT NULL THEN 'industrial'
      WHEN landuse = 'village_green' OR leisure IN ('park', 'playground') THEN 'park'
      WHEN amenity IN ('school', 'university') THEN 'school'
      WHEN amenity = 'hospital' THEN 'hospital'
      ELSE bail_out('Unexpected landuse row with osm_id=%s', osm_id::TEXT)
    END AS class,
    z_order,
    way_area
  FROM planet_osm_polygon
  WHERE
    (
        (
            (
              "natural" = 'wood' OR landuse IN ('wood', 'forest')
              OR leisure IN ('national_reserve', 'nature_reserve', 'golf_course')
              OR boundary = 'national_park'
            )
            AND z(!scale_denominator!) >= 7
        ) OR (
            (
              landuse IN ('cemetery', 'industrial', 'village_green')
              OR aeroway IS NOT NULL
              OR leisure IN ('park', 'playground')
              OR amenity IN ('school', 'university')
            )
            AND z(!scale_denominator!) >= 10
        ) OR (
            amenity = 'hospital'
            AND z(!scale_denominator!) >= 12
        )
    )
    AND way && !bbox!
    ORDER BY z_order, way_area DESC
) landuse
]]></Parameter>
       <Parameter name="type"><![CDATA[postgis]]></Parameter>
       <Parameter name="user"><![CDATA[]]></Parameter>
    </Datasource>
  </Layer>

[rouen-sk]

Yes, you can "optimize" the amount of data retrieved from datasource by using !bbox! and !scale_denominator! in query. But AFAIK this is just "performance optimization" - to prevent fetching too much data. But after that - filters in mapnik XML determine further, what will really go into the tile - by using maximum-scale-denominator on Layer elements (to prevent unnecessary queries entirely - if you know from style, that this layer wont go into this tile on this zoom), and MinScaleDenominator and MaxScaleDenominator on Rules of Styles - in combination with custom data columns filters on the rules. It would be very hard to write all these conditions into the SQL query.

So this is not implemented in mapnik-vector-tile at all?

[pnorman]

There are no filters in layers, but they can have Min/Max scales (normally expressed as zooms and processed to scales).

If you want different contents in a depending on zoom, you have to use !scale_denominator!.

[rouen-sk]

There are filters in layers - in Styles asociated with them. But OK, I can see the reasoning behind not filtering features by these - to allow more flexible client-side styling (for a cost of bigger tiles).
But not respecting even maximum-scale-denominator on Layer is very strange - if I have in my mapnik XML layer like this

<Layer name="housenum_label" srs="" maximum-scale-denominator="2500">
    <StyleName>housenum_label</StyleName>
  </Layer>

limiting house numbers to zoom >= 18, I certainly dont expect them in tiles for zoom 9. If mapnik-vector-tile takes mapnik's map with loaded stylesheet to work, why not respect this? For raster tiles with the same stylesheet, I dont have to write some condition with LIMIT 0 to datasource query for certain zooms - why should I have to "duplicate" this for vector maps, if the mechanism for this is already in place in form of maximum-scale-denominator on Layer elements?

[pnorman]

There are filters in layers - in Styles asociated with them

There are no styles associated with layers in vector tile generation.

[talaj]

According to the code, filtering by scale denominator should work on Layer level. Layers not passing scale denominator test should be rendered empty in the vector tile (which is probably going to change by #213).

These are relevant places in the code:
tile_layer::calc_query(), mapnik::layer::visible(), processor::update_tile()

But I haven't found any test covering this so it may not work.

Scale denominator filter on style level is not implemented but is simple to do. I'm also missing this feature and planning to implement it.

Universally, it would be best to filter by a range of scale denominators, allowing multiple zoom serialization into one tile.

[talaj]

The last two commits from this branch implement feature filtering by experssions in <Rule></Rule>.

There is new optional boolean parameter style_level_filter of vector_tile_impl::processor::create_tile() to control the filtering.

[springmeyer]

If mapnik-vector-tile takes mapnik's map with loaded stylesheet to work, why not respect this?

This is by design. The idea is that layer-only filtering (via the SQL and the layer level scale-denominator) would create vector tileset that could then be styled many different ways. So, all the XML I've used to render vector tiles do not have any <Style> objects at all. In the rare case XML used to create vector tiles do have <Style> (the tests probably have a few) then they are meant to be ignored. Again the idea is that you use mapnik-vector-tile to create vector tiles that might provide data to hundreds or 1000s of different styles with different filters.

However, all that said, if others desire this usecase - sounds like @talaj would find it useful - it would be fine to add it. I can imagine that it would be useful in the case that you have a single existing Mapnik XML that has been used for raster tile rendering and you want to port that to use vector tiles with the least effort. If this feature is added, it will need to be disabled by default I think - to avoid backwards incompatible behavior of filtering more data than intended (given current design).

[rouen-sk]

Yep, it certainly sounds awesome - just include all the data into vector tiles, and let all your dreams come true via client-side styling. But did you actually try this with any "rich" dataset (like OSM)? I did, and on certain zooms, I ended up with 50 MB+ for single tile. For any real-world production scenario, there is no way you can do without some filtering of features by combination of zoom and feature properties.

[talaj]

I haven't made pull request out of my style level feature filter implementation because I was not convinced about quality of that implementation. On the other hand it is backward compatible, it contains unit tests and I'm using it without any issues. I will take a look on it again.

[am2222]

@talaj Does your edited branch works now with mapnik 3.0.x? I have faced this problem too

[talaj]

@am2222 I think it's based on the code of 3.0.x series, but I cannot guarantee it's still compatible.

[am2222]

@talaj I edited 3.0.x with your branch but I still have a set of problems, It seems when I convert from vector tiles it does not apply style scale values, in fact it must apply doesnt it?
lets say I have an xml which includes layers I rendered it and it is as following image

later I converted that xml into vector tile. I built another xml which is exactly same as first xml but does not have any layers I rendered it

As you can see all the texts are rendered. But as styles are the same they must be just as the first image, why this happend? My style file is as following

osm_bright_style.zip

Do you think the problem is because of this issue?or I am looking for problem in wrong place?
I think this kind of rules are not applied correctly

<Rule>
  <MaxScaleDenominator>25000</MaxScaleDenominator>
  <MinScaleDenominator>12500</MinScaleDenominator>
  <TextSymbolizer size="16" character-spacing="6" wrap-width="400" text-transform="uppercase" fontset-name="fontset-0" placement="point" fill="#444444"><![CDATA[[name]]]></TextSymbolizer>
</Rule>

[talaj]

@am2222 Have you set the style_level_filter parameter of processor::create_tile() to true?

[am2222]

@talaj yes I have, It seems when I convert from vector tile into mapnik data source it does not apply rules in styles

[talaj]

It's strange that there is so many labels on the second image, because these features should already be filtered out during vector tile generation, if style level filters work as expected.

But it's also strange that the name of the city has bigger font on the second image. It could indicate that you are rendering these two images with different scale denominator.

If you are rendering the same way as in #277

    // create renderer and run it
    mapnik::agg_renderer<mapnik::image_rgba8> rend(map,im);
    rend.apply();

there may be some pitfall as node-mapnik is using a bit different approach for rendering from MVT to images. See here. It would nice to know the reason for this approach instead of rend.apply(). I don't know.

[am2222]

@talaj

It's strange that there is so many labels on the second image, because these features should already be filtered out during vector tile generation, if style level filters work as expected.

Why it does not filter based on styles?In fact if these titles exists in vector tiles they must not shown as we are filtering them by style when we want to convert from vector tile to mapnik layer.

But it's also strange that the name of the city has bigger font on the second image. It could indicate that you are rendering these two images with different scale denominator.

The names of small cities has a different style than the name of big cities, but I noticed that the big cities names are a bit bigger. I am using the same style for both normal layer and vector tile layer

So you mean I must apply rendering based on what you have shown in example?

mapnik::image_rgba8 & im_data = mapnik::util::get<mapnik::image_rgba8>(im); mapnik::agg_renderer<mapnik::image_rgba8> ren(map_in,m_req, closure->variables, im_data,closure->scale_factor); ren.start_map_processing(map_in); process_layers(ren,m_req,map_proj,layers,scale_denom,map_in.srs(),closure); ren.end_map_processing(map_in);

The problem is that I have no idea how to render using this approach.

[talaj]

In fact if these titles exists in vector tiles they must not shown as we are filtering them by style when we want to convert from vector tile to mapnik layer.

You are right. This really indicate scale denominator is not correct and denominator filters are not processed as expected.

You can try to set scale denominator to rend.apply().

It may be a bug and it would need some investigation, but I don't have much time currently.

[am2222]

@talaj thanks but is it possible to set scale denominator for each separate layer using this method?

[talaj]

The method renders all layers with given scale denominator.

[am2222]

@talaj I used the method you metioned, I passed scale denominator which I got from map object and nothing changed, It seems mapnik does not respect to rules in styles at all when we are using vector tiles is it possible?

[talaj]

I built another xml which is exactly same as first xml but does not have any layers

@am2222 Is it possible that those layers originally had properties minimum-scale-denominator or maximum-scale-denominator?

[am2222]

@talaj you mean in style or layer property? In styles yes but in ln layer property there is not any scale, I am very confused with the fact that when we convert from vector tile to mapnik map it does not respect styling

[talaj]

@am2222
I meant layer property. I do not think Mapnik does not respect styling. I think that the filter is processing different value of scale denominator than should. If there is such a rule:

<Rule>
  <MaxScaleDenominator>25000</MaxScaleDenominator>
  <MinScaleDenominator>12500</MinScaleDenominator>
  <TextSymbolizer size="16" character-spacing="6" wrap-width="400" text-transform="uppercase" fontset-name="fontset-0" placement="point" fill="#444444"><![CDATA[[name]]]></TextSymbolizer>
</Rule>

and this rule is passing even though it should not, change that interval [12500, 25000] to [10, 20], for example. This will proof if filtering works or not.

[am2222]

@talaj thanks very much. I will try to set those values and report the results. It seems strange to me as the style is the same as what is used in source data of vector tiles but it works with current intervals. thanks