[feature][expression] refFunction port to core

[olivierdalang]

Description

This PR ports the refFunction plugin to core. RefFunctions are spatial operators expressions, allowing to make spatial queries on other layers from withing expressions, opening up a lot of really powerful workflows.

The hard work was done by @enricofer (the author of the plugin) and @m-kuhn , this is a rebase and polishing of their work.

The following new expression functions are implemented :

geometry_overlay_intersects(layer[,expression][,filter][,limit][,cache])
geometry_overlay_contains(layer[,expression][,filter][,limit][,cache])
geometry_overlay_crosses(layer[,expression][,filter][,limit][,cache])
geometry_overlay_equals(layer[,expression][,filter][,limit][,cache])
geometry_overlay_touches(layer[,expression][,filter][,limit][,cache])
geometry_overlay_disjoint(layer[,expression][,filter][,limit][,cache])
geometry_overlay_within(layer[,expression][,filter][,limit][,cache])
geometry_overlay_nearest(layer[,expression][,filter][,limit][,max_distance][,cache])

They all work in the same way : the spatial operator is evaluated against the other layer. If an expression is provided, it returns an array of results. If no expression is provided, it returns a boolean telling whether at least one feature matched.
Features can optionally be filtered by an expression, and optionally be limited to a certain count.

The geometry_overlay_nearest has an additional max_distance filter.

I'll leave this PR as a draft for some time as I still need to test it a little more and there's some more optimisation that can be done.

[qgis-bot]

@olivierdalang
This pull request has been tagged as requiring documentation.

A documentation ticket will be opened at https://github.com/qgis/QGIS-Documentation when this PR is merged.

Please update the description (not the comments) with helpful description and screenshot to help the work from documentors.
Also, any commit having [needs-doc] or [Needs Documentation] in will see its message pushed to the issue, so please be as verbose as you can.

Thank you!

[qgis-bot]

@olivierdalang
A documentation ticket has been opened at qgis/QGIS-Documentation#6102
It is your responsibility to visit this ticket and add as much detail as possible for the documentation team to correctly document this change.
Thank you!

[M-Rick]

I tried these new functions in QGIS 3.16, but they don't seem to work. I am needing to keep the refFunction plugin for now.
I tried to collect an attribute from a line layer to an intersecting point layer. At the beginning I used those brand new features, but I had no results at all. I thought my layers contained errors. But I then tried with refFunction and I got the expected results.
With the following refFunction expression it does work out of the box.
geomintersects('Line','Ref')
But when I switch to the new overlay functions it doesn't work.
I tried this expression instead of refFunction and it doesn't return me any results.
overlay_intersects('Line','Ref’)

Version de QGIS	3.16.0-Hannover	Révision du code	4af1cbf
Compilé avec Qt	5.14.2	Utilisant Qt	5.14.2
Compilé avec GDAL/OGR	3.1.2	Utilisé avec GDAL/OGR	3.1.2
Compilé avec GEOS	3.8.1-CAPI-1.13.3	Utilisé avec GEOS	3.8.1-CAPI-1.13.3
Compilé avec SQLite	3.31.1	Fonctionne avec SQLite	3.31.1
Version du client PostgreSQL	12.3	Version de SpatiaLite	4.3.0a
Version de QWT	6.1.4	Version de QScintilla2	2.11.4
Compilé avec PROJ	6.3.2	Fonctionne avec PROJ	Rel. 6.3.2, May 1st, 2020
Version de l'OS	macOS 10.16
Extensions Python actives	QuickOSM; processing_js; location_lab; refFunctions; qgis2web; qfieldsync; SgmExpressionFunctions; Mergin; processing; db_manager; MetaSearch

[olivierdalang]

@M-Rick Thanks for reporting your issue !

Could you please open a new issue instead, so we can better keep track of it ? Also don't forget to add data and exact steps to reproduce the issue. Ping me in the issue (@olivierdalang) so I'll get a notification and look into it.

[DelazJ]

overlay_intersects('Line','Ref’)

This is not the correct syntax of the new function. It should be overlay_intersects('Line', Ref) assuming Ref is the field you want to collect data from. Not a QGIS issue IMHO.
See examples at https://docs.qgis.org/3.16/en/docs/user_manual/working_with_vector/functions_list.html#overlay-intersects Also keep in mind that it returns an array so either your destination field allows array type or you should add a conversion using array_to_string

[gioman]

This is not the correct syntax of the new function. It should be overlay_intersects('Line', Ref) assuming Ref is the field you want to collect data from. Not a QGIS issue IMHO.

@DelazJ after a quick test it does not seems to work anyway (the equivalent from the plugin works).

[DelazJ]

@gioman could you share the expression you tested and the context (ref comment above about array output), please?

[pigreco]

from a quick test to me it works:

OSGeo4W win 10 - QGIS 3.16 Hannover

[gioman]

from a quick test to me it works:

@pigreco with double quotes preview works, but then values are not saved to the target table/column.

[gioman]

@gioman could you share the expression you tested and the context (ref comment above about array output), please?

@DelazJ here it goes

exp_sample.zip

[pigreco]

@gioman

@pigreco with double quotes preview works, but then values are not saved to the target table/column.

it is normal because they are arrays, if you want to save them you have to define the new field as text and use the expression

array_to_string(
 overlay_intersects( 'Com01012020_g_WGS84',"comune")
 )

or if you just want an array value

overlay_intersects( 'Com01012020_g_WGS84',"comune")[0] → Barrafranca
overlay_intersects( 'Com01012020_g_WGS84',"comune")[1] → Villarosa
overlay_intersects( 'Com01012020_g_WGS84',"comune")[-1] → Enna

[gioman]

it is normal because they are arrays, if you want to save them you have to define the new field as text and use the expression

@pigreco ok I see. I little less convenient than the original function.

[DelazJ]

@pigreco with double quotes preview works,

@gioman Even without quotes, it works. The expression here is a field and as such can take double-quotes or not

[gioman]

@gioman Even without quotes, it works

@DelazJ I may be missing something, but it does not for me, unless using also "array_to_string" in the expression.

[DelazJ]

@gioman I meant "no need of quotes to have the expression return results. Ie, overlay_intersects( 'Com01012020_g_WGS84', "comune") is the same as overlay_intersects( 'Com01012020_g_WGS84', comune)."
But yes, if you want to store in a string field, you will always need the array_to_string function.

[gioman]

But yes, if you want to store in a string field, you will always need the array_to_string function.

actually I tried to extract a number and put it into a integer field, and it also do not work (also because extracting from polygons with points, so only 1 value is extracted).

[m-kuhn]

Even if only 1 value is extracted it will be an array ([1]), it needs to be extracted with array_first(overlay_intersects( 'Com01012020_g_WGS84', comune)).

[gioman]

Even if only 1 value is extracted it will be an array ([1]), it needs to be extracted with array_first(overlay_intersects( 'Com01012020_g_WGS84', comune)).

ok. Should we add this in the function help?

[DelazJ]

I was also about to propose to add an example that returns comma separated list, as string. I think it's more common than the array_first case, and covers your use case here

[m-kuhn]

Improving the function help would be very nice 👍

[gioman]

Improving the function help would be very nice +1

@m-kuhn after thinking about I'm not sure how is best to do that, after all is normal that in any function that returns an array, an array function must be used too... Maybe adding a general note if is not already there (can't find one, but docs are very large nowadays!)? @DelazJ

[m-kuhn]

Maybe we could add it to the expression builder dialog, where we know it's an array (and we do some magic to convert it to a string already).

Hint: this expression returns an array, <a href="link/to/the/expression_array_doc">learn how to use arrays</a>

[gioman]

Maybe we could add it to the expression builder dialog, where we know it's an array (and we do some magic to convert it to a string already).

sounds good to me.

[DelazJ]

@gioman array functions are referenced at https://docs.qgis.org/testing/en/docs/user_manual/working_with_vector/functions_list.html#array-functions.
Adding the note as suggested by @m-kuhn sounds a good idea. In The docs, we currently display a url to reach array functions from the overlay functions (end of https://docs.qgis.org/testing/en/docs/user_manual/working_with_vector/functions_list.html#overlay-contains eg). We will probably remove them once this note is in place to avoid redundancy.
What I actually was suggesting in my previous comment is adding in the help panel an example of the type array_to_string( overlay_intersects( 'regions', "name")) that returns a comma separated list of values (and not array, as most of the current examples)