Harvester arcgis

[ricardogsilva]

This PR implements a harvester for ESRI ArcGIS REST services catalogues. Supported services:

• MapService
• ImageService

These are the same services that were supported in the older GeoNode services implementation, which the harvesting is aiming to replace.

This uses the GeoNode arcrest package in order to interface with the remote ESRI services - this was already a dependency for GeoNode.

It introduces the harvesting.harvesters.arcigs.ArcgisHarvesterWorker as an additional harvester. This harvester is able to recognize remote URLs of these formats:

• {base}/rest/services - AKA the ArcGIS REST services catalog URL. The harvester worker is able to discover the relevant services offered by the catalog. Example: https://carto.nationalmap.gov/arcgis/rest/services

• {base}/rest/services/{service-name}/{service-type} - AKA the ArcGIS REST services service URL. Users can use this URL when they are specifically interested in a single service's resources. Example: https://carto.nationalmap.gov/arcgis/rest/services/transportation/MapServer

Available configuration options for the harvester worker:

• harvest_map_services - Whether the harvester should harvest resources from services of type MapServer. Defaults to True;
• harvest_image_services - Whether the harvester should harvest resources from services of type ImageServer. Defaults to True;
• resource_name_filter - Only harvest resources whose name contains the supplied string. Defaults to None, which will fetch all resources;
• service_names_filter - Only harvest resources from services whose name contains the supplied string. This is a list of service names. It defaults to None, which will look through all the services specified in the remote ArcGIS REST services catalog. Note that if the harvester's base_url already mentions a single service it has the same effect as setting this parameter to the name of the service. This means that these are equivalent:
  • base_url = "https://carto.nationalmap.gov/arcgis/rest/services" and service_names_filter = ["transportation"]
  • base_url = "https://carto.nationalmap.gov/arcgis/rest/services/transportation/MapServer"

NOTE: As of now, it seems the GeoNode mapstore frontend is not being able to correctly render the harvested services. This is something that this PR did not try to address, and was discussed offline with @afabiani - I imagine it will be done on a future iteration.

[ricardogsilva]

@afabiani please review at your earliest convenience - this is based on #8197, so please review that one first

[afabiani]

@ricardogsilva I have updated this and aligned with #8197
Except for the MapStore client issue (to be fixed on client side) so far so good for me.

If you could also fix the WMSHarvester accordingly by following the same approach you have done here, that would be awesome.

[ricardogsilva]

@afabiani

I've made the WMS fixes, including your patch (sent through another channel). Seems to be working fine now. Please review this and discard #8197, as this PR was based on that one - this includes both the GeoNode unified harvester, the ArcGIS one and the latest changes to the WMS one

[ricardogsilva]

@afabiani failing test in CI seems to be unrelated to this PR