Harvester geonode current

[ricardogsilva]

This PR adds the GeonodeUnifiedHarvesterWorker harvester worker for being able to harvest from remote GeoNode versions 3.2+

Implementation also introduces the GeonodeCurrentHarvester as a harvester class that is able to harvest from recent (3.2+) versions of GeoNode.

GeonodeUnifiedHarvesterWorker is then implemented as a thin layer on top that dispatches to GeonodeLegacyHarvester or GeonodeCurretnHArvester depending on the detected version of the remote GeoNode deployment. Version detection is done by checking the existence of the api/v2/datasets endpoint, as this endpoint has only been introduced on GeoNode after version 3.2 had been released.

This PR also includes a data migration to convert any pre-existing harvesters to use the new unified worker class.

[lgtm-com]

This pull request introduces 1 alert when merging a9e112a into 0bf76e1 - view on LGTM.com

new alerts:

• 1 for Unused import

[lgtm-com]

This pull request introduces 1 alert when merging 6d47621 into 0bf76e1 - view on LGTM.com

new alerts:

• 1 for Unused import

[ricardogsilva]

bump @afabiani ready for your review (and hopefully merge)

[afabiani]

thanks @ricardogsilva will review this ASAP

[afabiani]

@ricardogsilva I have been testing this PR so far, while it works quite well with the GeoNode Harvester, I'm afraid we broke something with the WMS one instead.

By trying to harvest a WMS dataset I notice that some info are missing, as an instance

"name": "",
"alternate": "",

The WMS link seems to be wrong too

"links": [
                {
                    "extension": "html",
                    "link_type": "OGC:WMS",
                    "name": "OGC WMS:  Service",
                    "mime": "text/html",
                    "url": "http://localhost:8080/geoserver/ows"
                }
            ]

I guess because of this
https://github.com/GeoNode/geonode/pull/8197/files#diff-bfbd859a5f6c4d9fc78fbbefd10f46d5fe34aaa115f3f8641a3e201ff737d955L255-L259

[afabiani]

Also this change here https://github.com/GeoNode/geonode/pull/8197/files#diff-2bfc7654963f56a9bfa2e71a0716d21ac1e6ef688fdccd9f6631006123cd6b59L73 causes an exception:

Traceback (most recent call last):
  File "/mnt/c/Work/geonode/geonode/services/serviceprocessors/base.py", line 177, in harvest_resource
    harvest_resources.apply_async(args=([resource_id, ], _h_session.pk))
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/app/task.py", line 569, in apply_async
    return self.apply(args, kwargs, task_id=task_id or uuid(),
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/app/task.py", line 790, in apply
    ret = tracer(task_id, args, kwargs, request)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/app/trace.py", line 467, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/app/trace.py", line 450, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/mnt/c/Work/geonode/geonode/harvesting/tasks.py", line 173, in harvest_resources
    harvesting_workflow.apply_async()
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/canvas.py", line 1408, in apply_async
    return self.apply(args, kwargs,
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/canvas.py", line 1427, in apply
    args=(tasks.apply(args, kwargs).get(propagate=propagate),),
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/canvas.py", line 1117, in apply
    return app.GroupResult(group_id, [
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/canvas.py", line 1118, in <listcomp>
    sig.apply(args=args, kwargs=kwargs, **options) for sig, _, _ in tasks
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/canvas.py", line 186, in apply
    return self.type.apply(args, kwargs, **options)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/app/task.py", line 790, in apply
    ret = tracer(task_id, args, kwargs, request)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/app/trace.py", line 467, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/app/trace.py", line 450, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/mnt/c/Work/geonode/geonode/harvesting/tasks.py", line 213, in _harvest_resource
    harvested_resource_info = worker.get_resource(harvestable_resource)
  File "/mnt/c/Work/geonode/geonode/harvesting/harvesters/wms.py", line 315, in get_resource
    distribution=resourcedescriptor.RecordDistribution(
TypeError: __init__() got an unexpected keyword argument 'legend_url'
__init__() got an unexpected keyword argument 'legend_url'
Traceback (most recent call last):
  File "/mnt/c/Work/geonode/geonode/services/serviceprocessors/base.py", line 177, in harvest_resource
    harvest_resources.apply_async(args=([resource_id, ], _h_session.pk))
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/app/task.py", line 569, in apply_async
    return self.apply(args, kwargs, task_id=task_id or uuid(),
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/app/task.py", line 790, in apply
    ret = tracer(task_id, args, kwargs, request)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/app/trace.py", line 467, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/app/trace.py", line 450, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/mnt/c/Work/geonode/geonode/harvesting/tasks.py", line 173, in harvest_resources
    harvesting_workflow.apply_async()
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/canvas.py", line 1408, in apply_async
    return self.apply(args, kwargs,
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/canvas.py", line 1427, in apply
    args=(tasks.apply(args, kwargs).get(propagate=propagate),),
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/canvas.py", line 1117, in apply
    return app.GroupResult(group_id, [
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/canvas.py", line 1118, in <listcomp>
    sig.apply(args=args, kwargs=kwargs, **options) for sig, _, _ in tasks
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/canvas.py", line 186, in apply
    return self.type.apply(args, kwargs, **options)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/app/task.py", line 790, in apply
    ret = tracer(task_id, args, kwargs, request)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/app/trace.py", line 467, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/app/trace.py", line 450, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/mnt/c/Work/geonode/geonode/harvesting/tasks.py", line 213, in _harvest_resource
    harvested_resource_info = worker.get_resource(harvestable_resource)
  File "/mnt/c/Work/geonode/geonode/harvesting/harvesters/wms.py", line 315, in get_resource
    distribution=resourcedescriptor.RecordDistribution(
TypeError: __init__() got an unexpected keyword argument 'legend_url'

[afabiani]

@ricardogsilva so, it looks that most of the issues above have been fixed on the arcgis PR, nevertheless we did lose something on WMS... it looks like the ows_url is not correctly populated. Taking a look at this.

[afabiani]

@ricardogsilva I commented this on #8229 too, IMHO we should implement def _get_resource_descriptor methods setting the name, alternate and ows_url for the WMSHarvester too.

[afabiani]

Superseded by #8229