Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor thumbnail creation service #7060

Closed
giohappy opened this issue Mar 10, 2021 · 0 comments
Closed

Refactor thumbnail creation service #7060

giohappy opened this issue Mar 10, 2021 · 0 comments
Assignees
Labels
backport 3.2.x PR should be backported to target version enhancement performance Issues regarding server Performance and Speed
Milestone

Comments

@giohappy
Copy link
Contributor

The current creation of thumbnails for layers and maps is quite a mess, and the following points are especially problematic:

  • there are multiple implementations of thumbnail rendering of layers. At least the following two are implemented:
  • both layer_thumbnail and map_thumbnail use the pritng Geoserver module for the actaul rendering. This module is stale and quite a black box. Probably this code reflects the module that we're bundling inside the Geoserver build for GeoNode
  • the creation of the thumbnail for a map apparently takes different paths in case the thumbnail is generated from the Django template pages or from the "Save" action inside the GeoNode MapStore client. Different thumbnails (different zoom level and positioning) are generated for the two cases

This module will be in charge of:

  • provide a unified interface for the GeoNode components that trigger the creation of a thumbnail:
    • post layer creation tasks
    • map saving
    • metadata wizard thumbnail widget
    • set thumbnail action from layer and map detail page
  • receiving a configuration for the map to be rendered (bbox, dimensions, etc.)
  • delegating to specialized classes the retrieval of the layers images (WMS, XYZ, etc.)
  • composing the final rendered image
@giohappy giohappy added enhancement backport 3.x backport 3.2.x PR should be backported to target version labels Mar 10, 2021
@afabiani afabiani added this to the 3.2 milestone Mar 12, 2021
@afabiani afabiani added the performance Issues regarding server Performance and Speed label Mar 12, 2021
afabiani pushed a commit that referenced this issue Mar 17, 2021
* Bump urllib3 from 1.26.2 to 1.26.3 (#6908)

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.2 to 1.26.3.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/1.26.3/CHANGES.rst)
- [Commits](urllib3/urllib3@1.26.2...1.26.3)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Toni <toni.schoenbuchner@csgis.de>

* [Fixes #6880] Circle CI upload tests fail irregulary (#6881)

* [Fixes #6880] Circle CI upload tests fail irregulary

* CircleCI test fix: sometimes expires due to upload timeout in the test environment

* - Avoid infinite loop on upload testing

* Revert "CircleCI test fix: sometimes expires due to upload timeout in the test environment"

This reverts commit 66139fd.

Co-authored-by: Alessio Fabiani <alessio.fabiani@geo-solutions.it>
Co-authored-by: afabiani <alessio.fabiani@gmail.com>

* [Fixes #6914] Remove "add to basket" tool for documents and maps (#6915)

* Added malnajdi as contributor

* [Fixes #6910] meaningful filename for document download (#6911)

* get meaningful document filenames on download

* - Strip extension from document title before slugify it (e.g.: image.jpg instead of imagejpg.jpg)

Co-authored-by: afabiani <alessio.fabiani@gmail.com>
Co-authored-by: Alessio Fabiani <alessio.fabiani@geo-solutions.it>

* - CircleCI Upload Tests: trying to reduce more the risk of infinite loop on "wait_for_progress"

* [Fixes #6916] gsimporter.api.NotFound caused by missing trailing slash at the end of GEOSERVER_LOCATION (#6913)

* [Fixes #6916] gsimporter.api.NotFound caused by missing trailing slash at the end of GEOSERVER_LOCATION

* [Fixes #6916] unit test for GEOSERVER_LOCATION

* Bump django-cors-headers from 3.6.0 to 3.7.0 (#6901)

Bumps [django-cors-headers](https://github.com/adamchainz/django-cors-headers) from 3.6.0 to 3.7.0.
- [Release notes](https://github.com/adamchainz/django-cors-headers/releases)
- [Changelog](https://github.com/adamchainz/django-cors-headers/blob/master/HISTORY.rst)
- [Commits](adamchainz/django-cors-headers@3.6.0...3.7.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump amqp from 5.0.3 to 5.0.5 (#6905)

Bumps [amqp](https://github.com/celery/py-amqp) from 5.0.3 to 5.0.5.
- [Release notes](https://github.com/celery/py-amqp/releases)
- [Changelog](https://github.com/celery/py-amqp/blob/master/Changelog)
- [Commits](celery/py-amqp@v5.0.3...v5.0.5)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump pip from 21.0 to 21.0.1 (#6900)

Bumps [pip](https://github.com/pypa/pip) from 21.0 to 21.0.1.
- [Release notes](https://github.com/pypa/pip/releases)
- [Changelog](https://github.com/pypa/pip/blob/master/NEWS.rst)
- [Commits](pypa/pip@21.0...21.0.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump coverage from 5.3.1 to 5.4 (#6903)

Bumps [coverage](https://github.com/nedbat/coveragepy) from 5.3.1 to 5.4.
- [Release notes](https://github.com/nedbat/coveragepy/releases)
- [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst)
- [Commits](nedbat/coveragepy@coverage-5.3.1...coverage-5.4)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump pytest from 6.2.1 to 6.2.2 (#6907)

Bumps [pytest](https://github.com/pytest-dev/pytest) from 6.2.1 to 6.2.2.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst)
- [Commits](pytest-dev/pytest@6.2.1...6.2.2)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump djangorestframework-gis from 0.16 to 0.17 (#6902)

Bumps [djangorestframework-gis](https://github.com/openwisp/django-rest-framework-gis) from 0.16 to 0.17.
- [Release notes](https://github.com/openwisp/django-rest-framework-gis/releases)
- [Changelog](https://github.com/openwisp/django-rest-framework-gis/blob/master/CHANGES.rst)
- [Commits](openwisp/django-rest-framework-gis@v0.16.0...v0.17.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* - Algin setup.cfg to requirements.txt

* [Fixes #6922][REST API v2] Expose the curated thumbnail URL if it has… (#6923)

* [Fixes #6922][REST API v2] Expose the curated thumbnail URL if it has been uploaded

* - Add REST APIs test suite to CircleCI

* [Fixes #6918] Removal of QGIS support (#6919)

* [Cleanup and Refactor] Remove QGIS server backend dependencies

* [Cleanup and Refactor] Remove QGIS server backend dependencies

* - Fix LGTM issues

* allow Basic authenticated requests in LOCKDOWN mode

* fix to avoid circular import

* flake8 check fix

* added tests

* first draft

* remove service integration

* bbox structure fix

* layers and maps endpoints integration

* formatting

* flake formatting

* fix for longitude exceeding world width

* description

* fix for longitude exceeding world width 2

* fixes requested by PR review

* exceeding latitude fix

* immutability fix

* thumbs background configs

* poc of refactored wizard layer thumbnail

* missing EPSG prefix

* wizard map thumbnail

* correct layers order in a map

* security issue fix

* fix for small thumbnails

* comments to wizard wiget refactoring

* preserve thumbnail's ratio

* preserve thumbnail's ratio fix

* fix thumbnail URL for layer's wizard panels

* typo

* bbox method fix

* tests ongoing

* tests ongoing

* preparing the thumbnail.js for the refactoring in MapStore

* tests and formatting

* remove legacy thumbnail generation

* test fix

* minimal documentation of background generator options

* remove outdated tests

* add base.tests.ThumbnailTests cleanup

* formatting

* failing CI tests fix

* - Add thumbs tests

* - Fix test cases

* - Added few more consistency checks in order to make sure we catch 'zero-area' cases

* - Fix test cases

* - Fix test cases

* - Fix test cases

* fixes to thumbnail set frm detail pages

* - Fix test cases

* - Fix test cases

* - Fix test cases

* - Test fixes

* - Test fixes

* - Test fixes

Co-authored-by: allyoucanmap <bovio.stefano@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Toni <toni.schoenbuchner@csgis.de>
Co-authored-by: Alessio Fabiani <alessio.fabiani@geo-solutions.it>
Co-authored-by: afabiani <alessio.fabiani@gmail.com>
Co-authored-by: Florian Hoedt <gannebamm@gmail.com>
Co-authored-by: Mohammed Y. Alnajdi <mohdnagfy@gmail.com>
Co-authored-by: biegan <bieganowski.rev@gmail.com>
(cherry picked from commit c6c6ec4)

# Conflicts:
#	geonode/layers/views.py
afabiani pushed a commit that referenced this issue Mar 17, 2021
* Bump urllib3 from 1.26.2 to 1.26.3 (#6908)

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.2 to 1.26.3.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/1.26.3/CHANGES.rst)
- [Commits](urllib3/urllib3@1.26.2...1.26.3)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Toni <toni.schoenbuchner@csgis.de>

* [Fixes #6880] Circle CI upload tests fail irregulary (#6881)

* [Fixes #6880] Circle CI upload tests fail irregulary

* CircleCI test fix: sometimes expires due to upload timeout in the test environment

* - Avoid infinite loop on upload testing

* Revert "CircleCI test fix: sometimes expires due to upload timeout in the test environment"

This reverts commit 66139fd.

Co-authored-by: Alessio Fabiani <alessio.fabiani@geo-solutions.it>
Co-authored-by: afabiani <alessio.fabiani@gmail.com>

* [Fixes #6914] Remove "add to basket" tool for documents and maps (#6915)

* Added malnajdi as contributor

* [Fixes #6910] meaningful filename for document download (#6911)

* get meaningful document filenames on download

* - Strip extension from document title before slugify it (e.g.: image.jpg instead of imagejpg.jpg)

Co-authored-by: afabiani <alessio.fabiani@gmail.com>
Co-authored-by: Alessio Fabiani <alessio.fabiani@geo-solutions.it>

* - CircleCI Upload Tests: trying to reduce more the risk of infinite loop on "wait_for_progress"

* [Fixes #6916] gsimporter.api.NotFound caused by missing trailing slash at the end of GEOSERVER_LOCATION (#6913)

* [Fixes #6916] gsimporter.api.NotFound caused by missing trailing slash at the end of GEOSERVER_LOCATION

* [Fixes #6916] unit test for GEOSERVER_LOCATION

* Bump django-cors-headers from 3.6.0 to 3.7.0 (#6901)

Bumps [django-cors-headers](https://github.com/adamchainz/django-cors-headers) from 3.6.0 to 3.7.0.
- [Release notes](https://github.com/adamchainz/django-cors-headers/releases)
- [Changelog](https://github.com/adamchainz/django-cors-headers/blob/master/HISTORY.rst)
- [Commits](adamchainz/django-cors-headers@3.6.0...3.7.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump amqp from 5.0.3 to 5.0.5 (#6905)

Bumps [amqp](https://github.com/celery/py-amqp) from 5.0.3 to 5.0.5.
- [Release notes](https://github.com/celery/py-amqp/releases)
- [Changelog](https://github.com/celery/py-amqp/blob/master/Changelog)
- [Commits](celery/py-amqp@v5.0.3...v5.0.5)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump pip from 21.0 to 21.0.1 (#6900)

Bumps [pip](https://github.com/pypa/pip) from 21.0 to 21.0.1.
- [Release notes](https://github.com/pypa/pip/releases)
- [Changelog](https://github.com/pypa/pip/blob/master/NEWS.rst)
- [Commits](pypa/pip@21.0...21.0.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump coverage from 5.3.1 to 5.4 (#6903)

Bumps [coverage](https://github.com/nedbat/coveragepy) from 5.3.1 to 5.4.
- [Release notes](https://github.com/nedbat/coveragepy/releases)
- [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst)
- [Commits](nedbat/coveragepy@coverage-5.3.1...coverage-5.4)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump pytest from 6.2.1 to 6.2.2 (#6907)

Bumps [pytest](https://github.com/pytest-dev/pytest) from 6.2.1 to 6.2.2.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst)
- [Commits](pytest-dev/pytest@6.2.1...6.2.2)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump djangorestframework-gis from 0.16 to 0.17 (#6902)

Bumps [djangorestframework-gis](https://github.com/openwisp/django-rest-framework-gis) from 0.16 to 0.17.
- [Release notes](https://github.com/openwisp/django-rest-framework-gis/releases)
- [Changelog](https://github.com/openwisp/django-rest-framework-gis/blob/master/CHANGES.rst)
- [Commits](openwisp/django-rest-framework-gis@v0.16.0...v0.17.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* - Algin setup.cfg to requirements.txt

* [Fixes #6922][REST API v2] Expose the curated thumbnail URL if it has… (#6923)

* [Fixes #6922][REST API v2] Expose the curated thumbnail URL if it has been uploaded

* - Add REST APIs test suite to CircleCI

* [Fixes #6918] Removal of QGIS support (#6919)

* [Cleanup and Refactor] Remove QGIS server backend dependencies

* [Cleanup and Refactor] Remove QGIS server backend dependencies

* - Fix LGTM issues

* allow Basic authenticated requests in LOCKDOWN mode

* fix to avoid circular import

* flake8 check fix

* added tests

* first draft

* remove service integration

* bbox structure fix

* layers and maps endpoints integration

* formatting

* flake formatting

* fix for longitude exceeding world width

* description

* fix for longitude exceeding world width 2

* fixes requested by PR review

* exceeding latitude fix

* immutability fix

* thumbs background configs

* poc of refactored wizard layer thumbnail

* missing EPSG prefix

* wizard map thumbnail

* correct layers order in a map

* security issue fix

* fix for small thumbnails

* comments to wizard wiget refactoring

* preserve thumbnail's ratio

* preserve thumbnail's ratio fix

* fix thumbnail URL for layer's wizard panels

* typo

* bbox method fix

* tests ongoing

* tests ongoing

* preparing the thumbnail.js for the refactoring in MapStore

* tests and formatting

* remove legacy thumbnail generation

* test fix

* minimal documentation of background generator options

* remove outdated tests

* add base.tests.ThumbnailTests cleanup

* formatting

* failing CI tests fix

* - Add thumbs tests

* - Fix test cases

* - Added few more consistency checks in order to make sure we catch 'zero-area' cases

* - Fix test cases

* - Fix test cases

* - Fix test cases

* fixes to thumbnail set frm detail pages

* - Fix test cases

* - Fix test cases

* - Fix test cases

* - Test fixes

* - Test fixes

* - Test fixes

Co-authored-by: allyoucanmap <bovio.stefano@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Toni <toni.schoenbuchner@csgis.de>
Co-authored-by: Alessio Fabiani <alessio.fabiani@geo-solutions.it>
Co-authored-by: afabiani <alessio.fabiani@gmail.com>
Co-authored-by: Florian Hoedt <gannebamm@gmail.com>
Co-authored-by: Mohammed Y. Alnajdi <mohdnagfy@gmail.com>
Co-authored-by: biegan <bieganowski.rev@gmail.com>
(cherry picked from commit c6c6ec4)

# Conflicts:
#	.circleci/config.yml
#	geonode/base/tests.py
#	geonode/geoserver/helpers.py
#	geonode/layers/tests.py
#	geonode/layers/utils.py
#	geonode/layers/views.py
#	geonode/maps/views.py
#	geonode/security/tests.py
#	geonode/services/serviceprocessors/arcgis.py
#	requirements.txt
#	scripts/docker/env/production/django.env
#	setup.cfg
afabiani pushed a commit that referenced this issue Mar 18, 2021
afabiani pushed a commit that referenced this issue Mar 18, 2021
…esaurusAvailableForm form not present on 3.1.x model
afabiani pushed a commit that referenced this issue Mar 18, 2021
…esaurusAvailableForm form not present on 3.1.x model
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 3.2.x PR should be backported to target version enhancement performance Issues regarding server Performance and Speed
Projects
None yet
Development

No branches or pull requests

3 participants