2022 Proposal: a new metadata editor
| Date | 2022-03-24 | Contacts | olivier.guyot@camptocamp.com, florent.gravin@camptocamp.com |
| Status | To be proposed | Release | TBD |
| Resources | TBD | Ticket # | - |
| Source code | - | ||
| Funding | geo2France, DataGrandEst and other interested parties which are willing to join the effort | ||
This is a proposal for starting the development of a new metadata editor in GeoNetwork. It contains the general vision that Camptocamp has for such a project as well as technical considerations.
Authoring metadata in GeoNetwork has been a source of various complaints for a long time. This includes notably:
- Too many fields, tabs, buttons etc. present in the form
- Fields and inputs are unclear
- Producing valid metadata requires going through trial-and-error iterations, involving unclear errors
- The editing form is a direct representation of the underlying XML, one form cannot work across several schemas
- Slow and unresponsive UI *...and more
Offering custom editor forms with a better UX is currently doable and is something that we often do for customers. It is time-consuming though, very error-prone and does not rely on any kind of stable API so things might break randomly on version bumps. It also does not let us do all the things we'd like and we are still subject to the general limitations of the editor workflow in GeoNetwork.
We (Camptocamp) believe that a new metadata editor could be written as part of the geonetwork-ui project. This will allow us to solve most issues that users experience, while also opening up new possibilities for authoring metadata in other formats (inc. non XML ones).
The new metadata editor would be built on three principles:
-
Accessible to everyone
We want to offer an application that allows even the unexperienced users to describe their data without too much hassle, and without real training. The application will also accommodate for more advanced users by offering more customizations on the user's explicit request. By default, only a handful of relevant fields will be shown in each section to avoid burdening the user cognitive load too much. Producing valid metadata will be done by anticipation, meaning that invalid inputs will simply not be allowed.
-
Forget about metadata schemas
We want to build the new metadata editor in a way that enables it to work seamlessly on various metadata schemas; more on that in the technical section. Which metadata schema is used internally (if any) will absolutely not impact the user experience.
-
Build your own editor
We want to offer powerful tools and options for completely configuring the editor and the way it is laid out. Each organization will be able to choose what fields should be shown in what section, what are the default or allowed values, which parts should be hidden or conditionally shown, etc. This is similar in principle to what the
config-editor.xmlfile currently allows in GeoNetwork, except that our approach will be used from the start and not as an optional way to "tweak" an existing form.
We currently cannot settle on a precise architecture for this new project. I will describe here an objective that is both achievable and involves significant additions to the GeoNetwork Microservices project.
This goal involves three components:
- A frontend application using geonetwork-ui
- A backend microservice for storing metadata
- A backend microservice for transforming metadata from and to XML formats (iso19139, iso19115-3 etc.)
This will be a new application alongside the datahub and datafeeder. It will benefit from UX work from the start (mockups etc.), as well as existing work done in the datafeeder for user input.
Input fields for metadata will be organized into sections, with each section having at most 7 fields to limit cognitive load. An overview will be available to the user to give him/her a good understanding of what is expected of them.
The challenge for this application will be to design it so that every field in every section will be placed at runtime, i.e. not hardcoded in the HTML templates of the app.
Example mockup (french):
We want to make the editor form completely customizable from the ground up. Not in the sense that a default structure is provided and administrators can tweak around it, but in the sense that a new editor can be built from scratch simply using a configuration file.
Here is an example of what a configuration file could look like:
sections:
- generalInfo:
labelKey: editor.section.generalInfo
fields:
# optional, must be unique
- id: record_title
# name of the field in the data model
# Note: this is a optional, a field could be included but directly
# linked to a part of the data model (e.g. a toggle)
model: title
# translation keys for different parts of the field
labelKey: editor.fields.title.label
tipKey: editor.fields.title.tip
helpKey: editor.fields.title.help
# Types can include:
# - text
# - number
# - rich: text with formatting, images etc.
# - date
# - list
# - spatial_extent
# - temporal_extent
# - url
# - file: file to upload or remote url
# - toggle
type: text
# Special types an include:
# - object: creates a embedded form just for this object (e.g. contacts)
# - array: adds controls for adding and removing elements in the array
# This field can include an expression, e.g.
# required: ${{ isEmpty(record.fields['otherTitle']) }}
required: true
readOnly: false
# Each field type comes with its sets of special rules, for instance:
# - text.maxLength: 1234
# - text.minLength: 12
# - number.min: 100
# - number.max: 500
# - number.step: 50
# - date.minDateTime: 167000234234
# - date.maxDateTime: 167000854321
# - list.fromApi: /api/keywords?lang=fre
# - list.fromJson: https://my.site/files/list.json
# - list.fromValues: [“Keyword 1”, “Keyword 2”, “Keyword 3”]
# - rich.disallowFormatting: true
# - file.allowFileUpload: true
# - file.convertToInlineBase64: true
# Each rule will be documented
rules:
text.maxRecommendedLength: 200
- contactInfo:
labelKey: editor.section.contactInfo
fields:
# An example of an array of objets
- model: contactsForData
type: array
labelKey: editor.fields.contacts.label
items:
type: object
labelKey: editor.fields.contact.label
fields:
- model: firstName
labelKey: editor.fields.contact.firstName.label
type: text
- model: lastName
labelKey: editor.fields.contact.firstName.label
type: text
- model: email
labelKey: editor.fields.contact.firstName.label
type: text
required: true
rules:
- text.validateEmail: true
required: true
rules:
- array.minCount: 1
- distribution:
labelKey: editor.section.distribution
- accessAndUseConditions:
labelKey: editor.section.accessAndUseConditions
- datasetInfo:
labelKey: editor.section.datasetInfo
To facilitate the process of creating custom editor forms, a sandbox-type app could be hosted e.g. on GH pages to let people experiment with a config file and see the result right away. This would be similar to the Swagger Editor.
In order to uncouple the process of authoring metadata with the XML schema used to store metadata in GeoNetwork, a new "metadata editor" backend microservice will be introduced. Its responsibility will be to offer an API for interacting with metadata records in a native format instead of XML documents. The data model used for this will be built to accommodate the most common usages, for example: title, abstract, contacts, links to online resources, usage and download constraints, etc.
Its API will allow reading records by UUID, editing records, deleting and creating new records. Internally this microservice will interact with the GeoNetwork database; it will also rely on the converter microservice (see next paragraph) to transform metadata records from and to iso19139. This means that all the metadata records present in the GeoNetwork database can be edited in the new editor, and all metadata management functions in GeoNetwork can still be used on the records that were authored in the new editor. GeoNetwork is still the source of truth in that scenario.
This service will be responsible for converting metadata from and to established schemas of any kinds. Its responsibility will be limited to applying predefined processes to either native metadata objects coming from the editor microservice, or XML metadata coming from the GeoNetwork database to be then transformed to native objects.
A schema is nothing more than:
- A transformation from a native object to an interoperable document (out)
- A transformation from an interoperable document to a native object (in)
- A set of test cases validation the expected results for both transformations
How transformations will work exactly has yet to be determined.
This showcases what a new architecture relying on the above microservices would look like.
Vote Proposed the 6th of April 2022
- Jo Cook +1
- Florent Gravin +1
- Jeroen Ticheler +1
- Jose Garcia +1
- Simon Pigot +1
- François-Xavier Prunayre
- Emanuele Tajariol
- Paul van Genuchten
- All