Indexing / Add the capability to index the catalogue content in Elasticsearch. #1966
Conversation
fxprunayre
changed the title
| @@ -566,7 +568,7 @@ public void indexMetadata(final List<String> metadataIds) throws Exception { | |||
| /** | |||
| * TODO javadoc. | |||
| */ | |||
| public void indexMetadata(final String metadataId, boolean forceRefreshReaders) throws Exception { | |||
| public void indexMetadata(final String metadataId, boolean forceRefreshReaders, ISearchManager searchManager) throws Exception { | |||
There was a problem hiding this comment.
I think would be better to keep a method also with metadataId and forceRefreshReaders only that calls this one sending searchManager as null, that is used in all places changed to send searchManager to null. Wth the change there's quite changes where searchManager is send as null.
There was a problem hiding this comment.
Once we move totally to ES there will no need of this - having only one SearchManager. See what has been done last year in Solr POC https://github.com/titellus/core-geonetwork/blob/solr/core/src/main/java/org/fao/geonet/kernel/search/SolrSearchManager.java. I would say that we can tolerate this for the time being ?
| @@ -75,6 +76,9 @@ | |||
| public static final String TREE_FIELD_SUFFIX = "_tree"; | |||
| public static final String FEATURE_FIELD_PREFIX = "ft_"; | |||
|
|
|||
| @Value("es.index.features") | |||
| private String index = "features"; | |||
|
|
|||
| @Autowired | |||
| private EsClient client; | |||
There was a problem hiding this comment.
Will this work with multinode setup?
There was a problem hiding this comment.
Multinode mode could you use the same Elasticsearch index for features (and possibly catalog and search stats). BTW we should maybe ask if we want to keep this mode and support it, but that another issue.
|
Replaced by #2082 |
Overview
Main goal of this proposal is to be able to push GeoNetwork content in a remote index in order to build dashboards on the content of the catalogue. This work will also help on the future migration from Lucene to Elasticsearch.
Proposal
Add the capability to index the catalogue content in a remote elasticsearch index. Indexing is similar to actual indexing process ie.:
This will expose all metadata records in the Elasticsearch index (you've to take care of restricting access to not public record if needed).
From the admin trigger the indexation process:
API allows to remove index content and index the full catalogue or a selection:
Usages
Creating dashboards using Kibana
Dahsboard can be created on metadata content and on "internal" fields like validation status, publication groups, ... (which is a limitation of user of DAOBS project which only harvest records from CSW)
Dashboards can then be added to the admin console:
Reference document