Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/elastic search #87

Merged
merged 9 commits into from
Sep 5, 2023
Merged

Feature/elastic search #87

merged 9 commits into from
Sep 5, 2023

Conversation

lorenzomercurio
Copy link
Contributor

@lorenzomercurio lorenzomercurio commented Aug 12, 2023

Mainly implemented correct log querying from Elasticsearch (see below for more details)
Documentation regarding configuration of custom Elasticsearch cluster for this project

FEATURES IMPLEMENTED:

  • Managed most common HTTP server and logical errors that can occur while querying Elasticsearch (ES) or due to Spring API itself (e.g., job_id doesn't exist, no hits retrieved, void index, or invalid/deleted index).
  • Properly handled multiline log entries in grok filters for Logstash.
  • Correctly filtered the "job_id" field in Elasticsearch by enforcing exact matching (added ".keyword" to "job_id" => "job_id.keyword" inside ES queries, following the official Elasticsearch documentation).
  • Rewrote Elasticsearch querying using Java ES official mapping instead of the old raw text query.

FEATURES TO BE IMPLEMENTED:

  • Implement proper management of ES query results by using offset and limits instead of the temporary "/_search?size=10000" approach to fully benefit from proper ES hits-retrieving pagination (in accordance with ES official documentation).
  • Every hostname is treated as "valid" during SSL certificate verification => this has to be fixed.
  • Implement value calculation/return for some optional fields returned by the API (currently hardcoded at default or null values), following the OpenEO documentation specifications.
  • Make further code improvements.
  • Implement logEntry and add it to the logEntries list, then return the result with pagination links (this relates to the first to-do item).

@lorenzomercurio lorenzomercurio marked this pull request as draft August 12, 2023 16:37
@lorenzomercurio lorenzomercurio marked this pull request as ready for review August 12, 2023 16:38
@pierocampa pierocampa added the enhancement New feature or request label Aug 12, 2023
Copy link
Member

@pierocampa pierocampa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Log requests are properly handled, there's just a few italian comments to be translated, and the whole SSL context and host verification part which might probably better removed, letting the default Java implementations to be used.

docs/elk.md Outdated Show resolved Hide resolved
docs/elk.md Outdated Show resolved Hide resolved
@pierocampa pierocampa merged commit 3974e5e into master Sep 5, 2023
1 check passed
@pierocampa pierocampa deleted the feature/elastic-search branch September 5, 2023 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants