Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ingest jobs and job results into ES #236

Closed
jdries opened this issue Oct 8, 2022 · 11 comments
Closed

Ingest jobs and job results into ES #236

jdries opened this issue Oct 8, 2022 · 11 comments
Assignees
@soxofaan
Labels
architecture

Comments

@jdries
Copy link
Contributor

jdries commented Oct 8, 2022

Set up synchronization of jobs and job results with ES.
This allows us to build dashboards that give more insight into job specific metrics.
@soxofaan soxofaan mentioned this issue Oct 19, 2022
Closed
@soxofaan
Copy link
Member

Closed duplicate #246 I just created

Reference story for moving from Zookeeper to Elastic Search for batch job metadata.
Zookeeper was fine as initial POC, but scalability and flexibility (e.g. build an admin UI on top of it) is becoming a problem.
We decided to migrate to an Eleastic Search based solution

also related:

@soxofaan
Copy link
Member

internal ticket/ref: FD-5452

@jdries jdries assigned soxofaan and unassigned JohanKJSchreurs Dec 6, 2022
@soxofaan
Copy link
Member

soxofaan commented Dec 6, 2022

closing as duplicate of Open-EO/openeo-python-driver#153

@soxofaan soxofaan closed this as not planned Won't fix, can't repro, duplicate, stale Dec 6, 2022
soxofaan added a commit that referenced this issue Dec 8, 2022
@soxofaan
Issue #236 Rename original "JobRegistry" to "ZkJobRegistry"
e898aa4 
for clearer distinction with new ElasticJobRegistry
Open-EO/openeo-python-driver#153
soxofaan added a commit that referenced this issue Dec 8, 2022
@soxofaan
Issue #236 Rename original "JobRegistry" to "ZkJobRegistry"
b87dbac 
for clearer distinction with new ElasticJobRegistry
Open-EO/openeo-python-driver#153
soxofaan added a commit that referenced this issue Dec 8, 2022
@soxofaan
Issue #236 Use JOB_STATUS constants for easier job status tracking
1dd4962 
Open-EO/openeo-python-driver#153
soxofaan added a commit that referenced this issue Dec 12, 2022
@soxofaan
Issue #236 EJR: introduce some status changes
e175d46
soxofaan added a commit that referenced this issue Dec 13, 2022
@soxofaan
Issue #236 EJR: initial unit test coverage for job_tracker with YARN+ZK
8852a8d
soxofaan added a commit that referenced this issue Dec 14, 2022
@soxofaan
Make GpsBatchJobs.get_job_output_dir non-static
92e7f06 
preparation to make `GpsBatchJobs._OUTPUT_ROOT_DIR` non-global

related to #236 #283 #285
soxofaan added a commit that referenced this issue Dec 14, 2022
@soxofaan
Make GpsBatchJob._OUTPUT_ROOT_DIR non-global
dbd6a93 
related to #236 #283 #285
soxofaan added a commit that referenced this issue Dec 14, 2022
@soxofaan
Make GpsBatchJobs.get_job_output_dir non-static
25c35be 
preparation to make `GpsBatchJobs._OUTPUT_ROOT_DIR` non-global

related to #236 #283 #285
soxofaan added a commit that referenced this issue Dec 14, 2022
@soxofaan
Make GpsBatchJob._OUTPUT_ROOT_DIR non-global
f5bd918 
related to #236 #283 #285
soxofaan added a commit that referenced this issue Dec 14, 2022
@soxofaan
Issue #236 add job_tracker test coverage for elastic job registry calls
d2e89e8
soxofaan added a commit that referenced this issue Dec 14, 2022
@soxofaan
fixup! Issue #236 add job_tracker test coverage for elastic job regis…
2952eb7 
…try calls
soxofaan added a commit that referenced this issue Dec 14, 2022
@soxofaan
Issue #236 add job_tracker tests about job status check failure
fbc5ffc
soxofaan added a commit that referenced this issue Dec 14, 2022
@soxofaan
Issue #236 job_tracker: update/started/finished handling
c6f498a
soxofaan added a commit that referenced this issue Dec 15, 2022
@soxofaan
fixup! Issue #236 job_tracker: update/started/finished handling
4b8e1b9
soxofaan added a commit that referenced this issue Dec 15, 2022
@soxofaan
Issue #236 add job_tracker test coverage for elastic job registry calls
33dcfc6
soxofaan added a commit that referenced this issue Dec 15, 2022
@soxofaan
Issue #236 add job_tracker tests about job status check failure
70fa5f0
soxofaan added a commit that referenced this issue Dec 15, 2022
@soxofaan
Issue #236 job_tracker: update/started/finished handling
4538240
soxofaan added a commit that referenced this issue Dec 15, 2022
@soxofaan
Remove outdated/unused logic from JobTracker (#236 related)
2fb42da
soxofaan added a commit that referenced this issue Dec 15, 2022
@soxofaan
Issue #236 add job_tracker test coverage for elastic job registry calls
4753bd6
soxofaan added a commit that referenced this issue Feb 17, 2023
@soxofaan
Issue #236: retry HTTP requests in job_tracker
6f2851a
soxofaan added a commit that referenced this issue Mar 8, 2023
@soxofaan
Issue #236 add logging of job_id extra in DoubleJobRegistry._just_log…
089d1bb 
…_errors
soxofaan added a commit that referenced this issue Mar 22, 2023
@soxofaan
Issue #236 don't set up job registry from within batch job driver
d5de9cb
soxofaan added a commit that referenced this issue Apr 5, 2023
@soxofaan
Issue #236 run job_tracker_v2 without --fail-fast option now
541c413
@soxofaan soxofaan mentioned this issue May 9, 2023
Closed
@soxofaan
Copy link
Member

soxofaan commented May 9, 2023

with Open-EO/openeo-job-registry-elastic-api#28 being fixed, we can now proceed with improving the scope of EJR (elastic job registry) and DoubleJobRegistry.

for example: switching priority between ZK based registry and EJR registry for some operations

soxofaan added a commit that referenced this issue Aug 29, 2023
@soxofaan
Issue #236 eliminate ZK specific ZkJobRegistry.get_dependency_sources
6319bdb
soxofaan added a commit that referenced this issue Aug 29, 2023
@soxofaan
fixup! Issue #236 eliminate ZK specific `ZkJobRegistry.get_dependency…
76cd1b9 
…_sources`
soxofaan added a commit that referenced this issue Aug 29, 2023
@soxofaan
Issue #236 eliminate ZK specific zk_job_info_to_metadata in backend.py
ab82f50
soxofaan added a commit that referenced this issue Aug 29, 2023
@soxofaan
Issue #236 First steps making ZK optional in DoubleJobRegistry
2a6b5ef
soxofaan added a commit that referenced this issue Aug 30, 2023
@soxofaan
fixup! Issue #236 eliminate ZK specific zk_job_info_to_metadata in …
52ad579 
…backend.py
@soxofaan
Copy link
Member

Currently working on making the ZK side of DoubleJobRegistry optional

soxofaan added a commit that referenced this issue Aug 30, 2023
@soxofaan
Issue #236 make ZK code path optional in DoubleJobRegistry.set_status
93e2cc1
soxofaan added a commit that referenced this issue Aug 30, 2023
@soxofaan
Issue #236 make ZK code path optional in DoubleJobRegistry.get_user_jobs
a0ef190
soxofaan added a commit that referenced this issue Aug 30, 2023
@soxofaan
Issue #236 DoubleJobRegistry: keep working (best effort) on KazooTi…
d844d5e 
…meoutError
soxofaan added a commit that referenced this issue Aug 30, 2023
@soxofaan
Issue #236 optional ZK code path in DoubleJobRegistry.get_job and get…
37fd4ad 
…_job_metadata
@soxofaan soxofaan mentioned this issue Aug 30, 2023
Closed
soxofaan added a commit that referenced this issue Aug 30, 2023
@soxofaan
Issue #236/#498 optional ZK handling in DoubleJobRegistry.delete
903c755
soxofaan added a commit that referenced this issue Aug 30, 2023
@soxofaan
Issue #236/#498 optional ZK handling for set_dependencies/remove_depe…
1a5862d 
…ndencies
soxofaan added a commit that referenced this issue Aug 30, 2023
@soxofaan
Issue #236/#498 optional ZK handling for mark_ongoing
0efcc42
soxofaan added a commit that referenced this issue Aug 30, 2023
@soxofaan
Issue #236/#498 more optional ZK handling
82d1ee3 
set_dependency_status, set_dependency_usage, set_proxy_user, set_application_id
soxofaan added a commit that referenced this issue Aug 30, 2023
@soxofaan
Issue #236/#498 make zk_job_registry_factory optional in DoubleJobReg…
3020dd2 
…istry
soxofaan added a commit that referenced this issue Sep 12, 2023
@soxofaan
Issue #236/#498 transfer some todo notes to newer ticket #498
4a863b5
@soxofaan
Copy link
Member

closing this long standing issue, follow up is for #498

@soxofaan soxofaan mentioned this issue Sep 29, 2023
Closed
soxofaan added a commit that referenced this issue Sep 29, 2023
@soxofaan
Issue #523/#498/#236 Fall back on EJR when no job metadata in ZK
9a89505
soxofaan added a commit that referenced this issue Dec 6, 2023
@soxofaan
make GpsBatchJobs._start_job compatible with EJR based job info
70bd2c4 
refs: #498 #236 #531
soxofaan added a commit that referenced this issue Jan 4, 2024
@soxofaan
Add use_zk_job_registry config to optionally disable `ZkJobRegistry…
5475236 
…` usage #236/#498/#632
@soxofaan soxofaan mentioned this issue Sep 12, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@soxofaan soxofaan

Labels
architecture
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@soxofaan @bossie @jdries @JohanKJSchreurs @JanssenBrm