Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support multiple accounting backends in aggregator scenario #531

Closed
JanssenBrm opened this issue Oct 6, 2023 · 4 comments
Closed

Support multiple accounting backends in aggregator scenario #531

JanssenBrm opened this issue Oct 6, 2023 · 4 comments
Assignees

Comments

@JanssenBrm
Copy link

JanssenBrm commented Oct 6, 2023

With the deployment of the aggregator, processing can be divided on multiple back-ends. For example, when deploying the aggregator in CDSE, some of the processing could be done on the TerraScope environment.

In order to fully support this, the final processing environment (in the example above, TerraScope) should interact with the proper accounting backend (in the example, the CDSE marketplace-cost-api). Specifically, whenever TerraScope gets a request to process a job from CDSE it should:

  • Execute the credit check against the CDSE backend
  • Report the resources used to the CDSE backend

Some possible approaches for openEO to determine the correct accounting backend have already been discussed with @soxofaan :

  • A user agent could identify from which environment the request is coming.
  • Inspection of the token could achieve the same thing.
soxofaan added a commit that referenced this issue Oct 13, 2023
Various doc tweaks as side-effect of getting to know the ETL implementation
soxofaan added a commit that referenced this issue Oct 13, 2023
…lated)

Use simple method iso closure in __init__
soxofaan added a commit to Open-EO/openeo-python-driver that referenced this issue Oct 16, 2023
soxofaan added a commit to Open-EO/openeo-python-driver that referenced this issue Oct 16, 2023
soxofaan added a commit that referenced this issue Oct 16, 2023
@soxofaan
Copy link
Member

soxofaan added a commit that referenced this issue Oct 19, 2023
Various doc tweaks as side-effect of getting to know the ETL implementation
soxofaan added a commit to Open-EO/openeo-python-driver that referenced this issue Oct 19, 2023
soxofaan added a commit to Open-EO/openeo-python-driver that referenced this issue Oct 19, 2023
soxofaan added a commit to Open-EO/openeo-python-driver that referenced this issue Oct 19, 2023
soxofaan added a commit to Open-EO/openeo-python-driver that referenced this issue Oct 19, 2023
soxofaan added a commit to Open-EO/openeo-python-driver that referenced this issue Oct 20, 2023
soxofaan added a commit to Open-EO/openeo-python-driver that referenced this issue Oct 20, 2023
soxofaan added a commit that referenced this issue Oct 20, 2023
…lated)

Use simple method iso closure in __init__
soxofaan added a commit that referenced this issue Oct 20, 2023
soxofaan added a commit that referenced this issue Oct 20, 2023
soxofaan added a commit to Open-EO/openeo-python-driver that referenced this issue Oct 20, 2023
soxofaan added a commit to Open-EO/openeo-python-driver that referenced this issue Oct 20, 2023
soxofaan added a commit that referenced this issue Oct 20, 2023
…lated)

Use simple method iso closure in __init__
soxofaan added a commit that referenced this issue Oct 20, 2023
soxofaan added a commit that referenced this issue Oct 20, 2023
soxofaan added a commit that referenced this issue Oct 20, 2023
eliminates explicit access token handling in various places

related: #545
soxofaan added a commit that referenced this issue Oct 31, 2023
…lated)

Use simple method iso closure in __init__
soxofaan added a commit that referenced this issue Oct 31, 2023
soxofaan added a commit that referenced this issue Dec 14, 2023
Add TTL based caching feature to `get_etl_api` to avoid full EtlApi setup (including Http session andOIDC client cred auth) repeatedly
soxofaan added a commit that referenced this issue Dec 15, 2023
Add TTL based caching feature to `get_etl_api` to avoid full EtlApi setup (including Http session andOIDC client cred auth) repeatedly
soxofaan added a commit that referenced this issue Dec 15, 2023
Add TTL based caching feature to `get_etl_api` to avoid full EtlApi setup (including Http session andOIDC client cred auth) repeatedly
soxofaan added a commit that referenced this issue Dec 15, 2023
Add TTL based caching feature to `get_etl_api` to avoid repeated full EtlApi setup (including http session and OIDC client cred auth)
soxofaan added a commit that referenced this issue Dec 15, 2023
soxofaan added a commit that referenced this issue Jan 30, 2024
soxofaan added a commit that referenced this issue Feb 1, 2024
soxofaan added a commit that referenced this issue Feb 1, 2024
soxofaan added a commit that referenced this issue Feb 1, 2024
soxofaan added a commit that referenced this issue Feb 2, 2024
@soxofaan
Copy link
Member

soxofaan commented Feb 8, 2024

did first successful experiment today:
created job (j-240208eceedf489dbe7ad113d25e0bf0) on "openeo-dev.vito.be" backend with job_options={"etl_api_id": "CDSE-staging"} and the job tracker (for dev) sent the resource usage data to the CDSE staging ETL API

@soxofaan
Copy link
Member

soxofaan commented Feb 8, 2024

setup was done through openeogeotrellis.integrations.etl_api.MultiEtlApiConfig usage in GpsBackendConfig in openeo-deploy project

@soxofaan
Copy link
Member

I'm going to close this ticket for now.
There is still other work to be done (e.g. at level of aggregator: Open-EO/openeo-aggregator#135, and aligning OIDC providers in multiple deployments), but that is food for other tickets

soxofaan added a commit to Open-EO/openeo-python-driver that referenced this issue Apr 30, 2024
soxofaan added a commit that referenced this issue Apr 30, 2024
soxofaan added a commit to Open-EO/openeo-python-driver that referenced this issue May 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants