This project uses Elastic Cloud Billing APIs to get the usage costs of your clusters in a given organization
WARNINGs & Remarks:
- This project is not supported by Elastic
- This project was tested on Elastic 8.6.2
- This project needs an APM/integration server
- Elastic Cloud Billing APIs are still in Beta and subject to changes (https://www.elastic.co/guide/en/cloud/current/Billing_Costs_Analysis.html)
- Credits for origin and initial project goes to Jeff Vestal https://github.com/jeffvestal/ess-billing-ingest
In your Elastic Cluster, go to the Dev Tools and run everything inside the file ess.billing.elasticsearch.ilm_pipelines_mapping.json
In Elastic Cloud console, generate an API Key (Features > API keys > Create API key) and save it next to you. We will need it later. This API key is used to fetch the cost metrics of your clusters inside the organisation under which you are logged in Elastic Cloud. The cost metrics will be fetched from the service api.elastic-cloud.com
Let's call it billing_api_key => We will need it to populate config.ini
Get the cloud id of the cluster in which the scrapped cost metrics will be ingested => We will need it to populate config.ini , field billing_es_id
Inside your Elastic Cluster, go to Stack Management > API keys > Create API key and generate a key This key will be used by our python script to ingest cost metrics scrapped from api.elastic-cloud.com into the elastic cluster
Let's call it billing_es_api => We will need it to populate config.ini
fetch these information from Elastic APM integration in the integration policy under which your APM server is running
=> We will need them to populate config.ini in the dedicated section of apm
On a linux VM running python 3.x at least,
- install python packages using pip3 : elasticsearch, requests, elasticapm
- install ess-billing-ingest.py in the folder of your choice
- install config.ini in the same folder and populate it according previous chapters
And then run it as
nohup /<path to>/ess-billing-ingest.py /<path to>/config.ini > /dev/null 2>&1
Go to Stack Management>Kibana>Saved Objects and import the dashboards in the file ess.billing.kibana.export.ndjson
Setup a filebeat with corresponding settings in filebeat.yml to import the python script logs into the same Elastic Cluster where you import the scrapped cost metrics Logs are written in /var/log/ess_billing/ folder
- in case of api.elastic-cloud.com timeout, the script will catch the exception and may stop. You will have to launch it again
- Dashboards can be imported into 8.6.2 only, for versions below, you might get an error from the Elastic Cluster
- APM is enabled by default for now. ENABLED setting in config.ini is ignored for now
