[Metricbeat] Change Account ID to Project ID in gcp.billing module

[legoguy1000]

What does this PR do?

Currently the GCP Project ID is being ingested as cloud.account.id instead of cloud.project.id. This PR fixes that.

Why is it important?

See above

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Closes GCP billing reports wrong account ID  #26357

Use cases

Screenshots

Logs

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: kaiyan-sheng commented: /test

• Start Time: 2021-06-25T03:49:50.593+0000

• Duration: 78 min 36 sec

• Commit: 2352c7f

Test stats 🧪

Test	Results
Failed	0
Passed	1969
Skipped	192
Total	2161

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	1969
Skipped	192
Total	2161

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[kaiyan-sheng]

Hi @legoguy1000 thank you for your contribution!! Do you know if we can get the account ID and account name in GCP from the given project ID? I would like to report both account ID/Name and project ID in gcp.billing.

[legoguy1000]

@kaiyan-sheng based on this, https://cloud.google.com/billing/docs/how-to/export-data-bigquery-tables#data-schema, I made additional changes to the query to pull the account info. I don't have a GCP account with any actual billing so I don't have a good way to validate the query besides just looking at the schema in the above link. I was able to add the project.name and what I believe is the account.id. According to the schema, there isn't a field in the table for account.name.

[legoguy1000]

@kaiyan-sheng can you run the tests when you get a chance.

[kaiyan-sheng]

Thanks for the change. I tested it locally and here is what I saw:

2021-06-23T15:33:35.456-0600    ERROR   [billing]       billing/billing.go:242  bigquery Wait failed: googleapi: Error 400: SELECT list expression references column billing_account_id which is neither grouped nor aggregated at [6:33], invalidQuery

2021-06-23T15:54:33.736-0600    ERROR   [billing]       billing/billing.go:241  bigquery Wait failed: googleapi: Error 400: SELECT list expression references column cost_type which is neither grouped nor aggregated at [6:33], invalidQuery

I will look into it a bit more tomorrow :)

[kaiyan-sheng]

/test

[kaiyan-sheng]

OK, adding GROUP BY and ORDER BY fixed the issue:

query := fmt.Sprintf(`
			SELECT
				invoice.month,
				project.id,
				project.name,
				billing_account_id,
				cost_type,
			  (SUM(CAST(cost * 1000000 AS int64))
				+ SUM(IFNULL((SELECT SUM(CAST(c.amount * 1000000 as int64)) FROM UNNEST(credits) c), 0))) / 1000000
				AS total_exact
			FROM %s
			WHERE project.id IS NOT NULL
			AND invoice.month = '%s'
			AND cost_type = '%s'
			GROUP BY 1, 2, 3, 4, 5
			ORDER BY 1 ASC, 2 ASC, 3 ASC, 4 ASC, 5 ASC;`, tableMeta.tableFullID, month, costType)

[legoguy1000]

@kaiyan-sheng Query fixed. Should be good for CI tests now.

[kaiyan-sheng]

/test

[legoguy1000]

@kaiyan-sheng Looks good to merge now.