Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dev-utils/ship-ci-stats] fail when CI stats is down #90678

Merged

Conversation

spalger
Copy link
Contributor

@spalger spalger commented Feb 8, 2021

CI stats experienced an outage caused by a load balancer misconfiguration in Elastic cloud https://status.elastic.co/incidents/cxggjn7drfbh

During this outage the CI Stats reporter was unable to ship metrics to the service which lead to the metrics not being reported for a build. This wasn't identified as an issue because initially the metrics weren't mission critical and were for informational purposes and we didn't want to trigger issues while getting started. CI Stats is now a part of our CI suite and used to validate things, so we rely on them and should fail builds if CI Stats is down.

This updates the script introduced in #90482 to fail the CLI when metrics are not properly shipped to the service.

@spalger spalger added Team:Operations Team label for Operations Team v8.0.0 release_note:skip Skip the PR/issue when compiling release notes v7.12.0 labels Feb 8, 2021
@spalger spalger requested a review from a team as a code owner February 8, 2021 20:42
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-operations (Team:Operations)

@spalger spalger changed the title [dev-utils/ship-ci-stats] fail the ship script when CI stats doesn't respond [dev-utils/ship-ci-stats] fail when CI stats is down Feb 8, 2021
@spalger spalger added the auto-backport Deprecated - use backport:version if exact versions are needed label Feb 8, 2021
@spalger spalger enabled auto-merge (squash) February 8, 2021 21:32
@spalger spalger force-pushed the fix/do-not-ignore-metric-recording-failures branch from c1a7cb9 to 189da29 Compare February 8, 2021 22:28
@spalger spalger merged commit 3ef3e93 into elastic:master Feb 8, 2021
@kibanamachine
Copy link
Contributor

Backport result

{"level":"info","message":"POST https://api.github.com/graphql (status: 200)"}
{"level":"info","message":"POST https://api.github.com/graphql (status: 200)"}
{"meta":{"labels":["Team:Operations","auto-backport","release_note:skip","v7.12.0","v8.0.0"],"branchLabelMapping":{"^v8.0.0$":"master","^v7.12.0$":"7.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"},"existingTargetPullRequests":[]},"level":"info","message":"Inputs when calculating target branches:"}
{"meta":["7.x"],"level":"info","message":"Target branches inferred from labels:"}
{"meta":{"killed":false,"code":2,"signal":null,"cmd":"git remote rm kibanamachine","stdout":"","stderr":"error: No such remote: 'kibanamachine'\n"},"level":"info","message":"exec error 'git remote rm kibanamachine':"}
{"meta":{"killed":false,"code":2,"signal":null,"cmd":"git remote rm elastic","stdout":"","stderr":"error: No such remote: 'elastic'\n"},"level":"info","message":"exec error 'git remote rm elastic':"}
{"level":"info","message":"Backporting [{\"sourceBranch\":\"master\",\"targetBranchesFromLabels\":[\"7.x\"],\"sha\":\"3ef3e9324fd9d77dd442b09e3355fb005c4b3caf\",\"formattedMessage\":\"[dev-utils/ship-ci-stats] fail when CI stats is down (#90678)\",\"originalMessage\":\"[dev-utils/ship-ci-stats] fail when CI stats is down (#90678)\\n\\nCo-authored-by: spalger <spalger@users.noreply.github.com>\",\"pullNumber\":90678,\"existingTargetPullRequests\":[]}] to 7.x"}

Backporting to 7.x:
{"level":"info","message":"Backporting via filesystem"}
{"meta":{"killed":false,"code":1,"signal":null,"cmd":"git cherry-pick 3ef3e9324fd9d77dd442b09e3355fb005c4b3caf","stdout":"CONFLICT (modify/delete): packages/kbn-dev-utils/src/ci_stats_reporter/ship_ci_stats_cli.ts deleted in HEAD and modified in 3ef3e9324fd ([dev-utils/ship-ci-stats] fail when CI stats is down (#90678)). Version 3ef3e9324fd ([dev-utils/ship-ci-stats] fail when CI stats is down (#90678)) of packages/kbn-dev-utils/src/ci_stats_reporter/ship_ci_stats_cli.ts left in tree.\n","stderr":"error: could not apply 3ef3e9324fd... [dev-utils/ship-ci-stats] fail when CI stats is down (#90678)\nhint: after resolving the conflicts, mark the corrected paths\nhint: with 'git add <paths>' or 'git rm <paths>'\nhint: and commit the result with 'git commit'\n"},"level":"info","message":"exec error 'git cherry-pick 3ef3e9324fd9d77dd442b09e3355fb005c4b3caf':"}
Commit could not be cherrypicked due to conflicts

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Feb 8, 2021
Co-authored-by: spalger <spalger@users.noreply.github.com>
@kibanamachine
Copy link
Contributor

Backport result

{"level":"info","message":"POST https://api.github.com/graphql (status: 200)"}
{"level":"info","message":"POST https://api.github.com/graphql (status: 200)"}
{"meta":{"labels":["Team:Operations","auto-backport","release_note:skip","v7.12.0","v8.0.0"],"branchLabelMapping":{"^v8.0.0$":"master","^v7.12.0$":"7.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"},"existingTargetPullRequests":[]},"level":"info","message":"Inputs when calculating target branches:"}
{"meta":["7.x"],"level":"info","message":"Target branches inferred from labels:"}
{"meta":{"killed":false,"code":2,"signal":null,"cmd":"git remote rm kibanamachine","stdout":"","stderr":"error: No such remote: 'kibanamachine'\n"},"level":"info","message":"exec error 'git remote rm kibanamachine':"}
{"meta":{"killed":false,"code":2,"signal":null,"cmd":"git remote rm elastic","stdout":"","stderr":"error: No such remote: 'elastic'\n"},"level":"info","message":"exec error 'git remote rm elastic':"}
{"level":"info","message":"Backporting [{\"sourceBranch\":\"master\",\"targetBranchesFromLabels\":[\"7.x\"],\"sha\":\"3ef3e9324fd9d77dd442b09e3355fb005c4b3caf\",\"formattedMessage\":\"[dev-utils/ship-ci-stats] fail when CI stats is down (#90678)\",\"originalMessage\":\"[dev-utils/ship-ci-stats] fail when CI stats is down (#90678)\\n\\nCo-authored-by: spalger <spalger@users.noreply.github.com>\",\"pullNumber\":90678,\"existingTargetPullRequests\":[]}] to 7.x"}

Backporting to 7.x:
{"level":"info","message":"Backporting via filesystem"}
{"level":"info","message":"Creating PR with title: \"[7.x] [dev-utils/ship-ci-stats] fail when CI stats is down (#90678)\". kibanamachine:backport/7.x/pr-90678 -> 7.x"}
{"level":"info","message":"POST /repos/elastic/kibana/pulls - 201 in 1035ms"}
{"level":"info","message":"Adding assignees to #90706: spalger"}
{"level":"info","message":"POST /repos/elastic/kibana/issues/90706/assignees - 201 in 484ms"}
{"level":"info","message":"Adding labels: backport"}
{"level":"info","message":"POST /repos/elastic/kibana/issues/90706/labels - 200 in 366ms"}
View pull request: https://github.com/elastic/kibana/pull/90706

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@spalger spalger deleted the fix/do-not-ignore-metric-recording-failures branch February 9, 2021 00:34
jloleysens added a commit to jloleysens/kibana that referenced this pull request Feb 9, 2021
…timeline-and-rollover-info

* 'master' of github.com:elastic/kibana: (47 commits)
  [Fleet] Use TS project references (elastic#87574)
  before/beforeEach clean up (elastic#90663)
  [Vega] user should be able to set a specific tilemap service using the mapStyle property (elastic#88440)
  [Security Solution][Case] ServiceNow SIR Connector (elastic#88655)
  [Search Sessions] Enable extend from management (elastic#90558)
  [ILM] Delete phase redesign (rework) (elastic#90291)
  [APM-UI][E2E] use withGithubStatus step (elastic#90651)
  Add folding in kb-monaco and update some viewers (elastic#90152)
  [Grok Debugger] Changed test to wait for grok debugger container to exist to fix test flakiness (elastic#90543)
  Strongly typed EUI theme for styled-components (elastic#90106)
  Fix vega renovate label (elastic#90591)
  [Uptime] Migrate to TypeScript project references (elastic#90510)
  [Monitoring] Migrate data source for legacy alerts to monitoring data directly (elastic#87377)
  [Upgrade Assistant] Add A11y Tests (elastic#90265)
  [Time to Visualize] Adds functional tests for linking/unlinking panel from embeddable library (elastic#89612)
  [dev-utils/ship-ci-stats] fail when CI stats is down (elastic#90678)
  chore(NA): remove write permissions on Bazel remote cache for PRs (elastic#90652)
  chore(NA): move bazel workspace status from bash script into nodejs executable (elastic#90560)
  Use default ES distribution for functional tests (elastic#88737)
  [Alerts] Jira: Disallow labels with spaces (elastic#90548)
  ...

# Conflicts:
#	x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/components/timeline/timeline.tsx
#	x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/lib/absolute_timing_to_relative_timing.test.ts
#	x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/lib/absolute_timing_to_relative_timing.ts
spalger added a commit that referenced this pull request Feb 9, 2021
…90706)

Co-authored-by: spalger <spalger@users.noreply.github.com>

Co-authored-by: Spencer <email@spalger.com>
Co-authored-by: spalger <spalger@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport Deprecated - use backport:version if exact versions are needed release_note:skip Skip the PR/issue when compiling release notes Team:Operations Team label for Operations Team v7.12.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants