Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: Jest Integration Tests.src/core/server/integration_tests/saved_objects/serverless/migrations - Migration actions - serverless environment readWithPit returns a left es_response_too_large error when a read batch exceeds the maxResponseSize #167288

Closed
kibanamachine opened this issue Sep 26, 2023 · 38 comments · Fixed by #181566, #189815 or #189988
Closed

Failing test: Jest Integration Tests.src/core/server/integration_tests/saved_objects/serverless/migrations - Migration actions - serverless environment readWithPit returns a left es_response_too_large error when a read batch exceeds the maxResponseSize #167288

kibanamachine opened this issue Sep 26, 2023 · 38 comments · Fixed by #181566, #189815 or #189988
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Sep 26, 2023 •
edited
Loading

A test failed on a tracked branch

Error: Error pulling image. This is likely an issue authenticating with docker.elastic.co.
Visit https://docker-auth.elastic.co/github_auth to login.

Command failed with exit code 1: docker pull docker.elastic.co/kibana-ci/elasticsearch-serverless:latest-verified
no matching manifest for linux/amd64 in the manifest list entries
    at createCliError (/var/lib/buildkite-agent/builds/kb-n2-4-spot-4e36d6be5c9d72fa/elastic/kibana-on-merge/kibana/packages/kbn-es/src/errors.ts:14:24)
    at /var/lib/buildkite-agent/builds/kb-n2-4-spot-4e36d6be5c9d72fa/elastic/kibana-on-merge/kibana/packages/kbn-es/src/utils/docker.ts:369:25
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at maybePullDockerImage (/var/lib/buildkite-agent/builds/kb-n2-4-spot-4e36d6be5c9d72fa/elastic/kibana-on-merge/kibana/packages/kbn-es/src/utils/docker.ts:365:3)
    at setupDocker (/var/lib/buildkite-agent/builds/kb-n2-4-spot-4e36d6be5c9d72fa/elastic/kibana-on-merge/kibana/packages/kbn-es/src/utils/docker.ts:420:3)
    at runServerlessCluster (/var/lib/buildkite-agent/builds/kb-n2-4-spot-4e36d6be5c9d72fa/elastic/kibana-on-merge/kibana/packages/kbn-es/src/utils/docker.ts:590:3)
    at Cluster.runServerless (/var/lib/buildkite-agent/builds/kb-n2-4-spot-4e36d6be5c9d72fa/elastic/kibana-on-merge/kibana/packages/kbn-es/src/cluster.ts:549:28)
    at Object.start (/var/lib/buildkite-agent/builds/kb-n2-4-spot-4e36d6be5c9d72fa/elastic/kibana-on-merge/kibana/packages/core/test-helpers/core-test-helpers-kbn-server/src/create_serverless_root.ts:81:7)
    at startES (/var/lib/buildkite-agent/builds/kb-n2-4-spot-4e36d6be5c9d72fa/elastic/kibana-on-merge/kibana/packages/core/test-helpers/core-test-helpers-kbn-server/src/create_serverless_root.ts:49:35)
    at startEs (/var/lib/buildkite-agent/builds/kb-n2-4-spot-4e36d6be5c9d72fa/elastic/kibana-on-merge/kibana/src/core/server/integration_tests/saved_objects/serverless/migrations/actions.test.ts:19:28)
    at Object.<anonymous> (/var/lib/buildkite-agent/builds/kb-n2-4-spot-4e36d6be5c9d72fa/elastic/kibana-on-merge/kibana/src/core/server/integration_tests/saved_objects/migrations/group3/actions/actions_test_suite.ts:76:54)

First failure: CI Build - main
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Sep 26, 2023
@botelastic botelastic bot added the needs-team Issues missing a team label label Sep 26, 2023
@kibanamachine kibanamachine added the Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc label Sep 26, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Sep 26, 2023
@kibanamachine kibanamachine reopened this Feb 13, 2024
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine kibanamachine reopened this Apr 6, 2024
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@delanni
Copy link
Contributor

delanni commented Apr 8, 2024

/skip

delanni added a commit to delanni/kibana that referenced this issue Apr 8, 2024
@delanni
skip response-too-large test (elastic#167288)
d1e8e02
@delanni delanni mentioned this issue Apr 8, 2024
Merged
delanni added a commit that referenced this issue Apr 8, 2024
@delanni
[Tests] skip failing response-too-large test (#180254)
a3144b1 
## Summary
cc: @elastic/kibana-core 

skip response-too-large test
(#167288)
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@mistic
Copy link
Member

mistic commented Apr 8, 2024 •
edited
Loading

Skipped.

main: a3144b1
8.13: 63e9cde

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 8.13

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 8.13

kibanamachine referenced this issue Apr 25, 2024
@kibanamachine @gsoldevila
[8.14] Adjust esClient parameters to prevent flaky behaviors (#181566) (
3a0af1b 
#181676)

# Backport

This will backport the following commits from `main` to `8.14`:
- [Adjust esClient parameters to prevent flaky behaviors
(#181566)](#181566)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Gerard
Soldevila","email":"gerard.soldevila@elastic.co"},"sourceCommit":{"committedDate":"2024-04-25T09:37:37Z","message":"Adjust
esClient parameters to prevent flaky behaviors (#181566)\n\n##
Summary\r\n\r\nFixes
https://github.com/elastic/kibana/issues/152677\r\n\r\nIt seems that CI
eventually times out whilst creating SO indices for\r\ntesting
purposes.\r\nThe PR aims at preventing this by increasing that timeout
from `5.5s` to\r\n`10s`.\r\n\r\n\r\nLast 4 CI failures report the exact
same `Request timed out` error:\r\n\r\n<img width=\"1037\"
alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/25349407/8b4856f7-d667-4e95-b623-a085ab756e84\">\r\n\r\n---\r\n\r\nFixes
https://github.com/elastic/kibana/issues/167288\r\n\r\nThe PIT ids are
much longer on serverless. This causes issues with some\r\nour tests,
which were checking response sizes.\r\nPR
#180261 attempted at fixing
it,\r\nbut the parameters were not properly adjusted, because the tests
first\r\ndo a valid request, followed by an invalid
one.","sha":"75c228c4ea5eea1c6c87218382bafea827a185b9","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","release_note:skip","test-failure-flaky","backport:prev-minor","v8.15.0"],"title":"Adjust
esClient parameters to prevent flaky
behaviors","number":181566,"url":"https://github.com/elastic/kibana/pull/181566","mergeCommit":{"message":"Adjust
esClient parameters to prevent flaky behaviors (#181566)\n\n##
Summary\r\n\r\nFixes
https://github.com/elastic/kibana/issues/152677\r\n\r\nIt seems that CI
eventually times out whilst creating SO indices for\r\ntesting
purposes.\r\nThe PR aims at preventing this by increasing that timeout
from `5.5s` to\r\n`10s`.\r\n\r\n\r\nLast 4 CI failures report the exact
same `Request timed out` error:\r\n\r\n<img width=\"1037\"
alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/25349407/8b4856f7-d667-4e95-b623-a085ab756e84\">\r\n\r\n---\r\n\r\nFixes
https://github.com/elastic/kibana/issues/167288\r\n\r\nThe PIT ids are
much longer on serverless. This causes issues with some\r\nour tests,
which were checking response sizes.\r\nPR
#180261 attempted at fixing
it,\r\nbut the parameters were not properly adjusted, because the tests
first\r\ndo a valid request, followed by an invalid
one.","sha":"75c228c4ea5eea1c6c87218382bafea827a185b9"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.15.0","branchLabelMappingKey":"^v8.15.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/181566","number":181566,"mergeCommit":{"message":"Adjust
esClient parameters to prevent flaky behaviors (#181566)\n\n##
Summary\r\n\r\nFixes
https://github.com/elastic/kibana/issues/152677\r\n\r\nIt seems that CI
eventually times out whilst creating SO indices for\r\ntesting
purposes.\r\nThe PR aims at preventing this by increasing that timeout
from `5.5s` to\r\n`10s`.\r\n\r\n\r\nLast 4 CI failures report the exact
same `Request timed out` error:\r\n\r\n<img width=\"1037\"
alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/25349407/8b4856f7-d667-4e95-b623-a085ab756e84\">\r\n\r\n---\r\n\r\nFixes
https://github.com/elastic/kibana/issues/167288\r\n\r\nThe PIT ids are
much longer on serverless. This causes issues with some\r\nour tests,
which were checking response sizes.\r\nPR
#180261 attempted at fixing
it,\r\nbut the parameters were not properly adjusted, because the tests
first\r\ndo a valid request, followed by an invalid
one.","sha":"75c228c4ea5eea1c6c87218382bafea827a185b9"}}]}] BACKPORT-->

Co-authored-by: Gerard Soldevila <gerard.soldevila@elastic.co>
kpatticha pushed a commit to kpatticha/kibana that referenced this issue Apr 26, 2024
@mistic @kpatticha
skip flaky suite (elastic#167288)
89b6849
kpatticha pushed a commit to kpatticha/kibana that referenced this issue Apr 26, 2024
@gsoldevila @kpatticha
Adjust esClient parameters to prevent flaky behaviors (elastic#181566)
4f3bb9a 
## Summary

Fixes elastic#152677

It seems that CI eventually times out whilst creating SO indices for
testing purposes.
The PR aims at preventing this by increasing that timeout from `5.5s` to
`10s`.


Last 4 CI failures report the exact same `Request timed out` error:

<img width="1037" alt="image"
src="https://github.com/elastic/kibana/assets/25349407/8b4856f7-d667-4e95-b623-a085ab756e84">

---

Fixes elastic#167288

The PIT ids are much longer on serverless. This causes issues with some
our tests, which were checking response sizes.
PR elastic#180261 attempted at fixing it,
but the parameters were not properly adjusted, because the tests first
do a valid request, followed by an invalid one.
@szaffarano szaffarano mentioned this issue Jul 12, 2024
Closed
@kibanamachine kibanamachine reopened this Jul 23, 2024
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@TinaHeiligers
Copy link
Contributor

@gsoldevila new failure! Looks like we need another pass at #181566

@pgayvallet
Copy link
Contributor

Error:

Error pulling image. This is likely an issue authenticating with docker.elastic.co.
--
  | Visit https://docker-auth.elastic.co/github_auth to login.
  |  
  | Command failed with exit code 1: docker pull docker.elastic.co/kibana-ci/elasticsearch-serverless:latest-verified
  | Error response from daemon: Head "https://docker.elastic.co/v2/kibana-ci/elasticsearch-serverless/manifests/latest-verified": Get "https://docker-auth.elastic.co/auth?account=kibanamachine&scope=repository%3Akibana-ci%2Felasticsearch-serverless%3Apull&service=token-service": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

@kibanamachine kibanamachine reopened this Aug 2, 2024
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@afharo afharo mentioned this issue Aug 2, 2024
Merged
1 task
@kibanamachine kibanamachine reopened this Aug 6, 2024
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@afharo afharo mentioned this issue Aug 6, 2024
Merged
1 task
@kibanamachine kibanamachine reopened this Sep 6, 2024
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.15

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.15

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - 8.14

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - 8.14

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - 8.14

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - 8.14

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - 8.14

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - 8.14

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.14

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.14

@jloleysens jloleysens mentioned this issue Sep 13, 2024
Open
10 tasks
@gsoldevila gsoldevila closed this as not planned Won't fix, can't repro, duplicate, stale Sep 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
Projects
None yet
Milestone
No milestone
7 participants
@pgayvallet @mistic @delanni @TinaHeiligers @elasticmachine @gsoldevila @kibanamachine