Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unflake TestIntegration_UpdateRetentionPolicy #1632

Closed
jeanbza opened this issue Oct 22, 2019 · 3 comments · Fixed by #2687
Closed

unflake TestIntegration_UpdateRetentionPolicy #1632

jeanbza opened this issue Oct 22, 2019 · 3 comments · Fixed by #2687
Assignees
Labels
api: storage Issues related to the Cloud Storage API. type: cleanup An internal cleanup or hygiene concern.

Comments

@jeanbza
Copy link
Member

jeanbza commented Oct 22, 2019

https://sponge.corp.google.com/target?id=a8d868bf-56bf-4ede-972f-48bbf660a9ba&target=cloud-go-libraries/google-cloud-go/continuous/go112&searchFor=&show=ALL&sortBy=STATUS

=== RUN   TestIntegration_UpdateRetentionPolicy
--- FAIL: TestIntegration_UpdateRetentionPolicy (19.96s)
    integration_test.go:2974: integration_test.go:2508: update: googleapi: Error 429: The project exceeded the rate limit for creating and deleting buckets., rateLimitExceeded
@jeanbza jeanbza added api: storage Issues related to the Cloud Storage API. type: cleanup An internal cleanup or hygiene concern. labels Oct 22, 2019
gopherbot pushed a commit that referenced this issue Oct 22, 2019
Updates #1632

Change-Id: I78ec1c77d27e129ae4053ed0c5f10909bb3df263
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/47270
Reviewed-by: Chris Cotter <cjcotter@google.com>
@odeke-em
Copy link
Contributor

I believe the error exorcises the problem in
Error 429: The project exceeded the rate limit for creating and deleting buckets., rateLimitExceeded
because in that test we are creating and deleting a bucket on every single sub-test yet all we are doing is just updating the retention policy.
The quota for creating and deleting buckets is very low aka 1 operation every 2 seconds, as per
https://cloud.google.com/storage/quotas#buckets
Screen Shot 2020-04-17 at 5 09 29 PM
and if those subtests run for say 300 milliseconds, we'll hit the limit ASAP.

I think what we can do is:
a) Create a single bucket, on exiting the test delete it
b) On deferring each subTest, reset the retention policy to initial

That'll ensure that the retention policy conditions are set to the same level playing field for each test and then also sleep for a bit after each subTest runs since the quota for bucket updates is 1 per second.

@odeke-em
Copy link
Contributor

I've mailed https://code-review.googlesource.com/c/gocloud/+/55031 to deflake this test.

@tritone tritone assigned tritone and odeke-em and unassigned frankyn and tritone Apr 20, 2020
gopherbot pushed a commit that referenced this issue Apr 28, 2020
…ucket use"

This reverts commit fff13f4.

Reason for revert: This integration test consistently fails in Kokoro when re-enabled. Need to debug failure before merging again.

=== RUN   TestIntegration_UpdateRetentionPolicy
    TestIntegration_UpdateRetentionPolicy: integration_test.go:2641: input: <nil>
        got=-, want=+:
          (*storage.RetentionPolicy)(
        - 	nil,
        + 	&{RetentionPeriod: s"1m0s"},
          )
--- FAIL: TestIntegration_UpdateRetentionPolicy (7.98s)

Updates #1959
Updates #1632

Change-Id: I550a971dcd569f45ace508f56dc928ab38f2a19d
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/55410
Reviewed-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Cody Oss <codyoss@google.com>
@tritone
Copy link
Contributor

tritone commented Apr 28, 2020

Fix didn't work correctly in Kokoro:

=== RUN   TestIntegration_UpdateRetentionPolicy
    TestIntegration_UpdateRetentionPolicy: integration_test.go:2641: input: <nil>
        got=-, want=+:
          (*storage.RetentionPolicy)(
        - 	nil,
        + 	&{RetentionPeriod: s"1m0s"},
          )
--- FAIL: TestIntegration_UpdateRetentionPolicy (7.98s)

Reverted and re-opening this issue.

@tritone tritone reopened this Apr 28, 2020
@AlisskaPie AlisskaPie self-assigned this Jul 8, 2020
tritone added a commit to tritone/google-cloud-go that referenced this issue Aug 5, 2020
This test was skipped b/c of a rate limit issue, but we
were able to waive this rate limit for the testing project
via an internal change.

Fixes googleapis#1632
@tbpg tbpg closed this as completed in #2687 Aug 5, 2020
tbpg pushed a commit that referenced this issue Aug 5, 2020
This test was skipped b/c of a rate limit issue, but we
were able to waive this rate limit for the testing project
via an internal change.

Fixes #1632
tritone added a commit to tritone/google-cloud-go that referenced this issue Aug 25, 2020
This test was skipped b/c of a rate limit issue, but we
were able to waive this rate limit for the testing project
via an internal change.

Fixes googleapis#1632
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the Cloud Storage API. type: cleanup An internal cleanup or hygiene concern.
Projects
None yet
5 participants