Rate limit issue - DescribeChangeSet

[richardmilnerwatts]

I had an isolated error with an API rate limit being hit in factory. The error did not occur when I re-ran the pipeline:

CreateGenericVersionPipelineTask failed
parameters:
  category: stack
  name: **REDACTED**
  options: <FrozenOrderedDict OrderedDict()>
  source: <FrozenOrderedDict OrderedDict([('Provider', 'S3'), ('Configuration', <FrozenOrderedDict
    OrderedDict([('BucketName', '**REDACTED**-${AWS::AccountId}'), ('S3ObjectKey',
    '**REDACTED**_version1.zip')])>)])>
  stages: <FrozenOrderedDict OrderedDict()>
  version: version1

Traceback (most recent call last):

  File "/root/.pyenv/versions/3.7.10/lib/python3.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()

  File "/root/.pyenv/versions/3.7.10/lib/python3.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()

  File "/root/.pyenv/versions/3.7.10/lib/python3.7/site-packages/servicecatalog_factory/workflow/generic/create_generic_version_pipeline_task.py", line 63, in run
    StackName=friendly_uid, TemplateBody=template,

  File "/root/.pyenv/versions/3.7.10/lib/python3.7/site-packages/betterboto/cloudformation.py", line 85, in create_or_update
    StackName=stack_name

  File "/root/.pyenv/versions/3.7.10/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)

  File "/root/.pyenv/versions/3.7.10/lib/python3.7/site-packages/botocore/client.py", line 635, in _make_api_call
    raise error_class(parsed_response, operation_name)

botocore.exceptions.ClientError: An error occurred (Throttling) when calling the DescribeChangeSet operation (reached max retries: 4): Rate exceeded

[mtrampic]

Hi @richardmilnerwatts , I had it too, but i also deleted pending ChangeSets that didn't had any update.

[richardmilnerwatts]

Interesting, I didn't have to do anything other than re-run the CodePipeline for Factory. I assume it was a temporary issue which caused the retry / rate limit to be hit.

"Re-run" it is probably an acceptable solution being honest :) I'm only reporting it in case the incidence of this is higher than expected - I've never seen this before in hundreds of Factory runs.

[mtrampic]

Btw , are you running in eu-central-1 region ? Me too, for 7 AWS Organizations , first time i see this, but i see also piled up a lot of changeset faileds ( becasue thate was no change ) - which is expected normal behavior.

However i did hit another problem today , with Setuptools and Troposphere 2to3.

[richardmilnerwatts]

I've had further recurrences of this, it looks like each time we run the full Factory pipeline it creates a Change Set in the CFN which creates the CodePipelines for each product. At some point a critical threshold of "failed" (no change) Change Sets is reached, which Factory seems to iterate through causing a rate limit to be hit.

I manually deleted around 100 change sets, this is stopped the issue occuring and shaved some time of the Factory pipeline as well it seems.

[eamonnfaherty]

would it make sense if I disabled the use of changesets ?

[richardmilnerwatts]

That or some mechanism to clean up FAILED change sets after we've checked to see if there are any updates to the stack(s) in question.

Thanks,

[richardmilnerwatts]

It's this feature in the aws-cli project:

aws/aws-cli#4534

[eamonnfaherty]

change sets with zero changes cannot be executed. I have added something to delete them in better boto and will include it in the next release: https://github.com/eamonnfaherty/better-boto/releases/tag/0.43.0

[mtrampic]

That was a fast change.

[eamonnfaherty]

There is a change in 0.68.0 that will delete changesets when there are no changes. This should avoid the issue from occurring. To clean up existing change sets you can use the following script: https://gist.github.com/eamonnfaherty/5eec7ff831adbd0eaa8e922433979fec