Bigtable: Snippet setup fails due to rate-based quota.

[tseaver]

From this failiure today:

google/cloud/bigtable/cluster.py:340: in delete
    client.instance_admin_client.delete_cluster(self.name)
google/cloud/bigtable_admin_v2/gapic/bigtable_instance_admin_client.py:1179: in delete_cluster
    request, retry=retry, timeout=timeout, metadata=metadata
../api_core/google/api_core/gapic_v1/method.py:143: in __call__
    return wrapped_func(*args, **kwargs)
../api_core/google/api_core/retry.py:273: in retry_wrapped_func
    on_error=on_error,
../api_core/google/api_core/retry.py:182: in retry_target
    return target()
../api_core/google/api_core/timeout.py:214: in func_with_timeout
    return func(*args, **kwargs)
../api_core/google/api_core/grpc_helpers.py:59: in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
value = ResourceExhausted(u"Quota exceeded for quota group 'InstancesWriteGroup' and l... 'bigtableadmin.googleapis.com' for consumer 'project_number:1065521786570'.",)
from_value = <_Rendezvous of RPC that terminated with:
	status = StatusCode.RESOURCE_EXHAUS...ogleapis.com' for consumer 'project_number:1065521786570'.","grpc_status":8}"
>
    def raise_from(value, from_value):
>       raise value
E       ResourceExhausted: 429 Quota exceeded for quota group 'InstancesWriteGroup' and limit 'CLIENT_PROJECT-1d' of service 'bigtableadmin.googleapis.com' for consumer 'project_number:1065521786570'.
.nox/snippets-2-7/lib/python2.7/site-packages/six.py:737: ResourceExhausted

Including many more failures. @sduskis, could we get this particular quota / rate limit eased for our CI project?

[sduskis]

InstancesWriteGroup relates to the quota for admin writes. The service can't do more than some number of updates per time period. InstancesWriteGroup/CLIENT_PROJECT-1d is 500 per day. Basically, the tests run too often, and Cloud Bigtable isn't happy about it.

[mf2199]

@tseaver @igorbernstein2 Any prospects of where do we intend to go with this? If the limits stay, maybe this can be reflected in Best Practices or something. Any thoughts?

[tseaver]

@mf2199 Running the actual system tests less often seems to be the only cure. #8476 seems to have helped with this in the mono-repo: we see far fewer Bigtable runs which don't involve actual changes to Bigtable source in the PR. Splitting the repository should be the final remdeiation.

I'm going to close this for now, given that #8476 seems to have helped (I don't think we've seen this failure since).

[mf2199]

@tseaver 👍 Thanks!