-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
(aws-eks): Construct Library custom resources doesn't use proxy properly #12469
Comments
@iliapolo could you take a look when have time please, could it be related to tls check, can we somehow disable it? |
@alexey-boyko
Yes, all functions should be connected to the VPC but only the one interacting with the EKS API receives the
Not sure what you mean by that, disable what? Are you able to create an EKS cluster via the proxy using the SDK? we use the |
thanks i will try it right now to test via api call, but i noticed that call is made from ProviderframeworkonEvent lambda function, which doesn't have proxy set during deploy, only OnEventHandler lambda function |
@alexey-boyko The |
@iliapolo unfortunately i can't make it work( i tried to create lambda function in vpc with code below, but anyway i got same timeout as using OnEventHandler
|
anything i can try, can i increase log level?, also i wonder why timeout to ec2-63-32-73-253.eu-west-1.compute.amazonaws.com. not eks.eu-west-1.compute.amazonaws.com, and in cloudwatch only logs from ProviderframeworkonEven |
@alexey-boyko The Do the subnets you are using have outgoing internet access? If not, you will need to configure VPC endpoints. See #12171 for more details. This will allow the function calling EKS api's to be invoked, from that point on your proxy configuration should be applied. As for the proxy configuration itself, I'm afraid I don't have any insight on this...I suggest we take it step by step by first making sure that the Thanks |
hey @iliapolo thanks a lot for answer, "Do the subnets you are using have outgoing internet access? If not, you will need to configure VPC endpoints." we don't have internet, we use proxy for accessing aws api's which don't have vpc endpoints, for rest we use vpc endpoints. I have several lambda function which use proxy to call eks api, but they use python3 and boto3, |
Makes sense, so just make sure you configure all the necessary endpoints as mentioned here. |
@iliapolo i was missing lambda and step function endpoints, now OnEventHandler lambda gets invoked but anyway i receive not verbose error(( so it hard to understand what is happening
ProviderframeworkonEvent log:
It happens anyway even if i add proxy variable or not |
@alexey-boyko
I guess that makes in case the proxy isn't being applied correctly. I did a little digging and it seems that unlike python, nodejs doesn't use any global env variables to configures proxy passes. Looks like every http client implements this independently. We might need to do add explicit support for this in the library. I'll try adding some more information soon. |
## Summary Currently when a user wants to route all of the EKS lambda's SDK requests through a proxy then they are [instructed to configure an env var named `HTTP_PROXY` or `http_proxy`](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-eks-readme.html#cluster-handler). e.g. ```ts const cluster = new eks.Cluster(this, 'hello-eks', { version: eks.KubernetesVersion.V1_21, clusterHandlerEnvironment: { 'http_proxy': 'http://proxy.myproxy.com' } }); ``` However the JS SDK [requires further configuration to enable proxy support](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/node-configuring-proxies.html). This PR: - Adds a `package.json` with the dependency 'proxy-agent' to the `cluster-resource-handler/` lambda bundle - Uses `NodeJSFunction` to install lambda dependencies and bundle. - Adds a condition that checks the environment for `HTTP_PROXY` or `http_proxy` values. If present then configures the aws-sdk to use that proxy (using `proxy-agent`). Note: I placed the `proxy-agent` in the `devDependencies` of `package.json`. If the dependency is placed in the `dependencies` section then the CDK builder [throws an error: `NPM Package cluster-resources-handler inside jsii package '@aws-cdk/aws-eks', can only have devDependencies`](https://github.com/aws/aws-cdk/blob/7dae114b7aac46321b8d8572e6837428b4c633b2/tools/pkglint/lib/rules.ts#L1332) Fixes: SIM D29159517, #12469 ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
|
## Summary Currently when a user wants to route all of the EKS lambda's `aws-sdk-js` requests through a proxy then they are [instructed to configure an env var named `HTTP_PROXY` or `http_proxy`](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-eks-readme.html#cluster-handler). e.g. ```ts const cluster = new eks.Cluster(this, 'hello-eks', { version: eks.KubernetesVersion.V1_21, clusterHandlerEnvironment: { 'http_proxy': 'http://proxy.myproxy.com' } }); ``` However the JS SDK [requires further configuration to enable proxy support](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/node-configuring-proxies.html). This PR: **The below changes have been refactored to avoid use of `NodeJsFunction`. See the PR comments below for [reasoning](#16657 (comment)) and [updated changes](#16657 (comment) - ~~Adds a `package.json` with the dependency ['http-proxy-agent'](https://github.com/TooTallNate/node-http-proxy-agent) to the `cluster-resource-handler/` lambda bundle~~ - ~~Uses `NodeJSFunction` to install lambda dependencies and bundle.~~ - Adds a condition that checks the environment for `HTTP_PROXY` or `http_proxy` values. If present then configures the aws-sdk to use that proxy (using `http-proxy-agent`). ~~Note: I placed the `http-proxy-agent` in the `devDependencies` of `package.json`. If the dependency is placed in the `dependencies` section then the CDK builder [throws an error: `NPM Package cluster-resources-handler inside jsii package '@aws-cdk/aws-eks', can only have devDependencies`](https://github.com/aws/aws-cdk/blob/7dae114b7aac46321b8d8572e6837428b4c633b2/tools/pkglint/lib/rules.ts#L1332)~~ Fixes: SIM D29159517, #12469 Tested this using squid proxy on an ec2 instance within the same VPC as the EKS cluster. ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
…ll cluster handler functions (#17200) ## Summary This PR is intended for CDK EKS users who require all traffic to be routed through a proxy. Currently if a user does not allow internet connections to the VPC without going through a proxy, then deploying an EKS cluster will result in a timeout error: ```sh Received response status [FAILED] from custom resource. Message returned: Error: 2021-10-20T14:20:47.028Z d86e3ef4-45ce-4130-988f-c4663f7f8c80 Task timed out after 60.06 seconds ``` Fixes: #12469, SIM D29159517 Related to but does not resolve: `https://github.com/aws/aws-cdk/issues/12171` ## ⚙️ Changes _Expand each list item for additional details._ <details> <summary><strong>Corrected "Cluster Handler" docs to clarify that 2 lambdas are created (<code>onEventHandler</code>, <code>isCompleteHandler</code>)</strong></summary> <br /> Our docs [currently describe the "Cluster Handler" as one Lambda function that interacts with the EKS API](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-eks-readme.html#cluster-handler). However this is not accurate. The "Cluster Handler" actually creates [two Lambdas](https://github.com/aws/aws-cdk/blob/0cabb9f2d2f50c03337cd6f35bf47fc54ada3a21/packages/%40aws-cdk/aws-eks/lib/cluster-resource-provider.ts#L69-L96) for the Custom Resource, `onEventHandler` and `isCompleteHandler`, both interact with the AWS API. </details> <details> <summary><strong>Passes the <code>clusterHandlerEnvironment</code> to both Cluster Handler Lambdas</strong></summary> <br /> The `clusterHandlerEnvironment` is the [recommended method](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-eks-readme.html#cluster-handler) of passing a proxy url (i.g. `http_proxy: 'http://my-proxy.com:3128'`) to the Cluster Handler. Currently the `clusterHandlerEnvironment` is only passed to the Cluster Handler's `onEventHandler` Lambda. [The `onEventHandler` was believed to be the only Cluster Handler Lambda that interacts with the AWS EKS API](#12469 (comment)), however this is not entirely true. Both the `onEventHandler` and `isCompleteHandler` call the AWS EKS API. Following the execution process of `isCompleteHandler` when creating an EKS cluster: 1. [`index.isComplete()` (this is the Lambda handler)](https://github.com/aws/aws-cdk/blob/0cabb9f2d2f50c03337cd6f35bf47fc54ada3a21/packages/%40aws-cdk/aws-eks/lib/cluster-resource-handler/index.ts#L48) 2. [`common.isComplete()`](https://github.com/aws/aws-cdk/blob/0cabb9f2d2f50c03337cd6f35bf47fc54ada3a21/packages/%40aws-cdk/aws-eks/lib/cluster-resource-handler/common.ts#L59) 3. [`cluster.isCreateComplete()`](https://github.com/aws/aws-cdk/blob/0cabb9f2d2f50c03337cd6f35bf47fc54ada3a21/packages/%40aws-cdk/aws-eks/lib/cluster-resource-handler/cluster.ts#L56) 4. [`cluster.isActive()`](https://github.com/aws/aws-cdk/blob/0cabb9f2d2f50c03337cd6f35bf47fc54ada3a21/packages/%40aws-cdk/aws-eks/lib/cluster-resource-handler/cluster.ts#L196) 5. [Request to EKS API](https://github.com/aws/aws-cdk/blob/0cabb9f2d2f50c03337cd6f35bf47fc54ada3a21/packages/%40aws-cdk/aws-eks/lib/cluster-resource-handler/cluster.ts#L198) (results in timeout because proxy is not used) This change allows the user to pass proxy urls as environment variables to **both** Lambdas using `clusterHandlerEnvironment`. </details> <details> <summary><strong>Renames the prop <code>onEventLayer</code> -> <code>proxyAgentLayer</code>, and provides the layer to both Cluster Handler Lambdas</strong></summary> <br /> The proxy-agent layer is now used in both `onEventHandler` and `isCompleteHandler` lambdas in order to support proxy configurations. Because of this change, i've deprecated the original `onEventLayer` and created a new prop `proxyAgentLayer` since we will now be passing this prop into more than just the `onEventHandler` Lambda. The `onEventLayer` prop was introduced [a few weeks ago (sept 24)](#16657) so it should not impact many users (if any). The prop would only be used if the user wishes to bundle the layer themselves with a custom proxy agent. This prop follows the [same user customization we allow with the kubectl handler](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-eks.Cluster.html#kubectllayer). Another suitable name for this prop could have been `clusterHandlerLayer` but I chose `proxyAgentLayer` because it represents **what** the layer is used for, instead of describing **where** it's used. This also follows the convention of the pre-existing [`kubectlLayer` prop](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-eks.Cluster.html#kubectllayer). </details> <details> <summary><strong>Adds the EKS cluster prop <code>clusterHandlerSecurityGroup</code></strong></summary> <br /> If a proxy address is provided to the Cluster Handler Lambdas, but the proxy instance is not open to the world, then the dynamic IPs of the Cluster Handler Lambdas will be denied access. To solve this, i've implemented a new Cluster prop `clusterHandlerSecurityGroup`. This `clusterHandlerSecurityGroup` prop will allow the user to pass a Security Group to both Lambda functions and the Custom Resource provider. This is very similar to how we [already allow users to pass Security Groups to the Kubectl Handler](https://github.com/aws/aws-cdk/blob/7f194000697b85deb410ae0d7f7d4ac3c2654bcc/packages/%40aws-cdk/aws-eks/lib/kubectl-provider.ts#L83) </details> ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
## Summary Currently when a user wants to route all of the EKS lambda's `aws-sdk-js` requests through a proxy then they are [instructed to configure an env var named `HTTP_PROXY` or `http_proxy`](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-eks-readme.html#cluster-handler). e.g. ```ts const cluster = new eks.Cluster(this, 'hello-eks', { version: eks.KubernetesVersion.V1_21, clusterHandlerEnvironment: { 'http_proxy': 'http://proxy.myproxy.com' } }); ``` However the JS SDK [requires further configuration to enable proxy support](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/node-configuring-proxies.html). This PR: **The below changes have been refactored to avoid use of `NodeJsFunction`. See the PR comments below for [reasoning](aws/aws-cdk#16657 (comment)) and [updated changes](aws/aws-cdk#16657 (comment) - ~~Adds a `package.json` with the dependency ['http-proxy-agent'](https://github.com/TooTallNate/node-http-proxy-agent) to the `cluster-resource-handler/` lambda bundle~~ - ~~Uses `NodeJSFunction` to install lambda dependencies and bundle.~~ - Adds a condition that checks the environment for `HTTP_PROXY` or `http_proxy` values. If present then configures the aws-sdk to use that proxy (using `http-proxy-agent`). ~~Note: I placed the `http-proxy-agent` in the `devDependencies` of `package.json`. If the dependency is placed in the `dependencies` section then the CDK builder [throws an error: `NPM Package cluster-resources-handler inside jsii package '@aws-cdk/aws-eks', can only have devDependencies`](https://github.com/aws/aws-cdk/blob/7dae114b7aac46321b8d8572e6837428b4c633b2/tools/pkglint/lib/rules.ts#L1332)~~ Fixes: SIM D29159517, aws/aws-cdk#12469 Tested this using squid proxy on an ec2 instance within the same VPC as the EKS cluster. ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
## Summary Currently when a user wants to route all of the EKS lambda's `aws-sdk-js` requests through a proxy then they are [instructed to configure an env var named `HTTP_PROXY` or `http_proxy`](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-eks-readme.html#cluster-handler). e.g. ```ts const cluster = new eks.Cluster(this, 'hello-eks', { version: eks.KubernetesVersion.V1_21, clusterHandlerEnvironment: { 'http_proxy': 'http://proxy.myproxy.com' } }); ``` However the JS SDK [requires further configuration to enable proxy support](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/node-configuring-proxies.html). This PR: **The below changes have been refactored to avoid use of `NodeJsFunction`. See the PR comments below for [reasoning](aws#16657 (comment)) and [updated changes](aws#16657 (comment) - ~~Adds a `package.json` with the dependency ['http-proxy-agent'](https://github.com/TooTallNate/node-http-proxy-agent) to the `cluster-resource-handler/` lambda bundle~~ - ~~Uses `NodeJSFunction` to install lambda dependencies and bundle.~~ - Adds a condition that checks the environment for `HTTP_PROXY` or `http_proxy` values. If present then configures the aws-sdk to use that proxy (using `http-proxy-agent`). ~~Note: I placed the `http-proxy-agent` in the `devDependencies` of `package.json`. If the dependency is placed in the `dependencies` section then the CDK builder [throws an error: `NPM Package cluster-resources-handler inside jsii package '@aws-cdk/aws-eks', can only have devDependencies`](https://github.com/aws/aws-cdk/blob/7dae114b7aac46321b8d8572e6837428b4c633b2/tools/pkglint/lib/rules.ts#L1332)~~ Fixes: SIM D29159517, aws#12469 Tested this using squid proxy on an ec2 instance within the same VPC as the EKS cluster. ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
…ll cluster handler functions (aws#17200) ## Summary This PR is intended for CDK EKS users who require all traffic to be routed through a proxy. Currently if a user does not allow internet connections to the VPC without going through a proxy, then deploying an EKS cluster will result in a timeout error: ```sh Received response status [FAILED] from custom resource. Message returned: Error: 2021-10-20T14:20:47.028Z d86e3ef4-45ce-4130-988f-c4663f7f8c80 Task timed out after 60.06 seconds ``` Fixes: aws#12469, SIM D29159517 Related to but does not resolve: `https://github.com/aws/aws-cdk/issues/12171` ## ⚙️ Changes _Expand each list item for additional details._ <details> <summary><strong>Corrected "Cluster Handler" docs to clarify that 2 lambdas are created (<code>onEventHandler</code>, <code>isCompleteHandler</code>)</strong></summary> <br /> Our docs [currently describe the "Cluster Handler" as one Lambda function that interacts with the EKS API](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-eks-readme.html#cluster-handler). However this is not accurate. The "Cluster Handler" actually creates [two Lambdas](https://github.com/aws/aws-cdk/blob/0cabb9f2d2f50c03337cd6f35bf47fc54ada3a21/packages/%40aws-cdk/aws-eks/lib/cluster-resource-provider.ts#L69-L96) for the Custom Resource, `onEventHandler` and `isCompleteHandler`, both interact with the AWS API. </details> <details> <summary><strong>Passes the <code>clusterHandlerEnvironment</code> to both Cluster Handler Lambdas</strong></summary> <br /> The `clusterHandlerEnvironment` is the [recommended method](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-eks-readme.html#cluster-handler) of passing a proxy url (i.g. `http_proxy: 'http://my-proxy.com:3128'`) to the Cluster Handler. Currently the `clusterHandlerEnvironment` is only passed to the Cluster Handler's `onEventHandler` Lambda. [The `onEventHandler` was believed to be the only Cluster Handler Lambda that interacts with the AWS EKS API](aws#12469 (comment)), however this is not entirely true. Both the `onEventHandler` and `isCompleteHandler` call the AWS EKS API. Following the execution process of `isCompleteHandler` when creating an EKS cluster: 1. [`index.isComplete()` (this is the Lambda handler)](https://github.com/aws/aws-cdk/blob/0cabb9f2d2f50c03337cd6f35bf47fc54ada3a21/packages/%40aws-cdk/aws-eks/lib/cluster-resource-handler/index.ts#L48) 2. [`common.isComplete()`](https://github.com/aws/aws-cdk/blob/0cabb9f2d2f50c03337cd6f35bf47fc54ada3a21/packages/%40aws-cdk/aws-eks/lib/cluster-resource-handler/common.ts#L59) 3. [`cluster.isCreateComplete()`](https://github.com/aws/aws-cdk/blob/0cabb9f2d2f50c03337cd6f35bf47fc54ada3a21/packages/%40aws-cdk/aws-eks/lib/cluster-resource-handler/cluster.ts#L56) 4. [`cluster.isActive()`](https://github.com/aws/aws-cdk/blob/0cabb9f2d2f50c03337cd6f35bf47fc54ada3a21/packages/%40aws-cdk/aws-eks/lib/cluster-resource-handler/cluster.ts#L196) 5. [Request to EKS API](https://github.com/aws/aws-cdk/blob/0cabb9f2d2f50c03337cd6f35bf47fc54ada3a21/packages/%40aws-cdk/aws-eks/lib/cluster-resource-handler/cluster.ts#L198) (results in timeout because proxy is not used) This change allows the user to pass proxy urls as environment variables to **both** Lambdas using `clusterHandlerEnvironment`. </details> <details> <summary><strong>Renames the prop <code>onEventLayer</code> -> <code>proxyAgentLayer</code>, and provides the layer to both Cluster Handler Lambdas</strong></summary> <br /> The proxy-agent layer is now used in both `onEventHandler` and `isCompleteHandler` lambdas in order to support proxy configurations. Because of this change, i've deprecated the original `onEventLayer` and created a new prop `proxyAgentLayer` since we will now be passing this prop into more than just the `onEventHandler` Lambda. The `onEventLayer` prop was introduced [a few weeks ago (sept 24)](aws#16657) so it should not impact many users (if any). The prop would only be used if the user wishes to bundle the layer themselves with a custom proxy agent. This prop follows the [same user customization we allow with the kubectl handler](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-eks.Cluster.html#kubectllayer). Another suitable name for this prop could have been `clusterHandlerLayer` but I chose `proxyAgentLayer` because it represents **what** the layer is used for, instead of describing **where** it's used. This also follows the convention of the pre-existing [`kubectlLayer` prop](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-eks.Cluster.html#kubectllayer). </details> <details> <summary><strong>Adds the EKS cluster prop <code>clusterHandlerSecurityGroup</code></strong></summary> <br /> If a proxy address is provided to the Cluster Handler Lambdas, but the proxy instance is not open to the world, then the dynamic IPs of the Cluster Handler Lambdas will be denied access. To solve this, i've implemented a new Cluster prop `clusterHandlerSecurityGroup`. This `clusterHandlerSecurityGroup` prop will allow the user to pass a Security Group to both Lambda functions and the Custom Resource provider. This is very similar to how we [already allow users to pass Security Groups to the Kubectl Handler](https://github.com/aws/aws-cdk/blob/7f194000697b85deb410ae0d7f7d4ac3c2654bcc/packages/%40aws-cdk/aws-eks/lib/kubectl-provider.ts#L83) </details> ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
I'm trying to test new feature related to provision within the VPC lambda functions related to EKS configuration like
ClusterHandler
. Lambdas are placed to vpc and it's great. But I got error connecting to EKS api via proxyReproduction Steps
What did you expect to happen?
successful cluster creation
What actually happened?
Proxy is fine i tested it against same endpoint ( strange that it calls ec2 and nor eks api
Environment
Other
I noticed that 5 lambda functions are created but only 1 of them "OnEventHandler" receives proxy configuration, but looks like it only one which interacts with api
This is 🐛 Bug Report
The text was updated successfully, but these errors were encountered: