-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
aws-cdk-lib/aws-lambda-nodejs: Respect the --asset-parallelism flag when bundling via local esbuild install #24456
Comments
We believe this is a super important issue. We recently did a refactoring and we now have more than 60-70 lambdas dispatched in multiple nested stacks gathered in one of our stack. Using That kills the developer experience. We have searched around asset bundling and esbuild but it seems like we have no other way than to split to top-level stacks for now, so we can mitigate those long repeated processes per lambda. Are we right? |
@fab-mindflow Unfortunately the bundling logic is not centralized and is instead a side effect of To clarify for everyone else who comes across this: the feature is limited to |
For public visibility: I've gotten feedback from internal AWS folks in out of band channels. As I understand, the CDK team is supportive of this request and have asked that I open a PR to move it forward. I will do that as soon as I am not insanely swamped with work 😅 |
Hi. I've also experimented with parallel bundling, however in a more hacky way. Tested with a stack of 100x lambda functions that only include a I'd definitely like to see this natively supported. |
I have breathing room now and made some progress on this over the weekend. My goal is to have a PR out before the end of the week. |
Just joining the conversation to share our experience, as we faced the same issue stating with around 50 functions, and maybe it's useful to others. It may be a bit of a special setup since we use Nx with its esbuild plugin to invoke CDK, but anyway, I tested it also with plain We moved the handlers into some folders with a common suffix (something like Now, we can run esbuild before invoking CDK, and put all Lambdas into a build folder in the repo, with the same tree structure as the CDK code (inside In the gist is the example configuration for Nx 1. yarn esbuild libs/**/*.fn/index.ts --bundle --platform=node --minify --sourcemap=external --outdir=dist/lambda Finally, we have a construct 1 (heavily inspired by the NodejsFunction construct 2), that will easily reconstruct the full path of the pre-bundled lambda, based on the name of the folder. esbuild takes ~2 seconds to bundle those 50 Lambda Functions, before it would take more than 1 minute. Additionally, through Nx we get caching (not that it makes that much of a difference at this point, but it's still nice to have) Footnotes |
That looks super appealing! |
@GusBuonv and others, have you considered if the AWS CDK |
@bestickley Today, I have migrated to turborepo in combination with yarn v3. Now, I bundle the code before synthesizing the CDK stack. Turborepo provides build caching, but it's not a major concern because esbuild is incredibly fast. I'm following a similar approach to @borgoat, but instead of Nx, I'm using turborepo. The folder structure also adheres to the convention Here's the bash script I use for bundling: esbuild **/*.fn/index.ts --bundle --platform=node --outdir=dist --external':@aws-sdk/*' To enable build caching, it's important to separate the lambda code into a separate package. Otherwise, the file hash will change with every CDK update. I've documented my setup in a gist, which you can find here: https://gist.github.com/NimmLor/a9ae54ef9e55e96193508a83fb50c495 |
@NimmLor, that's great. But wouldn't it be even nicer if |
@bestickley Definitely, I'd love to see parallel builds implemented in I believe that implementing build caching in CDK could pose challenges due to the multitude of factors involved in generating a hash. You may want to explore turborepo's hashing strategy for further insights. For me, the preferred approach is to organize the lambda functions in a separate package. This ensures that unnecessary dependencies are not inadvertently bundled. I have frequently encountered situations where I unintentionally included some CDK dependencies in my bundle. |
We are also experiencing immense pain from slow builds. It is exacerbated by the fact that we deploy the same solution to many environments. And the construct is not smart enough to realize this, and rebuilds the asset for each stage again and again, even though the entry handler file is the same. In the CodePipeline/CodeBuild environment we are at 8 minutes of build time now, and it keeps getting worse with each new stage (app instance) added. |
@moltar I've used the |
What does that mean tho? I think our setup is correct. |
Also, I don't like an asset-per-pipeline setup, as it has a ton of overhead. Plus our quantity of Lambas exceeds the limits of CodePipeline. |
Any news on this? |
I would love to have that feature within CDK, it takes ages to build our Lambda dependant stacks! |
yes... ran into the same problem. We are in need of a built in solution for this. |
+1 (bump) to have this feature added |
This issue has received a significant amount of attention so we are automatically upgrading its priority. A member of the community will see the re-prioritization and provide an update on the issue. |
This would be very helpful to my organization as well! |
+1 |
Describe the feature
I propose that the bundling process for NodejsFunctions be refactored such that the aws cdk toolkit can orchestrate bundling in parallel with synth, and that the toolkit performs this orchestration when the
--asset-parallelism
option is truthy.Use Case
Currently,
NodejsFunction
s inefficiently bundle synchronously during synth regardless of the--asset-parallelism
option for deployments. For heavily serverless projects with dozens of Lambdas or more, this massively slows synth and deployment time. The proposed feature will reduce deployment times and, alongside options like hotswap and watch, increase developer's iteration velocity in heavily serverless NodeJs projects.Proposed Solution
The following is intended for illustration. It is neither elegant nor, frankly, a "good" solution to this problem.
The
Bundling
class of the module can expose the interface to allow the toolkit to command parallel bundling:CDK Toolkit must set
Bundling.parallelBundling = true
to force this mode of operation. It can then callbundleInParallel
to create a promise that will resolve once all bundling operations return successfully. For example:You may have noticed that
bundleInParallel
relies on a hash map of cached bundling functions stored in the static propstaged
. This would be populated by the@aws-cdk/AssetStaging
class's invocation of thetryBundling
function returned byBundling.getLocalBundlingProvider
. When invoked byAssetStaging
,tryBundling
will capture the arguments passed into it within a new closure and cache the resulting function tostaged
. Example:Other Information
I may be able to implement this feature and open a PR, but would like feedback on alternatives to the above approach. Specifically is there a preferred way to integrate optional behavior like this with the toolkit? I assume such a direct coupling of a top level CLI command with such a specific construct is a non-starter. However, I'd be happy to be wrong about that as well.
Acknowledgements
CDK version used
2.67.0
Environment details (OS name and version, etc.)
N/A
The text was updated successfully, but these errors were encountered: