Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bundling - multiple docker image builds #9424

Closed
2 tasks
jarikujansuu opened this issue Aug 4, 2020 · 7 comments · Fixed by #10478
Closed
2 tasks

bundling - multiple docker image builds #9424

jarikujansuu opened this issue Aug 4, 2020 · 7 comments · Fixed by #10478
Assignees
@rix0rrr
Labels
@aws-cdk/core Related to core CDK functionality effort/small Small work item – less than a day of effort feature-request A feature should be added or improved. in-progress This issue is being actively worked on. p2

Comments

@jarikujansuu
Copy link

When use same Lambda for multiple places CDK builds same docker image for each case.

Use Case

We have 2 Custom Resources that are used in multiple places. So when doing synth for some stack there are 26 or so docker image builds happening for those Lambdas. We have devs, staging and production so for each place it is used image is built 3 times (not all services have all environments so not 3 * 2 * n)

When I tried to cache created Code objects that are generated only with input of path and runtime I get errors about cross stack referencing.

Proposed Solution

Build docker image only once per runtime & path combination. Also wouldn't mind if synth for one stack wouldn't build everything for all stacks.

Other

  • 👋 I may be able to implement this feature request
  • ⚠️ This feature might incur a breaking change

This is a 🚀 Feature Request
@jarikujansuu jarikujansuu added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Aug 4, 2020
@github-actions github-actions bot added the @aws-cdk/aws-lambda Related to AWS Lambda label Aug 4, 2020
@nija-at nija-at added @aws-cdk/core Related to core CDK functionality and removed @aws-cdk/aws-lambda Related to AWS Lambda labels Aug 4, 2020
@nija-at nija-at changed the title [aws-lambda] Reuse code from same directory bundling - multiple docker image builds Aug 4, 2020
@nija-at nija-at assigned eladb and unassigned nija-at Aug 4, 2020
@nija-at
Copy link
Contributor

nija-at commented Aug 4, 2020

@eladb - sending your way since this looks related to the new ability to bundle assets.
cc @jogold

@jogold
Copy link
Contributor

jogold commented Aug 4, 2020

We need to cache during synth based on the inputs

@SomayaB SomayaB removed the needs-triage This issue or PR still needs to be triaged. label Aug 4, 2020
@maoosi
Copy link

maoosi commented Aug 12, 2020 •
edited
Loading

I do have an app with multiple stacks and I am facing a similar issue when bundling Lambda functions with the NodejsFunction class.

Even though the functions are only used in one of the stacks, the assets bundling is still happening multiple times during a single deploy (number of NodejsFunction X number of stacks). I do have 11 stacks and 3 functions, so this is quite impacting.

I'm thinking this might be linked to #8882 as well?

@eladb
Copy link
Contributor

eladb commented Aug 17, 2020

@jogold yes we can probably avoid multiple builds if the exact same input is used. Wondering though why isn't docker doing that for us?

@eladb eladb added p2 effort/small Small work item – less than a day of effort labels Aug 17, 2020
@eladb eladb assigned rix0rrr and unassigned eladb Aug 17, 2020
@jogold
Copy link
Contributor

jogold commented Aug 17, 2020

Wondering though why isn't docker doing that for us?

Because we are running multiple times not building multiple times.

@jogold
Copy link
Contributor

jogold commented Aug 18, 2020

Requires #9576

@jogold jogold mentioned this issue Aug 20, 2020
Closed
2 tasks
jogold added a commit to jogold/aws-cdk that referenced this issue Sep 22, 2020
@jogold
chore(core): cache asset hashes
a06c252 
Cache asset hashes when staging. This avoids rebundling an asset with
`AssetHashType.OUTPUT` when it is used in multiple stacks in an app.

This also removes unnecessary file system operations for other asset
hash types.

Closes aws#9424
@jogold jogold mentioned this issue Sep 22, 2020
Merged
@SomayaB SomayaB added the in-progress This issue is being actively worked on. label Sep 22, 2020
@mergify mergify bot closed this as completed in #10478 Oct 5, 2020
mergify bot pushed a commit that referenced this issue Oct 5, 2020
@jogold
chore(core): cache asset hashes (#10478)
317d9cf 
Cache asset hashes. This avoids file system and bundling operations
when the same asset with the same configuration is used in multiple
stacks in an app.

Closes #9424


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions
Copy link

github-actions bot commented Oct 5, 2020

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rix0rrr rix0rrr

Labels
@aws-cdk/core Related to core CDK functionality effort/small Small work item – less than a day of effort feature-request A feature should be added or improved. in-progress This issue is being actively worked on. p2
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

chore(core): cache asset hashes jogold/aws-cdk
7 participants
@rix0rrr @eladb @jarikujansuu @maoosi @jogold @nija-at @SomayaB