Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pipelines: additionalInputs documentation is lacking in detail #30188

Open
HansFalkenberg-Visma opened this issue May 13, 2024 · 4 comments
Open
Labels
@aws-cdk/pipelines CDK Pipelines library documentation This is a problem with documentation. effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p2

Comments

@HansFalkenberg-Visma
Copy link

HansFalkenberg-Visma commented May 13, 2024

Describe the issue

We have one repository with tooling and another repository with application code to be analyzed. This seems to fit with using toolingRepo as the main input and appRepo as an additionalInput. However, the documentation does not have enough detail to set this up correctly without trial and failure.

At first attempt I got the error message

additionalInputs: "../app-repo" must not exist yet. If you want to merge multiple artifacts, use a "cp" command.

Google led me to #17224 which is unhelpful, because the error message was actually true: The sibling directory app-repo did indeed exist.

It turns out CodePipeline action provider "GitHub (Version 2)" and CodePipeline action input artifacts works together to put the repository source code in a sibling directory named the same as the repository. So the documentation may mention something like:

The construct works by making the specified directory into a symlink to the output file set. Note that the output file set may already exist as a sibling directory. Eg. source code can exist with the repository name as a sibling directory name, in which case symlink creation will fail and you need to use a different sibling directory name.

Once I had understood the above, my next task failed because a file was not found. It appears that when there are multiple source repositories, any step that does not explicitly set its input will just get one of them, including ones used in additionalInputs. So the documentation might make it clear:

When you have specified a CodePipelineSource as an additional input, CDK will no longer know which source to use as the default for other steps. You must explicitly specify input for all steps in the pipeline.

Links

https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.pipelines-readme.html#additional-inputs
https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.pipelines.ShellStepProps.html#additionalinputs

@HansFalkenberg-Visma HansFalkenberg-Visma added documentation This is a problem with documentation. needs-triage This issue or PR still needs to be triaged. labels May 13, 2024
@github-actions github-actions bot added the @aws-cdk/pipelines CDK Pipelines library label May 13, 2024
@HansFalkenberg-Visma
Copy link
Author

HansFalkenberg-Visma commented May 13, 2024

Here's an example that showcases the two issues above. The documentation's example could be changed to incorporate elements from this example.

This example also illustrates that it is not necessary to have all source code repositories as input for the synth step. A CodePipelineSource used in any step will be part of the Source stage.

    const toolingRepo = pipelines.CodePipelineSource.connection('my-org/tooling-repo', 'main', {
      connectionArn: codeStarArn,
    });
    const appRepo = pipelines.CodePipelineSource.connection('my-org/app-repo', 'main', {
      connectionArn: codeStarArn,
    });

    const pipeline = new pipelines.CodePipeline(this, 'Pipeline', {
      synth: new pipelines.ShellStep('Synth', {
        input: toolingRepo,
        commands: ['npm ci', 'npm run synth'],
      }),
    });

    pipeline.addWave('Analyze', {
      post: [
        new pipelines.CodeBuildStep('AnalyzeStep', {
          input: toolingRepo,
          additionalInputs: {
            // NOT '../app-repo'
            ['../target']: appRepo,
          },
          commands: [
            'toolPath=$PWD/tool',
            // same as key in `additionalInputs`
            'cd ../target',
            'cat app-repo-file.json | "$toolPath"',
          ],
        }),
      ],
    });

    pipeline.addWave('AnotherStage', {
      post: [
        new pipelines.CodeBuildStep('AnotherStep', {
          // might default to `appRepo` if not specified
          input: toolingRepo,
          commands: [
            // might be obvious, but: ../app-repo does not exist
            'tool --whatever'
          ],
        }),
      ],
    });

@pahud
Copy link
Contributor

pahud commented May 16, 2024

Thank you for bringing this to our attention. Yes we should improve the doc to get it clarified. Would you like to submit a PR for this?

@pahud pahud added p2 effort/medium Medium work item – several days of effort and removed needs-triage This issue or PR still needs to be triaged. labels May 16, 2024
@moelasmar moelasmar added the feature-request A feature should be added or improved. label Aug 28, 2024
@dennisvang
Copy link

There also seems to be some issue with additionalInputs and powershell, on windows images.

@dennisvang
Copy link

Thank you for bringing this to our attention. Yes we should improve the doc to get it clarified. Would you like to submit a PR for this?

@pahud Wouldn't it be fitting for the AWS engineers responsible for additionalInputs to write their own documentation?

I assume they are the ones most familiar with the idiosyncrasies of their implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/pipelines CDK Pipelines library documentation This is a problem with documentation. effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p2
Projects
None yet
Development

No branches or pull requests

4 participants