Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(stepfunctions-tasks): add AutoTerminationPolicy.IdleTimeout to EmrCreateCluster #29954

Closed

Conversation

valentinlagunes
Copy link

@valentinlagunes valentinlagunes commented Apr 24, 2024

Issue #29926

Closes #29926.

Reason for this change

Add support for AutoTerminationPolicy.IdleTimeout in emrCreateCluster task for Step Functions. This will terminate an EMR cluster automatically after it has been idle for a specified amount of time. This feature is currently already supported by CFN as shown here. The parameter is also available in the AWS Console for Step Functions. It is also available if the Step Function definition is specified entirely as ASL JSON in CDK.

Description of changes

This feature was previously supported by CDK. I essentially reverted this commit, but I added some extra checks. The feature was removed because Step Functions did not support the parameter at the time, but it has since been added.

I added the AutoTerminationPolicy as an optional parameter for the EmrCreateClusterProps object. The AutoTerminationPolicy has a required idleTimeout Duration variable. This Duration must be between 60 seconds and 7 days as specified by the CFN documentation above. I added a check for this in the constructor to throw an error if the Duration is outside of this range.

Description of how you validated changes

I added back a unit test that checks that the output JSON includes the AutoTerminationPolicy. I confirmed the existing unit tests all passed. I also confirmed the existing integration tests passed as well.

I also ran a sample integration test and deployed it in my personal account. I verified that a Step Function was created and that it had an EmrCreateCluster task. I executed the Step Function and verified that an EMR cluster was created with an idleTimeout of 2 minutes. I also verified that the EMR cluster was automatically terminated afterwards.

I added the idleTimeout parameter to an existing integration test and verified that the resulting Step Function had the correct parameters as well. I deployed that integration test in my personal account and verified the changes.

Checklist


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@github-actions github-actions bot added effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p2 beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK labels Apr 24, 2024
@aws-cdk-automation aws-cdk-automation requested a review from a team April 24, 2024 20:28
Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

@aws-cdk-automation aws-cdk-automation dismissed their stale review April 24, 2024 21:14

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

@valentinlagunes valentinlagunes marked this pull request as ready for review April 24, 2024 23:19
@godwingrs22 godwingrs22 self-assigned this Apr 24, 2024
@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: 39191b3
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@aws-cdk-automation aws-cdk-automation added the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Apr 24, 2024

if (this.props.autoTerminationPolicy !== undefined && !cdk.Token.isUnresolved(this.props.autoTerminationPolicy.idleTimeout)) {
if (this.props.autoTerminationPolicy.idleTimeout.toSeconds() < 60 || this.props.autoTerminationPolicy.idleTimeout.toSeconds() > 604800) {
throw new Error(`Idle Timeout must be between 60 and 604800 seconds but got ${this.props.autoTerminationPolicy.idleTimeout.toSeconds()}.`);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can we update the error description to explicity mention the idle timeout is for autoTerminationPolicy.

*
* @default None
*/
readonly idleTimeout: cdk.Duration;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't this property needs to be optional?

@@ -1733,3 +1733,55 @@ test('Task throws if WAIT_FOR_TASK_TOKEN is supplied as service integration patt
});
}).toThrow(/Unsupported service integration pattern. Supported Patterns: REQUEST_RESPONSE,RUN_JOB. Received: WAIT_FOR_TASK_TOKEN/);
});

test('Create Cluster with AutoTerminationPolicy', () => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also add a test to throw error when the IdleTimeout is given outside the accepted seconds?


const app = new App();

const stack = new Stack(app, 'aws-cdk-emr-create-cluster-with-on-demand-instance-fleet');

const step = new EmrCreateCluster(stack, 'EmrCreateCluster', {
releaseLabel: 'emr-5.36.1',
releaseLabel: 'emr-6.14.0',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason on why we need to update the version? As from the readme, it is mentioned emr release label above 5.30.0 should work.

*/
export function AutoTerminationPolicyPropertyToJson(property: EmrCreateCluster.AutoTerminationPolicyProperty) {
return {
IdleTimeout: cdk.numberToCloudFormation(property.idleTimeout.toSeconds()),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IdleTimeout is an optional property in cloudformation. In that case shouldn't we handle it as optional?

@pahud
Copy link
Contributor

pahud commented Jul 16, 2024

This PR has been stale for more than 2 months. I am adding a response-requested label on it. @valentinlagunes do you have any updates on this?

@pahud pahud added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Jul 16, 2024
Copy link

This PR has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

@github-actions github-actions bot added closing-soon This issue will automatically close in 4 days unless further comments are made. closed-for-staleness This issue was automatically closed because it hadn't received any attention in a while. and removed closing-soon This issue will automatically close in 4 days unless further comments are made. labels Jul 18, 2024
@github-actions github-actions bot closed this Jul 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK closed-for-staleness This issue was automatically closed because it hadn't received any attention in a while. effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p2 pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

stepfunctions-tasks: emr-create-cluster - add support for AutoTerminationPolicy.IdleTimeout
4 participants