Automatically guess EC2 endpoint based on metadata

[dadoonet]

While I was working on #27464, I came to this part of the AWS SDK code:

    /**
     * Sets the endpoint configuration (service endpoint & signing region) to be used for requests. If neither region {@link #setRegion(String)}
     * or endpoint configuration are explicitly provided in the builder the {@link #DEFAULT_REGION_PROVIDER} is consulted.
     *
     * <p><b>Only use this if using a non-standard service endpoint - the recommended approach for configuring a client is to use {@link #setRegion(String)}</b>
     *
     * @param endpointConfiguration The endpointConfiguration to use
     */
    public final void setEndpointConfiguration(EndpointConfiguration endpointConfiguration) {
        withEndpointConfiguration(endpointConfiguration);
    }

Important part is:

Only use this if using a non-standard service endpoint - the recommended approach for configuring a client is to use setRegion(String)

Which basically means that we should revisit the decision we made in #22758 and remove endpoint and reintroduce region.

Creating a AWS EC2 client would be as simple as:

        AmazonEC2ClientBuilder builder = AmazonEC2ClientBuilder.standard()
            .withRegion(region)
            .withCredentials(buildCredentials(logger, settings))
            .withClientConfiguration(buildConfiguration(logger, settings));

Even better, we can actually retrieve automatically the region if not explicitly set from the metadata instance by calling:

String region = new InstanceMetadataRegionProvider().getRegion();

Which would simplify even more the usage of the plugin as people won't normally have to define anything but:

discovery.zen.hosts_provider: ec2

Authentication is done using IAM Role credentials by default.

@rjernst @tlrx thoughts?

[dadoonet]

@rjernst shared his thoughts in #27925 (comment)

Quoting here:

Part of the reason we moved away from region was it required bumping the aws client dep whenever a new region was added. By setting endpoint, we avoid that. For this reason, I do not think we should go back to setting region.

So here is my new proposal:

• endpoint is kept as is.
• We don't add region back
• But if endpoint is not set, we try to automatically find it based on the metadata of the running instance.

[rjernst]

But if endpoint is not set, we try to automatically find it based on the metadata of the running instance.

I'm fine with this. It matches the behavior with repository-s3 of auto finding the region based on bucket name (which the s3 client does for us).

[sihil]

👍It would be great to have this.

If users don't typically have to explicitly configure either the region or endpoint then it makes little difference which can be configured in advanced scenarios.

[sihil]

It's worth noting that it's non-trivial to automatically generate the endpoint from the region (Chinese regions use com.cn). I'd suggest something like this:

• Get region from instance metadata
• Try to look up region endpoint in the SDK
• If it couldn't be looked up then try to guess it (but log a warning that it might not be right)

Alternatively, it might be worth changing the way the client is built such that it uses the built in logic to automatically use the local region.

[GoodMirek]

@sihil In order to obtain the EC2 endpoint, I propose to use AWS API action DescribeRegions with its RegionName parameter set to the region string obtained from EC2 metadata.

Looking up the endpoint in the SDK can be backup option, if e.g. instance profile does not permit the AWS API action. Although I prefer to make ability to call the action DescribeRegions a mandatory requirement, as there are other AWS API actions required to make this plugin working. Guessing endpoint using SDK seems less reliable e.g. once a new region is added.

[reidca]

I came across this problem when creating a new Elasticsearch cluster using a CloudFormation stack. Took me a couple of hours debugging as the behaviour was contrary to the documentation and, in my opinion counter-intuitive.

I am aiming for 100% automation of building and configuring the Elasticsearch cluster and I did not want to hard code the endpoint because we deploy our resources to multiple regions and this was likely to get messy.

I ended up writing a CloudFormation custom resources backing off to a lambda function that is called when the stack is provisioning to populate the endpoint in the elasticsearch.yml configuration file. This function uses the DescribeRegions API call mentioned by @GoodMirek above. Whilst it works well for me it requires a lot of work for the consumer.

It would be a great idea to see the Endpoint being an optional parameter that can be overridden if required but if not set using a combination of EC2 metadata and the DescribeRegions API call to automatically get the endpoint.

[bnm22]

+1

[joeykhashab]

+1, right now to support multi region installations I have to add custom logic to lookup the correct endpoint and then change this parameter during startup of the ec2 instance.

[noaginzbursky]

+1

[yogevyuval]

+1

[lpkirby]

I just ran across this issue because I was troubleshooting why discovery wasn't working in a new region. I spent too much time troubleshooting this because I didn't notice in the documentation that discovery.ec2.endpoint had a default value and my test instances happened to be started in us-east-1 (the default). I thought it was getting the region from the ec2 instance where elasticsearch was running.

I would argue there shouldn't be a default value here. The plugin should put an error in the logs that the endpoint isn't set until it can be guessed automatically.

[arteam]

This is not needed anymore since the AWS SDK guesses the region internally when AmazonEC2 gets built. See https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/client/builder/AwsClientBuilder.java#L458.

AwsClientBuilder uses the DefaultAwsRegionProviderChain that tries to lookup the region from different sources and set its via a setRegion call.