Split auto-generated importsfiles into smaller files

[trondhindenes]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

As a pycharm user, I'm noticing that the auto-generated imports.aws package from the quickstart doesnt perform auto-complete and similar. This seems to be because the imports.aws.__init__.py file size is around 17MB, which exceeds pycharm's default of 2.56MB.

It's totally doable to increase that size in pycharm's settings, but I'd still argue that an 18MB text file is rather large (there's almost half a million lines of code in that file)

Since part of the reason for considering CDK is to have the support of rich programming languages, and many people rely on full IDE's to take advantage of that support, it would be good if terraform-cdk tried to be a "good citizen" and generated imports in a way that didn't lead to overly large files.

Step to repro: Create the following cdk-tf file using python:

#!/usr/bin/env python
from constructs import Construct
from cdktf import App, TerraformStack
from imports.aws import Instance, AwsProvider


class MyStack(TerraformStack):
    def __init__(self, scope: Construct, ns: str):
        super().__init__(scope, ns)

        # define resources here
        aws_provider = AwsProvider(self, 'Aws', region='us-east-1')
        a = Instance(self, 'myinstance', 'ami-123', 't2.micro')


app = App()
MyStack(app, "hello-terraform")

app.synth()

Open the project in Pycharm, and witnes how the Instance, AwsProvider has red squiggles meaning the import is not valid.

Change the system property idea.max.intellisense.filesize (https://www.jetbrains.com/help/pycharm/tuning-the-ide.html#configure-platform-properties) to enable intellisense for the imports.aws module, but notice how slow the indexing operation is, because of the large file size.

References

[jsteinich]

@trondhindenes I have just a couple questions which may help find a solution:

1. Can you try using the prebuilt provider and see if the same issue exists there? (It's the same code, but I'm wondering if consuming as a package makes a difference)
2. Do you happen to know if that same content spread across 150-200 (~# of aws services) submodules would perform better?

[trondhindenes]

On 2) I would assume yes, since one typically only would import a small subset of those 150-200 files.

Will try 1) and report back!

[trondhindenes]

regarding 1), it's not clear to me how to use prebuilt providers in python - the examples are only for typescript, as far as I can see.

[skorfmann]

regarding 1), it's not clear to me how to use prebuilt providers in python - the examples are only for typescript, as far as I can see.

We do publish python packages as well, e.g. the AWS Provider. They should be usable as any other Python package.

[trondhindenes]

Thanks! Found it. As far as I can see, the prepackaged aws provider also consists of a single monster-file (about half a million lines). It's interesting that the source (https://github.com/terraform-cdk-providers/cdktf-provider-aws/tree/master/src) seems to be nicely separated per service, so I guess there is at least a theoretical option of avoding bunching everything together in a single file?

[skorfmann]

so I guess there is at least a theoretical option of avoding bunching everything together in a single file?

This would be something for jsii. I'm not aware of an option for the code generation to split this up into multiple files. Perhaps a good issue to add this to might be this one aws/jsii#1919

[trondhindenes]

how about iterating over the source *ts files and generate (jsii-compile) one python (sub)module per?

Note: I know very little about the internals of jsii so this may not be viable.

[skorfmann]

how about iterating over the source *ts files and generate (jsii-compile) one python (sub)module per?

I haven't verified it, but I don't think jsii is capable of doing this at the moment. I'll create an issue in jsii to track this.

[njsnx]

+10 to this issue - the generated Python being difficult to use with auto-complete/easily openable by symbol etc makes it very difficult to figure out what is going on. Between that and what seems to be a lack in documentation for Python modules for each provider, there is a lot of trial and error going on to get anything working with Python + CDK TF

[osulli]

@trondhindenes I have just a couple questions which may help find a solution:

1. Can you try using the prebuilt provider and see if the same issue exists there? (It's the same code, but I'm wondering if consuming as a package makes a difference)
2. Do you happen to know if that same content spread across 150-200 (~# of aws services) submodules would perform better?

FWIW still a huge time on the pre-built. I've no number but it was 30s+ at least.

[osulli]

Example of documentation appearing with settings below:

# idea.properties
idea.max.intellisense.filesize=2000000

---

Output

[DanielMSchmidt]

Closing this as we now split the AWS provider into smaller chunks by using namespaces

[github-actions]

I'm going to lock this issue because it has been closed for 30 days. This helps our maintainers find and focus on the active issues. If you've found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.