Megalinter is very slow #3422
Labels
bug
Something isn't working
O: stale 🤖
This issue or pull request is stale, it will be closed if there is no activity
Comments
|
On which platform do you run MegaLinter ? Your system performances seems really slow |
|
We're running the job on Google Cloud Compute Engine instances, through GitLab CI runners: |
|
Do you have issues with other CI/CD jobs, especially docker based ones ? |
|
Our other CI/CD jobs all run as expected. The Megalinter jobs have been taking 20 - 30 minutes consistently for weeks. |
|
I feel your pain... indeed with such hardware it should take 5 mn not 30 :o :o Please could you share the beginning of the log like the following one, especially the last line with number of cores detected ? ----------------------------------------------------------------------------------------------------
------------------------------------ MegaLinter, by OX Security ------------------------------------
----------------------------------------------------------------------------------------------------
- Image Creation Date: 2024-03-14T20:25:19Z
- Image Revision: 8388d4830fa3234a1cd93bedd2c577e0855cd5e8
- Image Version: beta
----------------------------------------------------------------------------------------------------
The MegaLinter documentation can be found at:
- https://megalinter.io/beta
----------------------------------------------------------------------------------------------------
MegaLinter initialization (expand for details)
MegaLinter now collects the files to analyse (expand for details)
Listing all files in directory [/github/workspace], then filter with:
- File extensions: .bash, .dash, .env, .gradle, .graphql, .groovy, .gvy, .htm, .html, .json, .json5, .jsonc, .ksh, .kt, .kts, .markdown, .md, .mdx, .nf, .proto, .py, .rst, .sh, .smk, .sql, .txt, .xml, .yaml, .yml
- File names (regex): Containerfile, Dockerfile, Jenkinsfile, Makefile, Snakefile
- Excluding regex: (\.automation/test|\.automation/generated|\.venv|\.github/workflows|docs/javascripts|docs/overrides|docs/json-schemas|flavors|clj-kondo|TEMPLATES)
- Excluding .gitignored files [73]: .automation/test/salesforce/bad/.sfdx/sfdx-config.json, .automation/test/salesforce/good/.sfdx/sfdx-config.json, megalinter-reports/megalinter.log, megalinter/__pycache__/Linter.cpython-312.pyc, megalinter/__pycache__/MegaLinter.cpython-312.pyc, megalinter/__pycache__/Reporter.cpython-312.pyc, megalinter/__pycache__/__init__.cpython-312.pyc, megalinter/__pycache__/alpaca.cpython-312.pyc, megalinter/__pycache__/config.cpython-312.pyc, megalinter/__pycache__/constants.cpython-312.pyc,…(full list in DEBUG)
Kept [892] files on [2168] found files
+----MATCHING LINTERS-------------------+------------------------------------------------------------------------+----------------+------------+
| Descriptor | Linter | Criteria | Matching files | Format/Fix |
+------------+--------------------------+------------------------------------------------------------------------+----------------+------------+
| PYTHON | pylint | .py | 208 | no |
| PYTHON | black | .py | 208 | yes |
| PYTHON | flake8 | .py | 208 | no |
| PYTHON | isort | .py | 208 | yes |
| PYTHON | bandit | .py | 208 | no |
| PYTHON | mypy | .py | 208 | no |
| PYTHON | pyright | .py | 208 | no |
| PYTHON | ruff | .py | 208 | yes |
| BASH | bash-exec | .sh|.bash|.dash|.ksh | 5 | no |
| BASH | shellcheck | .sh|.bash|.dash|.ksh | 5 | no |
| BASH | shfmt | .sh|.bash|.dash|.ksh | 5 | yes |
| COPYPASTE | jscpd | | project | no |
| DOCKERFILE | hadolint | Containerfile|Dockerfile | 127 | no |
| JSON | jsonlint | .json | 19 | no |
| JSON | eslint-plugin-jsonc | .json|.json5|.jsonc | 21 | yes |
| JSON | v8r | .json | 21 | no |
| MAKEFILE | checkmake | Makefile | 1 | no |
| MARKDOWN | markdownlint | .md | 261 | yes |
| MARKDOWN | markdown-link-check | .md | 261 | no |
| MARKDOWN | markdown-table-formatter | .md | 261 | yes |
| OPENAPI | spectral | .yml|.yaml|.json | 1 | no |
| REPOSITORY | checkov | | project | no |
| REPOSITORY | git_diff | | project | no |
| REPOSITORY | grype | | project | no |
| REPOSITORY | secretlint | | project | no |
| REPOSITORY | trivy | | project | no |
| REPOSITORY | trivy-sbom | | project | no |
| REPOSITORY | trufflehog | | project | no |
| SPELL | cspell | | 682 | no |
| SPELL | lychee | .md|.mdx|.markdown|.html|.htm|.rst|.txt|.json|.jsonc|.json5|.yaml|.yml | 3[41](https://github.com/oxsecurity/megalinter/actions/runs/8288438431/job/22682948251#step:4:42) | no |
| XML | xmllint | .xml | 3 | yes |
| YAML | prettier | .yml|.yaml | 160 | yes |
| YAML | yamllint | .yml|.yaml | 161 | no |
| YAML | v8r | .yml|.yaml | [102](https://github.com/oxsecurity/megalinter/actions/runs/8288438431/job/22682948251#step:4:104) | no |
+------------+--------------------------+------------------------------------------------------------------------+----------------+------------+
Processing linters on [4] parallel cores… |
|
Definitely! Thanks 🙂 ----------------------------------------------------------------------------------------------------
------------------------------------ MegaLinter, by OX Security ------------------------------------
----------------------------------------------------------------------------------------------------
- Image Creation Date: 2023-11-19T16:08:44Z
- Image Revision: 5364b811bbe86576d832227be174ebc2aa0f5f49
- Image Version: v7.6.0
----------------------------------------------------------------------------------------------------
The MegaLinter documentation can be found at:
- https://megalinter.io/7.6.0
----------------------------------------------------------------------------------------------------
MegaLinter initialization (expand for details)
MegaLinter will analyze workspace [/builds/********]
[Activation] ACTION_ACTIONLINT has been set inactive, as subdirectory has not been found: .github/workflows (set value "any" to always activate)
[Activation] ANSIBLE_ANSIBLE_LINT has been set inactive, as subdirectory has not been found: ansible (set value "any" to always activate)
[Activation] EDITORCONFIG_EDITORCONFIG_CHECKER has been set inactive, as none of these files has been found: ['.editorconfig']
[Activation] KUBERNETES_KUBECONFORM has been set inactive, as subdirectory has not been found: kubernetes (set value "any" to always activate)
[Activation] KUBERNETES_HELM has been set inactive, as none of these files has been found: ['Chart.yml', 'Chart.yaml']
[Activation] KUBERNETES_KUBESCAPE has been set inactive, as none of these files has been found: ['Chart.yml', 'Chart.yaml']
[SemgrepLinter] Deactivated because no ruleset has been defined
[Activation] SALESFORCE_SFDX_SCANNER_APEX has been set inactive, as subdirectory has not been found: force-app (set value "any" to always activate)
[Activation] SALESFORCE_SFDX_SCANNER_AURA has been set inactive, as subdirectory has not been found: force-app (set value "any" to always activate)
[Activation] SALESFORCE_SFDX_SCANNER_LWC has been set inactive, as subdirectory has not been found: force-app (set value "any" to always activate)
[Activation] SALESFORCE_LIGHTNING_FLOW_SCANNER has been set inactive, as subdirectory has not been found: force-app (set value "any" to always activate)
[Activation] SPELL_PROSELINT has been set inactive, as none of these files has been found: ['.proselintrc', 'proselint/config.json']
[Activation] SPELL_VALE has been set inactive, as none of these files has been found: ['.vale.ini']
[Activation] SQL_SQLFLUFF has been set inactive, as none of these files has been found: ['.sqlfluff']
[Activation] SWIFT_SWIFTLINT has been set inactive, as none of these files has been found: ['.swiftlint.yml']
MARKDOWN_REMARK_LINT has been temporary disabled in MegaLinter, please use a previous MegaLinter version or wait for the next one !
Skipped linters: ACTION_ACTIONLINT, ANSIBLE_ANSIBLE_LINT, COPYPASTE_JSCPD, CSS_STYLELINT, EDITORCONFIG_EDITORCONFIG_CHECKER, JAVASCRIPT_ES, JAVASCRIPT_PRETTIER, JAVASCRIPT_STANDARD, KUBERNETES_HELM, KUBERNETES_KUBECONFORM, KUBERNETES_KUBESCAPE, MARKDOWN_REMARK_LINT, PYTHON_PYRIGHT, REPOSITORY_KICS, REPOSITORY_SEMGREP, SALESFORCE_LIGHTNING_FLOW_SCANNER, SALESFORCE_SFDX_SCANNER_APEX, SALESFORCE_SFDX_SCANNER_AURA, SALESFORCE_SFDX_SCANNER_LWC, SPELL_PROSELINT, SPELL_VALE, SQL_SQLFLUFF, SWIFT_SWIFTLINT, TYPESCRIPT_ES, TYPESCRIPT_PRETTIER, TYPESCRIPT_STANDARD
To receive reports as email, please set variable EMAIL_REPORTER_EMAIL
MegaLinter now collects the files to analyse (expand for details)
Listing all files in directory [/builds/********], then filter with:
- File extensions: , .R, .RMD, .Rmd, .bash, .bicep, .c, .c++, .cc, .cdxml, .clj, .cljc, .cljs, .coffee, .cpp, .cs, .cu, .cuh, .cxx, .dart, .dash, .edn, .env, .feature, .go, .gradle, .graphql, .groovy, .gvy, .h, .h++, .hcl, .hh, .hpp, .htm, .html, .hxx, .java, .json, .json5, .jsonc, .jsx, .ksh, .kt, .kts, .lua, .markdown, .md, .mdx, .nf, .p6, .php, .pl, .pl6, .pm, .pm6, .pp, .proto, .ps1, .ps1xml, .psd1, .psm1, .psrc, .pssc, .py, .r, .raku, .rakumod, .rakutest, .rb, .rs, .rst, .saas, .scala, .scss, .sh, .smk, .sql, .t, .tex, .tf, .tsx, .txt, .vb, .xml, .yaml, .yml
- File names (regex): Dockerfile, Jenkinsfile, Makefile, Snakefile, package\.json
- Excluding .gitignored files [89313]: .npm/_cacache/content-v2/sha1/ae/47/c1b48545cc2da33df3f42b4ce3399094197e, .npm/_cacache/content-v2/sha512/00/06/e69c41c342fe8249fad6039c17b251adccb3ed02d1d995e28e83175062fc7fb3be8593a4fa655b7bc3620657d8f552b15e0687134d0c5a5c6da846b0e160, .npm/_cacache/content-v2/sha512/00/0d/d3563fb40368ae2284245842bfb6a16306ada3fba3cee98d3325cbf32c016110520edc72f4be5b3d8562e77196c001b2b499aafba19e15d3bf48fea3ccc6, .npm/_cacache/content-v2/sha512/00/46/311fdde31853e7fdada2540c16f3b56e508911d45554281efb370305ee70530e40ebad3fc7a6dfc8ac2274417856dbb8d304371fe5963bc3a462a93330d9, .npm/_cacache/content-v2/sha512/00/5d/ebecfe5d5b12fc331c884d132539140d68e036224005693af893b054ba68cfb51a460d36699743dbd5708ee89783081769d76e8282cf6c331a928e063246, .npm/_cacache/content-v2/sha512/00/aa/5a6251e7f2de1255b3870b2f9be7e28a82f478bebb03f2f6efadb890269b3b7ca0d3923903af2ea38b4ad42630b49336cd78f2f0cf1abc8b2a68e35a9e58, .npm/_cacache/content-v2/sha512/00/e2/2049009ea62258c0fdc04671b1fb95674eed870587736c63f8e5e2f0d6faf7cc1def64b7b279dd6c0bd8676dc39cf7f4ab33233944f42b906cf8692f59a3, .npm/_cacache/content-v2/sha512/00/e8/41c722d576939a7774ee8ed0bd90bffdaf71e74e4c704398203ac62bbef64fd5c085bf31d625de57c590c5b939fe20df2f90a486e52109adddbbbc249b78, .npm/_cacache/content-v2/sha512/00/f3/0111a9e3c9b68fcd4adfa94d08314e48d00b3028c1fb93c3932ecd2fbd5e0a669131913918f3b9d1ff3a5bb933e3fa0011a29eda1946e5d75a670df5bac6, .npm/_cacache/content-v2/sha512/00/f3/341add4aa17073d2222491d07f91fbc63b00b852e245786fdcfb257bc1273babde9c1bf909d078980a7ce2bc3f52079ddc2d5d4c94714d7ceb8f69a7584b,…(full list in DEBUG)
Kept [219] files on [4661] found files
+----MATCHING LINTERS-------------------+------------------------------------------------------------------------+----------------+------------+
| Descriptor | Linter | Criteria | Matching files | Format/Fix |
+------------+--------------------------+------------------------------------------------------------------------+----------------+------------+
| BASH | bash-exec | .sh|.bash|.dash|.ksh | 2 | no |
| BASH | shellcheck | .sh|.bash|.dash|.ksh | 2 | no |
| BASH | shfmt | .sh|.bash|.dash|.ksh | 2 | no |
| DOCKERFILE | hadolint | Dockerfile | 3 | no |
| JSON | jsonlint | .json | 57 | no |
| JSON | eslint-plugin-jsonc | .json|.json5|.jsonc | 57 | no |
| JSON | v8r | .json | 57 | no |
| JSON | prettier | .json | 57 | no |
| JSON | npm-package-json-lint | .json|package\.json | project | no |
| MARKDOWN | markdownlint | .md | 13 | no |
| MARKDOWN | markdown-link-check | .md | 13 | no |
| MARKDOWN | markdown-table-formatter | .md | 13 | no |
| OPENAPI | spectral | .yml|.yaml|.json | 5 | no |
| REPOSITORY | checkov | | project | no |
| REPOSITORY | devskim | | project | no |
| REPOSITORY | dustilock | | project | no |
| REPOSITORY | git_diff | | project | no |
| REPOSITORY | gitleaks | | project | no |
| REPOSITORY | grype | | project | no |
| REPOSITORY | secretlint | | project | no |
| REPOSITORY | syft | | project | no |
| REPOSITORY | trivy | | project | no |
| REPOSITORY | trivy-sbom | | project | no |
| REPOSITORY | trufflehog | | project | no |
| SPELL | cspell | | 219 | no |
| SPELL | lychee | .md|.mdx|.markdown|.html|.htm|.rst|.txt|.json|.jsonc|.json5|.yaml|.yml | 83 | no |
| TSX | eslint | .tsx | 115 | no |
| YAML | prettier | .yml|.yaml | 13 | no |
| YAML | yamllint | .yml|.yaml | 13 | no |
| YAML | v8r | .yml|.yaml | 13 | no |
+------------+--------------------------+------------------------------------------------------------------------+----------------+------------+
Processing linters on [16] parallel cores… |
|
Ok so the 16 cores... I'm not a hardware master at all, but what is running too many stuff in parallel would generate some race conditions between the parallel processes... that are all calling git for example ^^ I'll make a PR to allow to force a maximum number of cores by defining a variable, and you'll be able to test with a beta version :) |
|
I added the new option in #3428 :) @Laurensdc please try you try again with beta version, and with |
|
Thanks a lot @nvuillam! +----SUMMARY----+--------------------------+---------------+-------+-------+--------+--------------+
| Descriptor | Linter | Mode | Files | Fixed | Errors | Elapsed time |
+---------------+--------------------------+---------------+-------+-------+--------+--------------+
| ✅ BASH | bash-exec | file | 2 | | 0 | 0.02s |
| ✅ BASH | shellcheck | list_of_files | 2 | | 0 | 0.89s |
| ✅ BASH | shfmt | list_of_files | 2 | | 0 | 0.21s |
| ✅ DOCKERFILE | hadolint | list_of_files | 3 | | 0 | 0.92s |
| ✅ JSON | eslint-plugin-jsonc | list_of_files | 57 | | 0 | 50.29s |
| ❌ JSON | jsonlint | list_of_files | 57 | | 1 | 2.25s |
| ✅ JSON | npm-package-json-lint | project | n/a | | 0 | 6.99s |
| ⚠️ JSON | prettier | list_of_files | 57 | | 1 | 9.72s |
| ✅ JSON | v8r | list_of_files | 57 | | 0 | 253.79s |
| ⚠️ MARKDOWN | markdownlint | list_of_files | 13 | | 97 | 7.05s |
| ✅ MARKDOWN | markdown-link-check | list_of_files | 13 | | 0 | 34.21s |
| ✅ MARKDOWN | markdown-table-formatter | list_of_files | 13 | | 0 | 1.52s |
| ✅ OPENAPI | spectral | file | 5 | | 0 | 46.31s |
| ❌ REPOSITORY | checkov | project | n/a | | 4 | 67.72s |
| ⚠️ REPOSITORY | devskim | project | n/a | | 2581 | 838.7s |
| ✅ REPOSITORY | dustilock | project | n/a | | 0 | 17.98s |
| ❌ REPOSITORY | gitleaks | project | n/a | | 2 | 1.71s |
| ✅ REPOSITORY | git_diff | project | n/a | | 0 | 1.21s |
| ❌ REPOSITORY | grype | project | n/a | | 80 | 85.98s |
| ✅ REPOSITORY | secretlint | project | n/a | | 0 | 267.48s |
| ✅ REPOSITORY | syft | project | n/a | | 0 | 188.98s |
| ❌ REPOSITORY | trivy | project | n/a | | 5 | 51.86s |
| ✅ REPOSITORY | trivy-sbom | project | n/a | | 0 | 48.07s |
| ✅ REPOSITORY | trufflehog | project | n/a | | 0 | 308.19s |
| ⚠️ SPELL | cspell | list_of_files | 220 | | 276 | 26.65s |
| ❌ SPELL | lychee | list_of_files | 83 | | 10 | 27.6s |
| ✅ TSX | eslint | list_of_files | 115 | | 0 | 68.38s |
| ⚠️ YAML | prettier | list_of_files | 13 | | 1 | 7.2s |
| ✅ YAML | v8r | list_of_files | 13 | | 0 | 37.25s |
| ❌ YAML | yamllint | list_of_files | 13 | | 5 | 4.89s |
+---------------+--------------------------+---------------+-------+-------+--------+--------------+Total run time: 19m28s |
|
That's much better :) MegaLinter is a tool but you own the strategy to use it :) Now is the time to get rid of linters you probably don't need , or exclude some files/folders that do not require to be checked ^^
|
|
This issue has been automatically marked as stale because it has not had recent activity. If you think this issue should stay open, please remove the |
github-actions
bot
added
the
O: stale 🤖
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
A run on our CI pipeline takes 20 - 30 minutes for our medium sized monorepo.
We've already disabled multiple linters.
Total run time is 35m.
The text was updated successfully, but these errors were encountered: