Increased memory usage in 1.24.0 causing out-of-memory #994

invidian · 2020-03-15T23:09:16Z

It seems that after upgrading golangci-lint for my project to 1.24.0, Travis CI is no longer able to lint the project, as it fails with out-of-memory errors: https://travis-ci.org/github/flexkube/libflexkube/jobs/662847029#L1086

After reverting the same PR to 1.23.8, linting passes: https://travis-ci.org/github/flexkube/libflexkube/jobs/662848310?utm_medium=notification&utm_source=github_status.

PR affected by the issue: flexkube/libflexkube#82

Offending patch (now reverted):

diff --git Makefile Makefile
index bb0b33c..caa47b4 100644
--- Makefile
+++ Makefile
@@ -11,7 +11,7 @@ GORUN=$(GOCMD) run
 GOBUILD=CGO_ENABLED=$(CGO_ENABLED) $(GOCMD) build -v -buildmode=exe -ldflags $(LD_FLAGS)
 
 CC_TEST_REPORTER_ID=6e107e510c5479f40b0ce9166a254f3f1ee0bc547b3e48281bada1a5a32bb56d
-GOLANGCI_LINT_VERSION=v1.23.8
+GOLANGCI_LINT_VERSION=v1.24.0
 BIN_PATH=$$HOME/bin
 
 GO_PACKAGES=./...

Thank you for creating the issue!

Yes, I'm using a binary release within 2 latest major releases. Only such installations are supported.
Yes, I've searched similar issues on GitHub and didn't find any.
Yes, I've included all information below (version, config, etc).

Please include the following information:

Version of golangci-lint

$ golangci-lint --version
golangci-lint has version 1.24.0 built from 6fd4383 on 2020-03-15T11:38:02Z

Config file

$ cat .golangci.yml
cat: .golangci.yml: No such file or directory```

</details>

<details><summary>Go environment</summary>

```console
$ go version && go env
go version go1.14 linux/amd64
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/invidian/.cache/go-build"
GOENV="/home/invidian/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/invidian/go"
GOPRIVATE=""
GOPROXY="http://localhost:8081,https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/invidian/repos/libflexkube/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build666784723=/tmp/go-build -gno-record-gcc-switches"

Verbose output of running

$ golangci-lint run --enable-all --disable=godox,lll,funlen --max-same-issues=0 --max-issues-per-linter=0 --build-tags integration --timeout 10m --exclude-use-default=false ./... -v
INFO [config_reader] Config search paths: [./ /home/invidian/repos/libflexkube/integration /home/invidian/repos/libflexkube /home/invidian/repos /home/invidian /home /] 
INFO [lintersdb] Active 38 linters: [bodyclose deadcode depguard dogsled dupl errcheck gochecknoglobals gochecknoinits gocognit goconst gocritic gocyclo gofmt goimports golint gomnd goprintffuncname gosec gosimple govet ineffassign interfacer maligned misspell nakedret prealloc rowserrcheck scopelint staticcheck structcheck stylecheck typecheck unconvert unparam unused varcheck whitespace wsl] 
INFO [lintersdb] Active 38 linters: [bodyclose deadcode depguard dogsled dupl errcheck gochecknoglobals gochecknoinits gocognit goconst gocritic gocyclo gofmt goimports golint gomnd goprintffuncname gosec gosimple govet ineffassign interfacer maligned misspell nakedret prealloc rowserrcheck scopelint staticcheck structcheck stylecheck typecheck unconvert unparam unused varcheck whitespace wsl] 
INFO [loader] Go packages loading at mode 575 (compiled_files|exports_file|files|imports|name|deps|types_sizes) took 607.943833ms 
INFO [runner/filename_unadjuster] Pre-built 0 adjustments in 5.466128ms 
INFO [runner/unused/goanalysis] analyzers took 0s with no stages 
INFO [runner/goanalysis_metalinter/goanalysis] analyzers took 0s with no stages 
INFO [runner] Issues before processing: 12, after processing: 0 
INFO [runner] Processors filtering stat (out/in): filename_unadjuster: 12/12, path_prettifier: 12/12, skip_files: 12/12, autogenerated_exclude: 12/12, exclude-rules: 12/12, nolint: 0/12, skip_dirs: 12/12, cgo: 12/12, identifier_marker: 12/12, exclude: 12/12 
INFO [runner] processing took 6.995723ms with stages: nolint: 6.544171ms, identifier_marker: 206.175µs, path_prettifier: 120.378µs, autogenerated_exclude: 65.971µs, skip_dirs: 44.667µs, cgo: 4.006µs, max_same_issues: 3.183µs, uniq_by_line: 1.573µs, filename_unadjuster: 1.453µs, max_from_linter: 1.038µs, diff: 984ns, skip_files: 677ns, source_code: 435ns, path_shortener: 415ns, max_per_file_from_linter: 225ns, exclude: 198ns, exclude-rules: 174ns 
INFO [runner] linters took 238.006384ms with stages: unused: 219.300356ms, goanalysis_metalinter: 11.516494ms 
INFO File cache stats: 0 entries of total size 0B 
INFO Memory: 10 samples, avg is 91.3MB, max is 138.3MB 
INFO Execution took 858.2966ms

The text was updated successfully, but these errors were encountered:

1.24.0 causing out-of-memory see [1] so for now pin the version to 1.23.8 [1] - golangci/golangci-lint#994

in golangci-lint 1.24.0 causing out-of-memory see [1] so for now pin the version to 1.23.8 [1] - golangci/golangci-lint#994

Limit golangci-lint version to 1.23.8 due to memroy issue[1]. In addition its generaly recommended to fix the linter version to avoid unexpectedly failing on the project in case of new checks that are added. [1] golangci/golangci-lint#994

…1106) * Roll back linter version to get around memory consumption problems * Wrong go.mod

v1.24.0 has memory usage regressions: golangci/golangci-lint#994

* fix makefile: explicitly set bash as shell; tune lint. * unquote SHELL. * downgrade golangci-lint to v1.23.6. v1.24.0 has memory usage regressions: golangci/golangci-lint#994

aeneasr · 2020-03-21T16:22:31Z

We are also experiencing this issue.

ldez · 2020-03-22T21:28:40Z

I think that is related to #987

invidian · 2020-03-23T07:32:06Z

I think that is related to #987

Why you think so?

Summary: v1.24 version of golangci-lint gets oomed killed during execution on circleci. see golangci/golangci-lint#994 for more info. Reviewed By: naor9991 Differential Revision: D20592270 fbshipit-source-id: 5a8b5f90555f18abfd0319615e9aaec1e7473ff4

ldez · 2020-03-23T13:00:12Z

Because there are some changes inside staticcheck about ssa.

ldez · 2020-03-25T08:50:17Z

friendly ping @jirfag

alethenorio · 2020-03-27T16:47:20Z

Hitting this issue constantly after upgrading to 1.14 and 1.24.0

StevenACoffman · 2020-04-17T18:28:28Z

The latest release of staticcheck is still 2020.1.3 as of this writing.

From comment:

Unused is the only linter that eats a ton of memory because it can't work incrementally. You can disable it. You can make an issue in staticcheck GitHub repository about it: I think its optimization is possible.

Past closed issues:

unused: significant performance and memory usage regression between 2017.2.2 and 2019.1 dominikh/go-tools#394
Investigate porting to go/analysis dominikh/go-tools#376

My understanding is that there is no open issue for the memory usage of unused in staticcheck.

I wonder if it might be a good idea to warn people who have unused enabled in golangci-lint until it is better optimized in staticcheck.

ernado · 2020-04-20T14:51:16Z

Hey, I'm very sorry for lack of response :(

You are right, this should be related to unused linter and staticcheck.
I will try to investigate how to fix that. I've hoped that switching from patched staticcheck to upstream will not brake things, but meh.

Current workaround is to disable unused linter.

Any PR that will help to mitigate this issue will be welcomed and reviewed ASAP.

Also, please ping me instead of jirfag.

johanbrandhorst · 2020-04-20T14:53:05Z

I believe @dominikh has been recently working on improving the footprint of his tools.

dominikh · 2020-04-20T15:06:54Z

I believe @dominikh has been recently working on improving the footprint of his tools.

That is true. However, the vast majority of the footprint lies in how the analyses are run, i.e. the staticcheck command or, in this case, golangci-lint. Though I am working on related changes in unused to aid the runner, which would also benefit golangci-lint.

StevenACoffman · 2020-04-20T20:37:56Z

Would emitting a warning message whenever the linter is started while unused is enabled cut down on end user confusion (and bug reports) until a better long term solution is in place?

ernado · 2020-04-20T21:14:14Z

Generating warnings for each unused linter run can be annoying, but seems like it is the only option until OOM is fixed.

Feel free to send PR ⚡

ernado · 2020-04-21T22:35:30Z

Also it is related to #898

Hotfix for `failed prerequisites` errors. Actual cause is still unknown. Revert: #987 Ref: #994, #995, #1011

ernado · 2020-04-22T23:11:40Z

I had to downgrade staticcheck to stabilize project.
Please try v1.25.0 version.

Also please don't blame staticcheck for those issues, they can be caused by the way we are using it.
Will update back only after fixing those issues (on any side).

ernado · 2020-04-25T15:00:44Z

Closing for now.

Please open new issue for any regression.

The primary improvement is in early clearing of analyzed package's TypeInfo, facts, etc for whole program analyzers (`unused`). Clear it when it becomes unused and GC collects them early. Initially this clearing was performed for all analyzers except `unused`. Update staticcheck from v0.0.1-2019.2.3 to v0.0.1-2020.1.4 Also in this commit: * speed up loading packages from export data (2.5s -> 2.1s for std) by not using mutex for export data since it was allowed in x/tools#07722704da13 * make an order of execution of linters stable * update renameio and robustio * use robustio in caching Relates: #987, #994, #995, #1011

jirfag · 2020-05-03T09:14:20Z

Hi,
I've reduced memory usage when running unused in #1063. The problem was in how our loader and runner clears unused fields.

Memory usage on flexkube dropped from 10 GB to 1 GB.

The primary improvement is in early clearing of analyzed package's TypeInfo, facts, etc for whole program analyzers (`unused`). Clear it when it becomes unused and GC collects them early. Initially this clearing was performed for all analyzers except `unused`. Update staticcheck from v0.0.1-2019.2.3 to v0.0.1-2020.1.4 Also in this commit: * speed up loading packages from export data (2.5s -> 2.1s for std) by not using mutex for export data since it was allowed in x/tools#07722704da13 * make an order of execution of linters stable * update renameio and robustio * use robustio in caching Relates: #987, #994, #995, #1011

Summary: v1.24 version of golangci-lint gets oomed killed during execution on circleci. see golangci/golangci-lint#994 for more info. Reviewed By: naor9991 Differential Revision: D20592270 fbshipit-source-id: 5a8b5f90555f18abfd0319615e9aaec1e7473ff4

davecheney mentioned this issue Mar 16, 2020

golangci-lint job is failing projectcontour/contour#2340

Closed

moshe010 added a commit to moshe010/ib-kubernetes that referenced this issue Mar 16, 2020

lint: downgrade golangci-lint to 1.23.8

c26bb31

1.24.0 causing out-of-memory see [1] so for now pin the version to 1.23.8 [1] - golangci/golangci-lint#994

moshe010 added a commit to moshe010/ib-kubernetes that referenced this issue Mar 16, 2020

lint: downgrade golangci-lint to 1.23.8

d355ad2

in golangci-lint 1.24.0 causing out-of-memory see [1] so for now pin the version to 1.23.8 [1] - golangci/golangci-lint#994

adrianchiris mentioned this issue Mar 17, 2020

Limit golangci-lint version k8snetworkplumbingwg/rdma-cni#13

Merged

ray-milkey referenced this issue in onosproject/onos-config Mar 17, 2020

Roll back linter version to get around memory consumption problems (#…

5c6c335

…1106) * Roll back linter version to get around memory consumption problems * Wrong go.mod

alpeb mentioned this issue Mar 17, 2020

Upgrade golangci-lint to v1.23.8 linkerd/linkerd2#4181

Merged

ray-milkey mentioned this issue Mar 17, 2020

Downgrade linters to work around memory consumption issues onosproject/onos-topo#103

Merged

raulk added a commit to testground/testground that referenced this issue Mar 18, 2020

downgrade golangci-lint to v1.23.6.

1d16706

v1.24.0 has memory usage regressions: golangci/golangci-lint#994

ray-milkey mentioned this issue Mar 18, 2020

Roll back linter version to work around memory exhaustion problems onosproject/onos-test#268

Merged

ggilmore mentioned this issue Mar 19, 2020

ci: gnu parallel set --memfree 100M sourcegraph/sourcegraph-public-snapshot#9136

Merged

anyasabo mentioned this issue Mar 19, 2020

Update linter elastic/cloud-on-k8s#2725

Merged

dtomcej mentioned this issue Apr 1, 2020

Upgrade traefik and deps to 2.2.0 traefik/mesh#480

Merged

dant24 mentioned this issue Apr 7, 2020

CircleCI 4GB Limit OOM Kills #1012

Closed

3 tasks

andy-shi88 mentioned this issue Apr 13, 2020

downgrade golangci-lint version zoobc/zoobc-core#748

Merged

1 task

jpreese mentioned this issue Apr 15, 2020

Update lint version runatlantis/atlantis#991

Merged

StevenACoffman mentioned this issue Apr 17, 2020

Homebrew version is outdated #1026

Closed

ernado added topic: memory Huge memory consumption help wanted Issue that needs help from a contributor labels Apr 20, 2020

ernado added a commit that referenced this issue Apr 22, 2020

mod: downgrade honnef.co/go/tools to v2019.2.3

6db0292

Hotfix for `failed prerequisites` errors. Actual cause is still unknown. Revert: #987 Ref: #994, #995, #1011

ernado mentioned this issue Apr 22, 2020

mod: downgrade honnef.co/go/tools to v2019.2.3 #1038

Merged

ernado added the cursed This issue is doomed. label Apr 22, 2020

ernado closed this as completed Apr 25, 2020

jirfag mentioned this issue May 3, 2020

reduce mem. usage of unused and update staticcheck #1063

Merged

dtomcej mentioned this issue May 5, 2020

Upgrade golangci-lint version to v1.26.0 traefik/mesh#550

Closed

StevenACoffman mentioned this issue May 7, 2020

Ongoing GOLANGCI-LINT timeouts failing PRs and breaking nightly release tektoncd/plumbing#241

Closed

raulk mentioned this issue May 12, 2020

Staticcheck Should Be Considered For "All" Repository CI Processes libp2p/go-libp2p#866

Closed

nicks mentioned this issue Aug 25, 2020

give more RAM to the build-linux job tilt-dev/tilt#3726

Merged

ldez removed the help wanted Issue that needs help from a contributor label Oct 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Increased memory usage in 1.24.0 causing out-of-memory #994

Increased memory usage in 1.24.0 causing out-of-memory #994

invidian commented Mar 15, 2020

aeneasr commented Mar 21, 2020

ldez commented Mar 22, 2020

invidian commented Mar 23, 2020

ldez commented Mar 23, 2020

ldez commented Mar 25, 2020

alethenorio commented Mar 27, 2020

StevenACoffman commented Apr 17, 2020

ernado commented Apr 20, 2020

johanbrandhorst commented Apr 20, 2020

dominikh commented Apr 20, 2020

StevenACoffman commented Apr 20, 2020

ernado commented Apr 20, 2020

ernado commented Apr 21, 2020

ernado commented Apr 22, 2020 •

edited

Loading

ernado commented Apr 25, 2020

jirfag commented May 3, 2020

Increased memory usage in 1.24.0 causing out-of-memory #994

Increased memory usage in 1.24.0 causing out-of-memory #994

Comments

invidian commented Mar 15, 2020

aeneasr commented Mar 21, 2020

ldez commented Mar 22, 2020

invidian commented Mar 23, 2020

ldez commented Mar 23, 2020

ldez commented Mar 25, 2020

alethenorio commented Mar 27, 2020

StevenACoffman commented Apr 17, 2020

ernado commented Apr 20, 2020

johanbrandhorst commented Apr 20, 2020

dominikh commented Apr 20, 2020

StevenACoffman commented Apr 20, 2020

ernado commented Apr 20, 2020

ernado commented Apr 21, 2020

ernado commented Apr 22, 2020 • edited Loading

ernado commented Apr 25, 2020

jirfag commented May 3, 2020

ernado commented Apr 22, 2020 •

edited

Loading