Allow usage of GOPROXY variable for go module downloads

[elivlo]

I am struggling to get ORT running with a private go proxy that serves some libraries that are not downloadable with git over HTTP. I tried to allow the environment variable with this config setting and set the variables:

ort:
  addAuthorsToCopyrights: true
  allowedProcessEnvironmentVariableNames:
    - ...
    - "GO111MODULE"
    - "GOPROXY"
    - "GONOSUMDB"

Go with go env shows this output:

GO111MODULE='on'
...
GONOSUMDB='<self-hosted-git-domain>/*'
...
GOPROXY='https://<domain>/artifactory/api/go/<repository>'

But when I run the analyzer it will not use the Proxy and tries to fetch the dependencies with Git:

16:05:06.421 [DefaultDispatcher-worker-1] ERROR org.ossreviewtoolkit.analyzer.PackageManager - GoMod failed to resolve dependencies for path 'go.mod': IOException: Running 'go list -m -json -buildvcs=false all' in '/project' failed with exit code 1:
go: downloading go1.21.9 (linux/amd64)
go: <self-hosted-git-domain>/<package1>@v1.0.2: invalid version: git ls-remote -q origin in /tmp/ort-GoMod12995763678056697516/pkg/mod/cache/vcs/558ea7aba5c64b4e8db4004f699c9cabfe805fd67f4f8e3d8992fe073c26d6e9: exit status 128:
	fatal: could not read Username for 'https://<self-hosted-git-domain>': terminal prompts disabled
Confirm the import path was entered correctly.

Is there any way to use the proxy set in the variable GOPROXY?

[sschuberth]

Maybe @fviernau or @haikoschol can assist here?

[fviernau]

Go proxy has been deliberately disabled in Ort's Go mod integration, solely to derive the VCS info entirely from the tooling. At the time of doing so (with the current Go version) this was the only way of figuring out the VCS info without re-implementing the rather complex logic from the Go tooling.

I'd look for options which work without the use of a Go proxy first. E.g. you say

are not downloadable with git over HTTP

So, how can you download sources from the corresponding VCSes? via SSH ?

[elivlo]

So, how can you download sources from the corresponding VCSes? via SSH ?

I tried using this configuration but then I need authentication with an SSH key:
git config --global url.git@<self-hosted-git-domain>:.insteadOf https://<self-hosted-git-domain>/

Then I need to add a SSH key and this configuration when I run ORT in a Gitlab-CI and this method skips the strengths of using a Go proxy. I looked into some packages that are served by our private Go proxy and found license files and much more which are needed by ort and scancode to determine the relevant information.

And I personally like the use of the Go proxy to avoid getting rate limited by global VCS like github.com 😄

[fviernau]

Then I need to add a SSH key and this configuration when I run ORT in a Gitlab-CI and this method skips the strengths of using a Go proxy.

By strength you are referring to execution time / speed (or anything else)?
The choice whether to disable the proxy was a trade-off:

• speed vs. correctness of result + much less complex implementation

..we've deliberately chosen the latter.

I looked into some packages that are served by our private Go proxy and found license files and much more which are needed by ort and scancode to determine the relevant information.

You say this to emphasize that skipping private packages is not an option?

And I personally like the use of the Go proxy to avoid getting rate limited by global VCS like github.com 😄

To my knowledge rate limits apply to the use of GitHubs REST Api, but not to normal Git operations, such as clones.
Would you have some evidence that using Go tooling with disabled proxy would run into rate limiting issues?

Then I need to add a SSH key and this configuration when I run ORT in a Gitlab-CI and this method skips the strengths of using a Go proxy.

This would already solve the entire issue, right? (Maybe an alternative could be to make the repos clonable without authentication, as they are accessible already anyway through the proxy)

[sschuberth]

• speed vs. correctness of result + much less complex implementation

..we've deliberately chosen the latter.

True. However, also see #8361 in this context.

[elivlo]

By strength you are referring to execution time / speed (or anything else)? The choice whether to disable the proxy was a trade-off:

* speed vs.  correctness of result + much less complex implementation

..we've deliberately chosen the latter.

Then I need to add a SSH key and this configuration when I run ORT in a Gitlab-CI and this method skips the strengths of using a Go proxy.

This would already solve the entire issue, right? (Maybe an alternative could be to make the repos clonable without authentication, as they are accessible already anyway through the proxy)

Yeah, the execution time was my concern in this case. And we typically run our Artifactory with more permissive permissions instead of the Gitlab instance, so the usage of the Go proxy would have been a nice option and is our state of truth in many cases. But I will have fallback to git over SSH in the current state.

---

I looked into some packages that are served by our private Go proxy and found license files and much more which are needed by ort and scancode to determine the relevant information.

You say this to emphasize that skipping private packages is not an option?

I am not aware that I can skip private packages. Please give me a quick hint on how to do that.
I think this is no option because of the nested dependencies that are used in the private packages.

---

And I personally like the use of the Go proxy to avoid getting rate limited by global VCS like github.com 😄

To my knowledge rate limits apply to the use of GitHubs REST Api, but not to normal Git operations, such as clones. Would you have some evidence that using Go tooling with disabled proxy would run into rate limiting issues?

I have looked up this and you are right!

---

A quick summary: I will do a fallback to git over SSH for Go dependencies.
A huge thank you for the quick response. That is not really common in many GitHub repositories.

[fviernau]

I am not aware that I can skip private packages. Please give me a quick hint on how to do that.
I think this is no option because of the nested dependencies that are used in the private packages.

I believe there was a misunderstanding. I was just trying to clarify why you mentioned this in the first place,
because I didn't get it.

A quick summary: I will do a fallback to git over SSH for Go dependencies.

@elivlo : Great, so it's fine to close this issue as "clarified" ?

[fviernau]

True. However, also see #8361 in this context.

@elivlo : I've wrote above assuming the discussion is around ORT's currently existing GoMod analyzer only, not around any not yet existing alternative implementations.

[sschuberth]

@elivlo : I've wrote above assuming the discussion is around ORT's currently existing GoMod analyzer only, not around any not yet existing alternative implementations.

I guess this was actually directed at me. Yes, I'm aware of that, but I wanted to point out that as least my perspective us unconstitutionally favoring accuracy over speed has relativized, and I think there's a fair use-case for favoring speed over accuracy. But let's continue any such discussion in #8361.

[fviernau]

fair use-case for favoring speed over accuracy.

In this case of GoMod the difference in implementation + maintainance cost is huge IMO, which is why
(given the development power we currently have) I believe this is very unrealistic to happen.
But sure let's not discuss this here.