not reading aws environmental variables

[ekimia]

So it fails, given ~/.aws/credentials doesn't exist. My build:
https://travis-ci.org/zerofinancial/relay/builds/269878346

**Rome version:** v0.12.0.31
**OS and version:** [Enter OS name and version here]

[tmspzz]

Thanks, I'll look into it

[ekimia]

thanks!

[tmspzz]

@ekimia this works for me if I do

export AWS_ACCESS_KEY_ID=<some access key>
export AWS_SECRET_ACCESS_KEY=<some secret>

Debug tip: Try to echo those two variables from inside your bash script. Seems to me like they are not visible to subprocesses of the shell that is launching rome.

Also, just a tip for your Romefile and travis config:

• avoid downloading rome every build. You can place rome in a cached directory on travis.
• use a local cache for rome and add that to the travis cached directories too.

See https://docs.travis-ci.com/user/caching/

[ekimia]

update on this. Travis is setup properly and the variables do exist in my build.sh script. As a workaround I generated the .aws/config file programmatically

[tmspzz]

@ekimia but do they exist in compile.sh ?

[ekimia]

yup!

[tmspzz]

@ekimia ok Thanks. The problem then seems to be much deeps than Rome.

I'll try to make a minimal experiment and see. Thanks for your understanding and for finding a workaround in the mean time.

[ekimia]

no problem, let me know if you need help

[tmspzz]

@itsthejb

Unless you're trying to use the "engine" feature, Rome doesn't launch any child process. It's using system provided access to env variables via http://hackage.haskell.org/package/base-4.12.0.0/docs/System-Environment.html#v:lookupEnv

I doubt this is bug in a core Haskell library. Rather something is interfering with your VM's env variables. On Travis for example reading env works just fine.

[itsthejb]

Debugging on a BuddyBuild remote session:

export AWS_REGION=eu-central-1
export AWS_ACCESS_KEY_ID=keyblahblah
export AWS_SECRET_ACCESS_KEY=secretblahblah

Then, rome download --skip-local-cache results in The AWS Access Key Id you provided does not exist in our records. for all dependencies

Note that I'm currently using v0.20.0.56 both locally and CI. Latest release didn't work locally

[tmspzz]

try AWS_REGION= ... AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... rome ...

[tmspzz]

Also, if you are using Minio instead of S3 you need AWS_ENDPOINT

If you use a custom profile you also need AWS_PROFILE

[itsthejb]

We aren't using Minio, that was my error since I found that issue in the issue list first.

Interesting discoveries, using version v0.20.0.56 on BuddyBuild:

AWS_REGION=eu-central-1 \
AWS_ACCESS_KEY_ID=foo \
AWS_SECRET_ACCESS_KEY=bar \
rome list

List dependencies that are in the bucket... whereas:

AWS_REGION=eu-central-1 \
AWS_ACCESS_KEY_ID=foo \
AWS_SECRET_ACCESS_KEY=bar \
rome download

Complains about the key. Currently the bucket is (even) completely public (for testing purposes!), so this is strange

[tmspzz]

Try running this on CI: https://mega.nz/#!ik1xVArD!7mq-Dtzb07yMZccNSc6rwjYB_enRw6t8XVxDNWEqcWY

Source Code

module Main where

import System.Environment

main :: IO ()
main = do
  aws_region <- lookupEnv "AWS_REGION"
  aws_id <- lookupEnv "AWS_ACCESS_KEY_ID"
  aws_secret <- lookupEnv "AWS_SECRET_ACCESS_KEY"
  print aws_region
  print aws_id
  print aws_secret

[itsthejb]

Can check that in a moment. However, more relevantly, perhaps; we use a wrapper around aws CLI here, which writes ~/.aws/credentials with the following keys:

[default]
aws_access_key_id = ...
aws_secret_access_key = ...
aws_session_token = ...
aws_security_token = ...

I suspect for this to work I potentially need to pass all of them...? Also, please forgive my relative AWS ignorance, not my speciality 😢

[itsthejb]

^ Got it now, I think! I just need to pass all of those as env vars:

• AWS_REGION
• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_SESSION_TOKEN
• AWS_SECURITY_TOKEN

Generally caused by my ignorance. Perhaps a docs improvement would be to list all of those keys as potential candidates, and something along the lines of; "if these are listed in your profile in credentials, you need to pass them all to Rome"?

[tmspzz]

I have no idea what AWS_SESSION_TOKEN or AWS_SECURITY_TOKEN are doing. I doubt I'm reading them on the latest version.

[itsthejb]

Now confirmed that I have something basically working, although not enough to exactly confirm what the issue is. I'll update again when I know. Other minor issue I'm now seeing is Rome produces no output when running via the build server. I also checked redirecting stderr to stdout to see if BuddyBuild only outputs stdout (seems unlikely). Could there be something sensitive that might prevent Rome producing output? Currently makes it quite hard to work out what's going on

[tmspzz]

No, there is no redirection of std input or output or error. You can use -v for force more verbose info