Debug CI #1273

chadwhitacre · 2022-01-24T22:21:06Z

CI is failing on master. Why?

chadwhitacre · 2022-01-24T22:31:03Z

curl: (22) The requested URL returned error: 502 Bad Gateway

https://github.com/getsentry/self-hosted/runs/4929127714?check_suite_focus=true#step:4:1402

chadwhitacre · 2022-02-01T14:04:15Z

Restarting debug ...

chadwhitacre · 2022-02-01T14:22:34Z

Actually, does it fail locally? 🤔

chadwhitacre · 2022-02-01T15:02:49Z

Passes locally.

chadwhitacre · 2022-02-01T16:04:54Z

curl: (22) The requested URL returned error: 502 Bad Gateway

https://github.com/getsentry/self-hosted/runs/5023770344?check_suite_focus=true#step:4:1378

That sounds like nginx is up but the app server is down. Why does it fail so late though? That's not the first time we hit the server, is it?

BYK · 2022-02-01T16:18:40Z

@chadwhitacre here is your error: https://github.com/getsentry/self-hosted/runs/5023770344?check_suite_focus=true#step:5:81

aminvakil · 2022-02-01T16:21:18Z

@chadwhitacre here is your error: https://github.com/getsentry/self-hosted/runs/5023770344?check_suite_focus=true#step:5:81

So maybe giving #1267 a little love helps here too?
And why it has started to fail now? What has been changed?

chadwhitacre · 2022-02-01T16:24:41Z

Hm, maybe something changed in Relay? 🧐

chadwhitacre · 2022-02-01T16:27:41Z

Trying to repro locally ...

chadwhitacre · 2022-02-01T16:50:18Z

Based on #1251 (comment) I made an attempt to get our relay invocation to not spit out credentials.json properly, but it did. 🤔

$ curl -L https://github.com/docker/compose/releases/download/1.28.0/run.sh -o docker-compose-1.28.0
$
$
$
$ ./test-relay 
docker-compose version 1.28.0, build d02a7b1
docker-py version: 4.4.1
CPython version: 3.9.0
OpenSSL version: OpenSSL 1.1.1i  8 Dec 2020
Creating sentry-self-hosted_relay_run ... done
{"secret_key":"foo","public_key":"bar","id":"baz"}
$
$
$
$ cat test-relay

#!/usr/bin/env zsh

dc="./docker-compose-1.28.0"
$dc version
$dc run \
    --no-deps \
    --volume "$(pwd)/relay/config.yml:/tmp/config.yml" \
    relay --config /tmp credentials generate --stdout

BYK · 2022-02-01T16:53:47Z

@chadwhitacre let's add a cat credentials.json and an ls before that in inspect failure step to see what we generate on CI?

chadwhitacre · 2022-02-01T16:55:36Z

Ooooh, GitHub sez I won. 😜

chadwhitacre · 2022-02-01T16:57:33Z

error: could not parse json config file (file /work/.relay/credentials.json)

My hunch is that the file is malformed rather than missing. We shall see ...

chadwhitacre · 2022-02-01T16:59:46Z

Hmm ... not seeing the cat? Guess we need that ls after all. :P

https://github.com/getsentry/self-hosted/runs/5024828441?check_suite_focus=true#step:4:520

BYK · 2022-02-01T17:03:57Z

You know what, I bet it's because we mount things to /tmp. What if it was something else, like /opt or something?

BYK · 2022-02-01T17:11:25Z

install/relay-credentials.sh

@@ -20,6 +21,10 @@ if [[ ! -f "$RELAY_CREDENTIALS_JSON" ]]; then
    > "$RELAY_CREDENTIALS_JSON"


Maybe we should add that -T here right after $dcr? Since the file is there but it is empty, makes me think this is a stdout redirection related issue.

Is there a reason 07c025c won't work?

Because, read the giant comment I left there? 😁

I read it but I couldn't make sense of it.

relay generate credentials tries to read the config and the credentials even with the --stdout and --overwrite flags and then errors out when the credentials file exists but not valid JSON. We hit this case as we redirect output to the same config folder, creating an empty credentials file before relay runs.

So no shell redirect means no credentials file created before relay runs means no error parsing JSON?

Hard for me to recollect the whole memory but let me try:

We do have a config for Relay in place, either one the end-user created or the one we copied from the example file

When you run relay generate credentials even with --overwrite it tries to read both the config and the credentials (this happens only when you have the config in place)

Relay yells at us if it cannot read/parse credentials file and stops

Even when we say "ignore that, overwrite or write the output to stdout" Relay still acts like a fussy toddler

chadwhitacre · 2022-02-01T17:23:14Z

Hrmm ... the file exists but is seemingly empty. 🤔

https://github.com/getsentry/self-hosted/runs/5024926314?check_suite_focus=true#step:4:617

BYK · 2022-02-01T22:43:24Z

Okay so that only works with docker compose not docker-compose

BYK · 2022-02-01T22:44:16Z

Okay, I think the only reasonable fix here is to run docker compose pull upfront.

chadwhitacre · 2022-02-01T22:44:16Z

Okay that explains https://github.com/getsentry/self-hosted/runs/5029137872?check_suite_focus=true#step:4:163.

I can use the tail call in 5f67f7b.

BYK · 2022-02-01T22:44:41Z

Btw, relay generate credentials fails, right?

chadwhitacre · 2022-02-01T22:45:38Z

Does it? Link?

BYK · 2022-02-01T22:46:24Z

Does it? Link?

Dunno, thought you tried that too :D

Btw setting COMPOSE_INTERACTIVE_NO_CLI=1 may help with that output.

chadwhitacre · 2022-02-01T22:49:19Z

Dunno, thought you tried that too :D

Ah, you mean without --stdout? Yes, I tried that somewhere back there and yes it failed you were right. ;)

chadwhitacre · 2022-02-01T22:50:55Z

Oh! Locally I already have the image cached.

chadwhitacre · 2022-02-01T22:55:30Z

Boom! 💥

I removed the relay nightly image and I can repro locally:

relay/credentials.json ------------------
nightly: Pulling from getsentry/relay
Digest: sha256:c126666fd4f9a99388e276686eb5c81dd71a58c114c719b18a2f1d78653fd900
Status: Downloaded newer image for getsentry/relay:nightly
{"secret_key":"f66RDFJ-K-C6jHpsO7mGjuKXc_zXypdynQedNYzNmlU","public_key":"gyuqZ-p-TxVMj6xt0_aNDd8k8LBWsOmnR1RfISUH93I","id":"05775dda-6d9a-42ef-8ac8-c914ea9a3f33"}
=========================================
Creating sentry-self-hosted_relay_run ... done
error: could not parse json config file (file /work/.relay/credentials.json)
  caused by: expected ident at line 1 column 2
ERROR: 1
$

💃

chadwhitacre · 2022-02-01T22:56:30Z

yes it failed you were right

But! We can work around the "redirect creates the empty file which confuses relay" issue by redirecting to a tmp file and moving that into place. So we're good there. 👍

Last thing I want to check is Docker Compose 2 behavior ...

chadwhitacre · 2022-02-01T23:05:37Z

Alright I'm outta time. I think with some fresh eyes tomorrow it shouldn't take too long to get a PR together.

Thanks for your help @BYK! 😁 💃 🌮 👏

chadwhitacre · 2022-02-01T23:08:32Z

Oh wow do the different CI runs use the same docker cache? 🤔

chadwhitacre · 2022-02-01T23:09:55Z

Because I'm surprised that 1.28.0 is passing 🤔

chadwhitacre · 2022-02-01T23:15:40Z

Looks like we need the -T for 1 and 2.2.3+. Okay! Gonna scrap this for parts. 💃

aminvakil · 2022-02-02T07:01:44Z

Wow! Glad it was sorted out at last!

This was referenced Jan 26, 2022

Revert "Add CI compose version 1.29.2 / 2.2.3" #1272

Merged

fix: unbound variable _group in reset/dc-detect-version script (#1283) #1284

Merged

Constrain tests to failing

bc03d78

chadwhitacre force-pushed the cwlw/debug-ci branch from bb6a550 to bc03d78 Compare February 1, 2022 14:04

chadwhitacre added 2 commits February 1, 2022 10:04

Log output of failing command

351dfd9

Blerp, cat here?

42d436f

What does credentials.json contain?

8617354

ls too

b290637

BYK reviewed Feb 1, 2022

View reviewed changes

chadwhitacre added 5 commits February 1, 2022 12:30

Why can't we do this? 🤔

07c025c

Blah

59fc64c

Wrong path

2ead232

Whittle down test case

0221e3e

More parity with local

dcd1e25

chadwhitacre added 2 commits February 1, 2022 17:41

Alt fix

dac40f5

Messed something up?

9adc223

Test a stderr hypothesis

b78d45a

chadwhitacre added 3 commits February 1, 2022 17:57

Test a couple other versions

d50ba99

Hack this tho

f6235fc

Smash, bash

75a6475

chadwhitacre added 2 commits February 1, 2022 18:05

Whaaa

3c1e3f3

explicit stderr

da25f15

Test image cache across matrix

0c6c396

chadwhitacre added 2 commits February 1, 2022 18:10

What was I seeing?

6e8d37d

Experiment with -T

24b81bd

chadwhitacre closed this Feb 1, 2022

chadwhitacre deleted the cwlw/debug-ci branch February 1, 2022 23:15

chadwhitacre mentioned this pull request Feb 2, 2022

Clean up relay credentials generation #1289

Merged

github-actions bot locked and limited conversation to collaborators Feb 18, 2022

		@@ -20,6 +21,10 @@ if [[ ! -f "$RELAY_CREDENTIALS_JSON" ]]; then
		> "$RELAY_CREDENTIALS_JSON"

Debug CI #1273

Debug CI #1273

Conversation

chadwhitacre commented Jan 24, 2022 • edited Loading

chadwhitacre commented Jan 24, 2022

chadwhitacre commented Feb 1, 2022

chadwhitacre commented Feb 1, 2022

chadwhitacre commented Feb 1, 2022

chadwhitacre commented Feb 1, 2022

BYK commented Feb 1, 2022

aminvakil commented Feb 1, 2022

chadwhitacre commented Feb 1, 2022

chadwhitacre commented Feb 1, 2022

chadwhitacre commented Feb 1, 2022

BYK commented Feb 1, 2022

chadwhitacre commented Feb 1, 2022

chadwhitacre commented Feb 1, 2022

chadwhitacre commented Feb 1, 2022

BYK commented Feb 1, 2022

BYK Feb 1, 2022 • edited Loading

Choose a reason for hiding this comment

chadwhitacre Feb 1, 2022

Choose a reason for hiding this comment

BYK Feb 1, 2022

Choose a reason for hiding this comment

chadwhitacre Feb 1, 2022

Choose a reason for hiding this comment

BYK Feb 1, 2022

Choose a reason for hiding this comment

chadwhitacre commented Feb 1, 2022

BYK commented Feb 1, 2022

BYK commented Feb 1, 2022

chadwhitacre commented Feb 1, 2022

BYK commented Feb 1, 2022

chadwhitacre commented Feb 1, 2022

BYK commented Feb 1, 2022

chadwhitacre commented Feb 1, 2022

chadwhitacre commented Feb 1, 2022

chadwhitacre commented Feb 1, 2022

chadwhitacre commented Feb 1, 2022

chadwhitacre commented Feb 1, 2022

chadwhitacre commented Feb 1, 2022

chadwhitacre commented Feb 1, 2022

chadwhitacre commented Feb 1, 2022

aminvakil commented Feb 2, 2022

chadwhitacre commented Jan 24, 2022 •

edited

Loading

BYK Feb 1, 2022 •

edited

Loading