Nightly jobs fail but only show as "scheduled"

[tripleee]

The latest couple of runs have failed silently.

The log shows

ERROR:root:Attempt 1: denied: unknown reason 'You can perform this action again in 1 second.'

This could be related to chat search being broken

[tripleee]

Chat being broken was a red herring. This keeps on happening.

Out of the last six runs, the last four all failed silently with this error message.

#312 was the latest job which succeeded. The next ones all failed.

[tripleee]

Oh wait, this is a bug I have actually fixed; the regex for "you can perform this action again in \d+ second" was changed upstream, but I don't seem to have the newest version.

It looks like the error in this case was relatively harmless; the latest few Sloshy runs seem to have run to something like completion in spite of the error message.

[tripleee]

Not my fix actually Manishearth/ChatExchange#159

[tripleee]

Builds failing for other reasons, so not properly fixed yet.

[tripleee]

I could not repro locally, but in a fresh env where I first ran pip install --upgrade pip I got into a complex maze of curtsies installation errors.

pypa/setuptools#1694 seems related, though the error message is slightly different. The symptom repeated as pip (the upgraded version, with an "improved" resolver) tried to install different versions of curtsies until it finally found one which it was able to install successfully.

It's not entirely clear to me how it ended up in this dependency branch; pip says ChatExchange requires bpython>=0.16 in requirements-dev.txt. This was on Python 3.6.12 in the Halflife EC2 image.

It did complete eventually, but I'm guessing the problem is around here.

[tripleee]

Even with an upgraded setuptools it goes into the same loop without wheel. (This might still be a red herring; I can't repro the failures from last night.)