[CI Problem] CI node out of memory

[manupak]

Examples :

• https://ci.tlcpack.ai/blue/organizations/jenkins/docker-images-ci%2Fdaily-docker-image-rebuild/detail/daily-docker-image-rebuild/143/pipeline
• https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/ci-docker-staging/175/pipeline/

This started with a timeout on https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/ci-docker-staging/173/pipeline.

It seems like the CI jobs starts with past images (maybe others are aware of this). Should we clean them ?

Possible resolutions for both TVM's Jenkins file and daily-docker-image-rebuild Jenkins file:

• Doing the cleanup_docker_image("ci_cpu") before the build is started
• Do a full prune and pull the docker images from the hub

Any other ideas to proceed are also welcome!

cc : @leandron @tqchen @areusch @u99127

[leandron]

I temporarily disabled this machine on Jenkins, due to storage issues, FYI.
https://ci.tlcpack.ai/computer/octo.aws.c4.44.242.35.246/

[manupak]

Thanks! @leandron

[areusch]

from Noah:

• docker prune script on static nodes (which I have running as a cron job) doesn't pass the 'volumes' flag
• manually fixed 5 nodes which were full (think 246 was one of them)
• permanent fix requires a rolling update of the fleet to bring in a new AMI. no time for this right now.

let's leave this bug open til we do the rolling update.

[leandron]

Cleaning up old tickets that are solved, so I'm closing this one.