Backup and disaster recovery

This page documents good practices for backup and disaster recovery of production Cloud Foundry clusters.

AWS, bosh-bootstrap & bosh-cloudfoundry

Assumptions:

• Inception-server via cloudfoundry-community/inception-server
• BOSH via cloudfoundry-community/bosh-bootstrap
• Cloud Foundry via cloudfoundry-community/bosh-cloudfoundry
• Deployed to AWS

Inception-server

Backup procedure

• Make an AWS AMI from the inception server
• Make a backup of [your workstation]::~/.inception-server

Recovery procedure

• Launch a new instance from your backup AWS AMI
• Restore the backup of [your workstation]::~/.inception-server
• Reassign the elastic IP (as specified in ~/.inception_server/settings.yml

MicroBOSH

Backup procedure

• Make sure you are taking regular snapshots of the persistent EBS volume attached to your MicroBOSH
  • You can find the vol id using inception_server$ grep disk_cid ~/.microbosh/deployments/bosh-deployments.yml
• bosh backup makes a .tgz of MicroBOSH director to the persistent disk of the MicroBOSH. That said, there is no documented way to restore this.

Recovery procedure

Recover terminated MicroBosh on AWS

1. Assumptions

   1. Persistent EBS volume for terminated MicroBosh is still around
   2. bosh-deployments.yml for terminated MicroBosh in git is up-to-date
   3. micro_bosh.yml for terminated MicroBosh in git is up-to-date

2. cd path/to/deployments (directory containing your bosh-deployments.yml file)

3. In bosh-deployments.yml for terminated MicroBosh

   1. Take note of its ami (e.g. ami-12345)
   2. Take note of its persistent disk's volume-id (e.g. vol-111111)

4. mv bosh-deployments.yml bosh-deployments-old.yml

5. Deploy a new MicroBosh

   1. be bosh micro deploy ami-12345
   2. Take note of the new persistent disk's volume-id (e.g. vol-222222)

6. On the new MicroBosh

   1. sudo -i
   2. monit stop all
   3. watch monit summary
   4. Wait until jobs are no longer being monitored
   5. sv stop agent
   6. mount
   7. Take note of device for /var/vcap/store (e.g. /dev/xvdf1, which corresponds to /dev/sdf in the AWS console)
   8. umount /var/vcap/store

7. In the AWS console

   1. Detach the new persistent EBS volume (vol-222222 from new bosh-deployments.yml) from the new MicroBosh instance
   2. Attach old persistent EBS volume (vol-111111 from old bosh-deployments.yml) to the new MicroBosh instance, use the same device as new persistent volume (e.g. /dev/sdf)

8. Back on the new MicroBosh

   1. sudo -i
   2. sed -i 's/vol-222222/vol-111111/' /var/vcap/bosh/settings.json
   3. sv start agent
   4. watch monit summary

9. sed -i 's/vol-222222/vol-111111/' bosh-deployments.yml

10. rm bosh-deployments-old.yml

11. git commit

12. git push

13. In the AWS console, delete the unused EBS volume vol-222222

Cloud Foundry

Backup procedure

Recovery procedure