-
Notifications
You must be signed in to change notification settings - Fork 9
Background Jobs
Heliotrope uses Resque to process background jobs, such as characterizing files with FITS and creating derivatives (thumbnails, web-friendly video, etc.). In production we recommend resque-pool for managing resque workers. In development, you can use the relevant rake tasks. The job queue is stored in Redis.
If you are using a Mac, you can use Homebrew to install redis:
brew install redis
If you're using a Vagrant box, redis server is probably already installed and running.
You can check with the following command: redis-cli ping, the server should respond with "pong".
To start the redis server:
redis-server /usr/local/etc/redis.conf
# (or on ubuntu)
sudo redis-server /etc/redis/redis.conf
# (or on a Mac with Homebrew, to auto-start on login)
brew services start redis
To start workers(s) to run the jobs (one worker per terminal window:
QUEUE=* VERBOSE=1 bundle exec rake resque:work
OR you can use Resque pool to bring up a pool of workers in a daemonized way. The queue and worker count are pulled from config/resque-pool.yml
RUN_AT_EXIT_HOOKS=true TERM_CHILD=1 VERBOSE=1 bundle exec resque-pool --daemon --environment development start
The first two added variables help with clean up when forked workers are killed, the third adds verbosity which can help you find problems in dev.
Resque-web is mounted onto the rails app, but is only available to platform-level admin users (superadmins). To see the status of recent jobs in the resque-web console, start up the server, log in as a platform admin, then navigate your browser to:
On nectar.umdl.umichi.edu there is a script to check if heliotrope's resque pool is running at:
/hydra-dev/heliotrope-testing/bin/check_resque_pool
If you haven't yet, you'll need to install FITS or some resque jobs like Characterize will fail. And if one job fails, the whole thing fails.
Download FITS and put it where Heliotrope expects it, /usr/local/bin/fits.sh.
- Download fits and unzip it somewhere (I did ~/bin/fits-0.10.1/)
sudo ln -s /path/to/your/fits-0.10.1/fits.sh /usr/local/bin/fits.sh
For new versions of fits (I've tried 1.0.7) you are done. If running /usr/local/bin/fits.sh gives an error, thn you can try changing the line in fits.sh from this:
. "$(dirname $BASH_SOURCE)/fits-env.sh"
To this:
. "/path/to/your/fits-env.sh"
- Homebrew on Mac is an alternative to the preceding