diff --git a/_posts/2023-09-25-Eddie.md b/_posts/2023-09-25-Eddie.md index 461ab082b3fa..09b781007e80 100644 --- a/_posts/2023-09-25-Eddie.md +++ b/_posts/2023-09-25-Eddie.md @@ -44,22 +44,22 @@ This is the shared group space in UoE. \ ## **Basic Commands on Eddie** To start an interactive session run: ```shell -$ qlogin +qlogin # The default maximum runtime is 48 hours. The default resources: 1 core, 1 GB memory. -$ qlogin -l h_vmem=1G +qlogin -l h_vmem=1G # request memory for an interactive session -$ qlogin -pe gpu 1 +qlogin -pe gpu 1 # request 1 gpu (need for torch and memory usage is at least 2GB) -$ qlogin -q staging +qlogin -q staging # choose the staging environment ``` Staging data (transfer data between Eddie and DataStore): ```shell -$ qsub stagein/out.sh +qsub stagein/out.sh # stagein/out.sh file content can be seen in Eddie handbook (need UUN login) ``` > ##### TIP @@ -68,15 +68,15 @@ $ qsub stagein/out.sh To view current running jobs ```shell -$ qstat +qstat ``` To cancel a job, ```shell -$ qdel +qdel ``` Detailed information about completed jobs is available with the qacct command: ```shell -$ qacct -j +qacct -j ``` ## **Set up the Anaconda Environment** @@ -84,73 +84,73 @@ $ qacct -j ### Add path and create a virtual environment Configure the path for your environments directory, i.e., the directory where all your conda environments will be stored, and packages (pkgs) directory used as a cache for your conda packages. By default cache for packages will be placed under `~/.conda/pkgs` and will very likely cause your home directory to go over quota as a result. We recommend using the group space you have used for your conda environments instead. **Note that this should be an existing directory, so you need to create it first if it doesn't already exist, before running the command below.** ```shell -$ module load anaconda # version 5.0.1 +module load anaconda # version 5.0.1 ``` Skip this step if already created the virtual environment ```shell -$ conda config --add envs_dirs /exports//eddie//groups//anaconda/envs +conda config --add envs_dirs /exports//eddie//groups//anaconda/envs -$ conda config --add pkgs_dirs /exports//eddie//groups//anaconda/pkgs +conda config --add pkgs_dirs /exports//eddie//groups//anaconda/pkgs ``` Once you have configured your `envs_dirs` and `pkgs_dirs`, you can create an environment. The example below creates an environment specifying particular Python versions (also the version used for this project): ```shell -$ conda create -n %env_name% python=3.9 +conda create -n %env_name% python=3.9 ``` To list your conda environments: ```shell -$ conda info --envs +conda info --envs ``` ### Activate your environment Switch to the created environment and deactivate the environment when no longer needed: ```shell -$ source activate %env_name% -$ source deactivate +source activate %env_name% +source deactivate ``` ### Install required packages ```shell -$ conda install pip -$ pip --no-cache-dir install -r requirements.txt +conda install pip +pip --no-cache-dir install -r requirements.txt # `--no-cache-dir` is to ensure torch installed successfully ``` **Some package management commands:** ```shell -$ conda list +conda list # To see what packages are in the active environment -$ conda search matplotlib +conda search matplotlib # To search for a package in the Anaconda repositories -$ conda install matplotlib=1.4.1 +conda install matplotlib=1.4.1 # To install a package e.g. matplotlib version 1.4.1 ``` #### Using pip If a package is not available in the Anaconda repositories it can still be installed with `pip` in the usual way. First install `pip` into your active Anaconda environment. You must install pip into your environment first before running a pip command. Otherwise, `pip install` will use the pip installed in the Anaconda directory (which users do not have write access to) and your installation will fail. Once you have installed pip in your environment, then you can install a package using pip into your environment: ```shell -$ conda install pip -$ pip install matplotlib +conda install pip +pip install matplotlib ``` ## **Set up R Environment** ### Configure R packages path ```shell -$ module load cmake/3.5.2 +module load cmake/3.5.2 # some packages may need this to be loaded first -$ module load igmm/apps/R/4.1.3 -$ export R_LIBS=/exports//eddie//groups//rlibs +module load igmm/apps/R/4.1.3 +export R_LIBS=/exports//eddie//groups//rlibs ``` **Install required packages in R** ```R -> install.packages("tidyverse") +install.packages("tidyverse") ``` ## **Run Code on Eddie** ### Through job submission **Example** ```shell -$ qsub run_job.sh +qsub run_job.sh # job submission files are available in shell folder ``` @@ -158,7 +158,7 @@ $ qsub run_job.sh First start an interaction session, specify max run time, memory usage (gpu) and activate the environment then run according to the following: **Run Jupyter notebook** ```shell -$ jupyter nbconvert --to notebook --inplace --execute jupyter-notebook.ipynb +jupyter nbconvert --to notebook --inplace --execute jupyter-notebook.ipynb ``` Note that `nbconvert` will fail if any cell takes longer than 30s to run, then add the following option to disable timing ``` @@ -166,5 +166,5 @@ Note that `nbconvert` will fail if any cell takes longer than 30s to run, then a ``` View jupyter notebook ouptputs ```shell -$ nbpreview jupyter-notebook.ipynb +nbpreview jupyter-notebook.ipynb ```