DEPRECATION NOTICE: The on-ramp codelabs are currently in the process of being deprecated in favor of content in the user manual. If anything seems out of date in this content, please refer to the user manual content on running software on the clusters.
If you are familiar with the Linux command line, the traditional terminal interface is the most efficient method for accessing the CRC compute and storage resources. username should be your Pitt ID.
YOUR-PC:~ YOUR-LOCAL-USERNAME$ ssh -X username@h2p.crc.pitt.edu
username@h2p.crc.pitt.edu's password:
Warning: untrusted X11 forwarding setup failed: xauth key data not generated
Last login: Thu Jan 13 12:09:21 2022
#########################################################################################################################################################################################
Welcome to h2p.crc.pitt.edu!
Documentation can be found at crc.pitt.edu/h2p
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
IMPORTANT NOTIFICATIONS
Renewal of CRC allocations requires you to acknowledge and add citations to our database, login to crc.pitt.edu and navigate to crc.pitt.edu/acknowledge for details and entry form
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
IMPORTANT REMINDERS
Don't run jobs on login nodes! Use interactive jobs: `crc-interactive --help`
Slurm is separated into 'clusters', e.g. if `scancel <jobnum>` doesn't work try `crc-scancel <jobnum>`. Try `crc-sinfo` to see all clusters.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
#########################################################################################################################################################################################
[username@login1 ~]$ pwd
/ihome/groupname/username
[username@login1 ~]$ ls
CRC Desktop zzz_cleanmeup
CRC uses the Lmod Environment Modules tool to manage and provision software applications. The command module spider shows if a package is available. For example
[username@login1 ~]$ module spider python
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
python:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Description:
Anaconda is the leading open data science platform powered by Python.
Versions:
python/anaconda2.7-4.2.0_westpa
python/anaconda2.7-4.2.0
python/anaconda2.7-4.4.0_genomics
python/anaconda2.7-5.2.0_westpa
python/anaconda2.7-5.2.0
python/anaconda2.7-2018.12_westpa
python/anaconda3.5-4.2.0-dev
python/anaconda3.5-4.2.0
python/anaconda3.6-5.2.0_deeplabcut
python/anaconda3.6-5.2.0_leap
python/anaconda3.6-5.2.0
python/anaconda3.7-5.3.1_genomics
python/anaconda3.7-2018.12_westpa
python/anaconda3.7-2019.03_astro
python/anaconda3.7-2019.03_deformetrica
python/anaconda3.7-2019.03
python/anaconda3.8-2020.11
python/anaconda3.9-2021.11
python/bioconda-2.7-5.2.0
python/bioconda-3.6-5.2.0
python/bioconda-3.7-2019.03
python/intel-3.5
python/intel-3.6_2018.3.039
python/intel-3.6_2019.2.066
python/intel-3.6
python/ondemand-jupyter-python3.8
python/3.7.0-dev
python/3.7.0-fastx
python/3.7.0
Other possible modules matches:
biopython openslide-python
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
To find other possible module matches do:
module -r spider '.*python.*'
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
For detailed information about a specific "python" module (including how to load the modules) use the module's full name.
For example:
$ module spider python/ondemand-jupyter-python3.8
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
This shows that we have several versions of Python available. Packages typically have dependencies. To discover these dependencies, apply the module spider command to a specific installed package
[username@login1 ~]$ module spider python/anaconda3.7-2019.03_deformetrica
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
python: python/anaconda3.7-2019.03_deformetrica
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Description:
Anaconda is the leading open data science platform powered by Python. Compatible with gcc/8.2.0
Other possible modules matches:
biopython, openslide-python
You will need to load all module(s) on any one of the lines below before the "python/anaconda3.7-2019.03_deformetrica" module is available to load.
gcc/8.2.0
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
To find other possible module matches do:
module -r spider '.*python/anaconda3.7-2019.03_deformetrica.*'
If you attempt to directly load this Python to your session environment, you will encounter an error because the gcc/8.2.0 dependency has not been satisfied
[username@login1 ~]$ module load python/anaconda3.7-2019.03_deformetrica Lmod has detected the following error: These module(s) exist but cannot be loaded as requested: "python/anaconda3.7-2019.03_deformetrica" Try: "module spider python/anaconda3.7-2019.03_deformetrica" to see how to load the module(s).
The gcc/8.2.0 module first before loading the desired Python
[username@login1 ~]$ module load gcc/8.2.0 [username@login1 ~]$ module list Currently Loaded Modules: 1) gcc/8.2.0 [username@login1 ~]$ module load python/anaconda3.7-2019.03_deformetrica [username@login1 ~]$ module list Currently Loaded Modules: 1) openmpi/3.1.1 2) gcc/8.2.0 3) python/anaconda3.7-2019.03_deformetrica
You can also load all the packages using a single commandline, making sure that the dependencies are to the left of the package
[username@login1 ~]$ module purge [username@login1 ~]$ module load gcc/8.2.0 python/anaconda3.7-2019.03_deformetrica [username@login1 ~]$ module list Currently Loaded Modules: 1) openmpi/3.1.1 2) gcc/8.2.0 3) python/anaconda3.7-2019.03_deformetrica
In the previous section, two new commands that were introduced.
They are module purge and module list. These commands do exactly as the words imply.
module list is used to list all the loaded software packages
module purge is to get remove of all the packages from your current session's environment
[username@login1 ~]$ module list Currently Loaded Modules: 1) openmpi/3.1.1 2) gcc/8.2.0 3) python/anaconda3.7-2019.03_deformetrica [username@login1 ~]$ module purge [username@login1 ~]$ module list No modules loaded [username@login1 ~]$
Now, you might be wondering if it is possible to remove a particular package while keeping others. Why don't we try it
[username@login1 ~]$ module purge
[username@login1 ~]$ module list
No modules loaded
[username@login1 ~]$ module spider matlab
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
matlab:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Description:
Matlab R2021b
Versions:
matlab/R2015a
matlab/R2017a
matlab/R2018a
matlab/R2019b
matlab/R2020b
matlab/R2021a
matlab/R2021b
Other possible modules matches:
matlab-mcr matlab-proxy
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
To find other possible module matches do:
module -r spider '.*matlab.*'
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
For detailed information about a specific "matlab" module (including how to load the modules) use the module's full name.
For example:
$ module spider matlab/R2021b
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[username@login1 ~]$ module load matlab/R2021b [username@login1 ~]$ module list Currently Loaded Modules: 1) fontconfig/2.10.95 2) matlab/R2021b [username@login1 ~]$ module load gcc/8.2.0 python/anaconda3.7-2019.03_deformetrica [username@login1 ~]$ module list Currently Loaded Modules: 1) fontconfig/2.10.95 2) matlab/R2021b 3) openmpi/3.1.1 4) gcc/8.2.0 5) python/anaconda3.7-2019.03_deformetrica [username@login1 ~]$ module rm matlab/R2021b [username@login1 ~]$ module list Currently Loaded Modules: 1) openmpi/3.1.1 2) gcc/8.2.0 3) python/anaconda3.7-2019.03_deformetrica
In the above commands, MATLAB R2021b was loaded and then Python was loaded afterwards. Notice that MATLAB can be loaded directly but that there is also a side effect of automatically loading fontconfig/2.10.95.
Next, the gcc/8.2.0 dependency was loaded before the specific Python package. This Python package also has a side effect of automatically loading openmpi/3.1.1.
Lastly, when MATLAB was unloaded, matlab/R2021b and fontconfig/2.10.95 are removed from the environment.
You might wonder, What happens if I unload Python instead of MATLAB? Let's try it out:
[username@login1 ~]$ module list Currently Loaded Modules: 1) openmpi/3.1.1 2) gcc/8.2.0 3) python/anaconda3.7-2019.03_deformetrica [username@login1 ~]$ module load matlab/R2021b [username@login1 ~]$ module list Currently Loaded Modules: 1) openmpi/3.1.1 2) gcc/8.2.0 3) python/anaconda3.7-2019.03_deformetrica 4) fontconfig/2.10.95 5) matlab/R2021b [username@login1 ~]$ module rm python/anaconda3.7-2019.03_deformetrica [username@login1 ~]$ module list Currently Loaded Modules: 1) fontconfig/2.10.95 2) matlab/R2021b
The effect of unloading a package module is to remove all dependencies and to return the session environment to the state before loading the package.
The command to unload a package is module rm .
You might also wonder if module unload might be a better choice of words for the command.
As a matter of fact, module rm and module unload are synonymous. Try it out.
These are few commands are useful to memorize to efficiently manipulate the software package environment. The reason why it is necessary to employ LMOD is because the CRC community uses a broad range of software applications, many of which are not compatible with one another.
The best way to get help on a specific issue is to submit a help ticket. You should log in to the CRC website using your Pitt credentials first.
Please return to the Pitt CRC Codelabs dashboard and find the "The SLURM Workload Manager" codelab, or use this link