img

Jupyter.math

Jupyter

URL: https://jupyter.math.uwaterloo.ca

Math's Jupyter server

Project Jupyter is a nonprofit organization created to "develop open-source software, open-standards, and services for interactive computing across dozens of programming languages".

University of Waterloo's Math faculty supports Jupyter with kernels for Python, Firedrake, Julia, Matlab, Octave, R, and other environments. There's also other helpful web based tools like launching Rstudio and Code.

Terms of service

Math Faculty Computing Facility (MFCF) does not take responsibility for lost files or missed deadlines.

You are responsible for backing up your work and having a backup plan in case of service outages.

The server defaults to volatile storage! To ensure your hard work persists, if available use _NetHome (campus storage). The volatile storage is fast and practical for big data courses. However volatile files are placed in an AUTO_DELETE folder after a week. The AUTO_DELETE files are permanently deleted once they age a few months. Volatile files may be deleted immediately if storage becomes full or a disk fails!

Instructors are encouraged to keep course work computationally inexpensive. Also remind students of the importance of keeping backups of their work and storing important files under _NetHome or their personal OneDrive.

Users abusing the service will get banned! Please be ethical and pleasant to the others on the server. Policy and computer ethics.

Always keep a backup plan in case the service goes down. We are running this service without a failover. If the hardware has a critical failure or overloaded, it might take a few days to restore.

FAQ

  • Can I use the service (Student or Instructor)? The service is open to all University of Waterloo students and instructors.
  • Can I use the service (Researcher or Grad)? Please don't use this service! Instead ask your faculty about their researcher resources. We don't want to overload the server impacting the student learning experience!
  • How can I access the service from China? If the site is not already working, you could try using Tor browser.
  • How best to work with the volatile storage? Please read the !ALERT-Use-NetHome.txt file found in your Jupyter base folder.

Issue or requests

Use the Issue or requests form if you are teaching a course and need a missing package added.

Server details

  • CPU: AMD EPYC 7702P 64-Core Processor
  • MEMORY: 512GB (25GB per user)
  • STORAGE: 3TB(U.2) (50GB per user) storage is volatile and auto deleted over time!
  • GPU: 16 x 1080ti & 8 x k80 (only authorized users) indirectly attached from Math's HPC teaching servers
  • NETWORK: 2 x 10GbE

System metrics: howbusy.math

Reduce server memory load

If you are doing some RAM (memory) heavy work in python notebooks, please add %reset -f at the end of the notebook. This will help keep the memory usage down allowing more users to use jupyter.math.

# at the end of each python notebook attempt to clear datasets from RAM
%reset -f

Custom conda environment

The server has limited storage. If you're an instructor, please do not have a large group of students use custom environments. Instead send a package request using the "request form". If there is a large collection of changes and it deviates too much from the norm we can add a new global Jupyter launcher icon tailored with your packages.

To create a custom conda environment, open a fresh Jupyter terminal and run:

# load anaconda environment into the shell
module load anaconda/3.2020.02
# create personal conda env installed at ~/.conda/envs/myconda
# feel free to use a different name than myconda
conda create --name myconda -c conda pip ipykernel
# activate your myconda so the python command uses it
conda activate myconda
# add personal icon to the Jupyter launcher for 'myconda' (relogin to Jupyter to see it)
python -m ipykernel install --user --name 'myconda' --display-name 'myconda'

To add programs to your myconda at a later date, open a fresh Jupyter terminal and run:

# activate environment
module load anaconda/3.2020.02
conda activate myconda
# install custom package
pip install lmfit

For more details see: conda environments

Personal packages

Examples installing to home directory:

./configure --prefix=${HOME}/.local
make --prefix=${HOME}/.local
pip install --user packagename
pip3 install --update --user packagename
python3 -m pip install --user packagename
npm install -g --prefix=${HOME}/.local packagename

R will use your home directory when using install.packages.

# basic example
if(!require(somepackage)) {
    install.packages("somepackage")
    library(somepackage)
}
# source code example
if (!require("EMMIXskew")) {
    install.packages('https://cran.r-project.org/src/contrib/Archive/EMMIXskew/EMMIXskew_1.0.3.tar.gz', repos = NULL, type="source")
    library(EMMIXskew)
}
# TODO: install_github might need you to configure a lib path
remotes::install_github("rspatial/rspatial")
# CLI example
R -q -e "install.packages('GGIR', repos='http://cran.utstat.utoronto.ca/')"

Project Contacts

Steve Weber Server administrator