Machine learning with Tensorflow: an introduction

Using Colaboratory

If you have a Google account, you can use Colaboratory to interactively edit and run notebooks by clicking the badge above. It will load the notebooks from the Github repository, so it will not be able to save there, but you can save a copy in your Google Drive space.

The environment is meant for machine learning (with Tensorflow), so no additional setup is required (resources are not guaranteed, but likely sufficient for anything we will do this afternoon).

Using binder

The badge above will launch a binder session with an environment prepared for the exercises. Resources will be limited, so some of the larger networks may not finish training. Please note that this also is a transient session, so if you want to keep the notebooks with your modifications you will have to download them.

On manneback

The simplest way (since you are assumed to have an account, and to be familiar with using SSH and the command line) if you have a stable internet connection is to open a terminal and launch an interactive job with slurm on the Manneback cluster.

First connect to the interactive node:

ssh gwceci.cism.ucl.ac.be # or gwcism.cism.ucl.ac.be
ssh manneback.cism.ucl.ac.be

We will in any case launch an interactive job on the worker (such that we don't all compete for resources on the same machine), but there are two options: the most convenient is to launch a Jupyter kernel, such that you can edit the notebook (and view some figures) from the browser on your personal computer.

This needs a tunnel to manneback, which can be set up with sshuttle following the CISM instructions, and changing the srun command to

srun --pty bash -c 'source /cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc9-opt/setup.sh; jupyter notebook --ip $(hostname -i) --no-browser'

which will run

source /cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc9-opt/setup.sh
jupyter notebook --ip $(hostname -i) --no-browser

in an interactive slurm job: source an environment setup script, and launch a Jupyter kernel.. Slurm may take a few seconds to assign you a worker node, then the kernel should print a few lines. At the end, there should be "Or copy and paste one of these URLs:", with at least one URL that starts with http://10..... Thanks to the tunnel you can open this in a web browser on your laptop/desktop (which is a better idea than running a web browser with X11 forwarding).

To check that everything is working correctly, you can create a test notebook somewhere in your home directory, and try the following to check that Tensorflow is correctly set up:

import tensorflow as tf
print(tf.__version_)

The main advantage of the notebook over an IPython session is that this gives an inline figure:

import numpy as np
from matplotlib import pyplot as plt
x = np.linspace(0., 5., 101)
plt.plot(x, np.sin(x))

Locally

If your internet connection is not very stable, or you are used to running python on your personal computer or elsewhere, you can also install Tensorflow in a conda environment, or (preferably in a virtual environment) with pip, more details can be found in the installation instructions.

With conda, all you should need is this:

conda config --add channels conda-forge # if not already the case
conda create -n mltftraining2020 tensorflow=2.3.0 tensorboard=2.3.0 ipython matplotlib ipykernel
conda activate mltftraining2020
ipython kernel install --user --name "mltftraining2020"

And with virtualenv and pip:

python -m venv mltftraining2020 # pick a name
source mltftraining2020/bin/activate
pip install tensorflow tensorboard ipython matplotlib ipykernel
ipython kernel install --user --name "mltftraining2020"

If you do not already have the Jupyter notebook server installed, you should add the notebook package to the conda or pip install command. The last line installs a kernel that you can select to run the notebook with.

The exercises assume that you are have at least version 2.1.0 of Tensorflow, which requires Python 3.5 or above (the binder and Colaboratory options above have the most recent version, 2.3.1).

You could also reuse the docker image used by binder above, it is available on dockerhub as pieterdavid/cism-mltf2020, and can be pulled with

docker pull pieterdavid/cism-mltf2020