Installing and Running Jupyter Notebooks on a Server12 Feb 2019
Jupyter Notebook is a powerful tool, but how can you use it in all its glory on a server? In this tutorial you will see how to set up Jupyter notebook on a server like Digital Ocean, AWS or most other hosting provider available. Additionally, you will see how to use Jupyter notebooks over SSH tunneling or SSL with with Let’s Encrypt.
Jupyter Notebook is an open source web application that enables interactive computing from the browser. You can create documents that feature live code, documentation with Markdown, equations, visualization and even widgets and other interesting capabilities. Jupyter comes from the three core languages that are supported: Julia, Python, and R. Jupyter connects to a kernel with a specific language, the most common being the IPython kernel. It supports a whole variety of kernels and you should find most languages you need. This tutorial was written in JupyterLab, the next developments of Jupyter notebook:
In this tutorial we will be working with Ubuntu 16.04/18.04 servers, but most steps should be fairly similar for Debian 8/9 distributions. We will first go through creating SSH keys, adding a new user on the server, and installing Python and Jupyter with Anaconda. Next, you will setup Jupyter to run on the server. Finally, you can either choose to run Jupyter notebooks over SSH tunneling or over SSL with Let’s Encrypt.
Create SSH Keys
We are starting with a fresh server and in order to add more security when accessing your server, you should consider using SSH key pairs. These key pairs consist of a public key which is uploaded to the server and a private key that stays on your machine. Some hosting providers require you to upload the public key before creating the server instance. To create a new SSH key you can use the ssh-keygen tool. To create the key pairs you can simply type the command:
this will prompt you to add the file path and a passphrase if you want to. There are other arguments of options you can choose from like public key algorithm or file name. You can find a very good tutorial here on how to create a new SSH key with ssh-keygen for Linux or macOS. If you are using Windows, you can create SSH-keys with PuTTYgen as described here. If your hosting provider does not need a public key before creation you can copy the public key with the ssh-copy-id tool:
ssh-copy-id -i ~/.ssh/jupyter-cloud-key user@host
Finally, you can connect to your server with:
ssh -i ~/.ssh/id_rsa root@host
~/.ssh/id_rsa is the path to your ssh private key and
host is the host address or IP address of you server instance.
Adding a New User
In some servers you start off as a root user. It is considered bad practice to work directly with the root since it has a lot of privileges which can be destructive if some commands are done by accident. If you already have a user you can skip this section. Note that you can replace
cloud-user in all the following commands with the user name you want. Start by creating a new user:
This command will ask you a couple of questions including a password. Next, you’ll want to grant administrative privileges to this user. You can do this by typing
usermod -aG sudo cloud-user
Now you are ready to switch to the new user with
su cloud-user or by connecting to your server with
ssh cloud-user@host. Optionally, you can add the SSH keys of the root user to the new user for additional security. Otherwise you can skip to the next section on how to install Anaconda. Now, if you you have existing SSH keys for the root user you can copy the public key from the root home folder to the users home folder like shown here:
mkdir /home/cloud-user/.ssh cp /root/.ssh/authorized_keys /home/cloud-user/.ssh/
Next, you need to change the permissions for both the folder and the public key:
cd /home/user/ chmod 700 .ssh/ chmod 600 .ssh/authorized_keys
If you are using a password for your user you need to update
There you want to find the line
PasswordAuthentication no and change the
no to a
yes to allow password authentication. Finally you want to restart the SSH service by typing
service ssh restart. For other distributions have a look into this guide, where you will also see how to set up a firewall.
Anaconda is an open-source distribution of Python (and R) for scientific computing including package management and deployment. With it, you have most tooling that you need including Jupyter. To install Anaconda, go to the downloads for linux and copy the Linux installer link for the latest Python 3.x version. Then you can download the installer with
Next you can install Anaconda by using
bash as follows:
During installation, it is important to type
yes when the following prompt appears during the installation:
Do you wish the installer to prepend the Anaconda3 install location to PATH in your /home/user/.bashrc ? [yes|no]
After you finished installing you want to initialize the
conda command line tool and package manager by Anaconda with:
source .bashrc conda update conda
These two commands set up Anaconda on your server. If you have run the Anaconda bash file with sudo, you will get a
Permission denied error. You can solve it as shown in this question by typing
sudo chown -R $$USER:$$USER /home/user/anaconda3. This changes the owner of this folder to the current user with the chown command.
Starting Jupyter Notebook Server
Jupyter is installed with Anaconda, but we need to do some configuration in order to run it on the server. First, you’ll want to create a password for Jupyter notebook. You can do this by starting the IPython shell with
ipython and generating a password hash:
from IPython.lib import passwd passwd()
Save this resulting hash for now, we will need it in a moment. Next, you want to generate a configuration file which you can create by typing.
Now open the configuration file with
sudo nano ~/.jupyter/jupyter_notebook_config.py and copy the following code into the file and replace the hash in this snippet with the one you have previously generated:
c = get_config() # get the config object # do not open a browser window by default when using notebooks c.NotebookApp.open_browser = False # this is the password hash that we generated earlier. c.NotebookApp.password = 'sha1:073bb9acaa67:b367308802ab66cb1d7654b6684eafefbd61d004'
Now you should be set up. Next, you can decide whether you want to use SSH tunneling or you want to use SSL encryption and access your jupyter notebook over your own domain name.
SSH Tunneling with Linux or MacOS
You can tunnel to your server by adding the
-L argument to the
ssh command, which is responsible for port forwarding. The first
8888 is the port you will access on your local machine (if you already use this port for another juypter instance you can use port 8889 or a different open port). You can access this then on your browser with
localhost:8888. The second part
localhost:8888 specifies the jump server address accessed from the server. Since we want to run the notebook locally on the server, this is again localhost. This would mean that we access
localhost:8888 from the server via port forwarding to
localhost:8888 on our machine. Here is how the command would look like:
ssh -L 8888:localhost:8888 cloud-user@host
If you have another Jupyter notebook running on your local machine already you can change the port to e.g.
8889 which would result in the command:
ssh -L 8889:localhost:8888 cloud-user@host
Now, you can create a notebook folder for your projects on the server and run Jupyter notebook inside:
mkdir notebook cd notebook/ jupyter-notebook
You can also use JupyterLab instead, which is a more powerful interface and it comes also pre-installed with Anaconda. You can start it by typing
jupyter-lab instead of
SSL Encryption with Let’s Encrypt
It is also possible to use SSL encryption for your jupyter notebook. This enables you to access your Jupyter notebooks through the internet which makes it handy to share results with your colleagues. To do this you can use Let’s Encrypt, which is a free Certificate Authority (CA) that provides an easy way for TLS/SSL certificates. This can be done fully automated with their certbot tool. To find the installation guide for your system have a look at this list. For Ubuntu 18.04 the installation looks as follows:
sudo apt-get update sudo apt-get install software-properties-common sudo add-apt-repository universe sudo add-apt-repository ppa:certbot/certbot sudo apt-get update sudo apt-get install certbot python-certbot-apache
Now, you can run certbot for the domain that you have:
sudo certbot certonly -d example.com
After going through the prompts, you should get to this output:
IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/example.com/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/example.com/privkey.pem Your cert will expire on 2019-05-09. To obtain a new or tweaked version of this certificate in the future, simply run certbot again with the "certonly" option. To non-interactively renew *all* of your certificates, run "certbot renew" - If you like Certbot, please consider supporting our work by: Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le
Great! You have your certificate and key file ready. Now you can use the certificate and key file in your jupyter notebook configuration file. Before you can do that, you need to change the owner of the certificate and key file with (change
user with your own user name):
sudo chown user /usr/local/etc/letsencrypt/live sudo chown user /usr/local/etc/letsencrypt/archive
Next, you can add the following code to the
~/.jupyter/jupyter_notebook_config.py configuration file:
# Path to the certificate c.NotebookApp.certfile = '/etc/letsencrypt/live/example.com/fullchain.pem' # Path to the certificate key we generated c.NotebookApp.keyfile = '/etc/letsencrypt/live/example.com/privkey.pem' # Serve the notebooks for all IP addresses c.NotebookApp.ip = '0.0.0.0'
Finally, you can access Jupyter notebooks securely over
https://example.com:8888. Just make sure to use
https:// instead of
http://. If you made any mistakes, you can delete the certbot certificate with
sudo certbot delete or
sudo certbot delete --cert-name example.com. If you are using a firewall, make sure that port
8888 is open. Here is a good guide on using the Uncomplicated Firewall (UFW) firewall.
You have learned how to set up Jupyter for a server from start to finish. This is a task that gets easier with every server set up that you do. Make sure to delve into the surrounding topics of Linux server administration since working with servers can be intimidating in the beginning. Using Jupyter you have access to a wide variety of kernels that enable you to use other languages. A list of all available kernels can be found here. I hope this was helpful and if you have any further questions or remarks, feel free to share them in the comments bellow.
I covered in a previous tutorial how to work with virtual environments in Jupyter notebook. There is also an option to run Jupyter as a Docker container. You can use for example the jupyter/datascience-notebook container. You can read more on how to work with Jupyter and Docker in this guide. Here are further links that I have learned from and that might be useful for you too:
- Initial Server Setup
- Running a notebook server
- How To Set Up Jupyter Notebook for Python 3
- How To Use Certbot Standalone Mode to Retrieve Let’s Encrypt SSL Certificates
- UFW Essentials: Common Firewall Rules and Commands
- Using Virtual Environments in Jupyter Notebook and Python
- Creating Slides with Jupyter Notebook