Pythonpath for jupyter notebooks?

Helpdesk Software and Environments
1

I have some small libraries that I typically just put on my $PYTHONPATH to import in notebooks. I have added them to $PYTHONPATH via .bashrc and can access them from python in the terminal, but not if I open a jupyter notebook via launcher. $PYTHONPATH seems to be empty when accessed from within the notebook.
What is the best way for me to access this code from notebooks? Those are my own small libraries and are not on pip/conda.

2

In the effort to keep the environments isolated, running kernels in the notebook don’t inherit environment variables by default. To achieve what you want, the easiest option is to define the path at the top of your code with:

import sys
sys.path.insert(0, '/path/to/my/libraries')

The other option is more involved and can give more flexibility. Modify the kernel file for one of the default environments. Say you want to add your path and use it with the sas environment.

  • Copy the kernel definitions to your home folder using a new name, say my-sas:
mkdir -p ~/.local/share/jupyter/kernels
cp -r $JUPYTER_DIR/share/jupyter/kernels/sas ~/.local/share/jupyter/kernels/my-sas
  • Edit your kernel file (my-sas/kernel.json) to include your environment variables under env. e.g:
"env": {
    ...
    "VIRTUAL_ENV": "/opt/envs/sas",
    "PYTHONPATH": "/path/to/my/libraries"
  }
  • In this you should also rename the kernel name in the same file so it is not confused with the default: display_name": "my-sas".

If you refresh the page or start a new session, you ‘custom’ kernel should show up.

1 Like
3

The first option works, the second does not. I am pasting below the my-sas kernel file edited like suggested. But the my-sas enviornment doesn’t show up as an option in the Launcher nor in the Juputer notebook (in the top right where you can change kernerls).

  1. Am I doing something wrong or should I be looking somewhere else to change the kernel?

  2. How do I add certain python packages to this environment so that they stay there and I don’t need to install them each time from scratch?

(python3) ogoann@jupyter-ogoann:~$ cat .local/share/jupyter/kernels/my-sas/kernel.json
{
“argv”: [
“/usr/local/bin/micromamba”,
“run”,
“-p”,
“/opt/envs/sas”,
“-r”,
“/opt/envs/..”,
“python”,
“-Xfrozen_modules=off”,
“-m”,
“ipykernel_launcher”,
“-f”,
“{connection_file}”
],
“display_name”: “my-sas”,
“language”: “python”,
“metadata”: {
“debugger”: true
},
“kernel_protocol_version”: “5.5”,
“env”: {
“PATH”: “/opt/envs/sas/bin:/opt/envs/base/bin:/opt/jupyter/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin”,
“VIRTUAL_ENV”: “/opt/envs/sas”
“PYTHONPATH”: “/home/ogoann/software/”
}
}

4

I think there is a missing comma in the VIRTUAL_ENV line. JSON files are strict about them.

5

In case it’s helpful in addition to what @azoghbi provided above, the relevant documentation is in the Compute Environments page: https://docs.fornax.sciencecloud.nasa.gov/compute-environments/
. For code that is not packaged or installed, the recommended notebook-level approach is to add the directory to sys.path at the top of the notebook. For packages or libraries you want to keep across sessions, the better long-term approach is to create a user environment with setup-pip-env --user or setup-conda-env --user, which also creates a Jupyter kernel for that environment.

6

Those are for creating new environments, not for adding packages to an existing environment. I don’t think we have this use case documented.

1 Like
7

The comma indeed was the issue!

Do I understand correctly that I should wait for example of how to add package to this already existing private environment (that’s a derivative of the sas environment)?

8

You should not wait. you can use one of the solutions above.

My comment meant that the docs should be updated to include the above solution.

9

I was going to try to pip install the packages and so I tried to switch to my-sas in the terminal but seems like this is not possible (see below). I guess this is because I copied the sas enviroment as instructed in this post and did not use conda/pip to set it up? Not sure how to set up my enviroment that has access to XMM SAS + additonal python packages I want in this case. Your help is greatly appreciated!

(python3) ogoann@jupyter-ogoann:~$ micromamba activate my-sas
critical libmamba Cannot activate, prefix does not exist at: ‘/home/ogoann/user-envs/my-sas’

10

The above solution only sets up the kernel for a notebook not a full environment. microconda activate is looking for an environment in ~/user-envs/, which does not exist.

From the terminal, you can use the default sas environment with your custom PYTHONPATH defined in ~/.bashrc like you were doing it originally.

To add packages from pip to the default sas environment, you can use the --prefix option pointing to the path used in PYTHONPATH: e.g. pip install --prefix /home/ogoann/software/ ...

Best way to parallel-reduce a lot of XMM-Newton data?