Cluster Jupyter Setup Guide¶

Running notebooks on an HPC cluster with pixi and connecting VS Code or a local browser.

Table of Contents¶

Environment Setup (pixi)
Running Notebooks on the Cluster
VS Code — Jupyter Kernel Connection
- Option A: Remote SSH (recommended)
- Option B: Local pixi kernel (CPU only)
Local Jupyter (no cluster)

Environment Setup (pixi)¶

The project uses pixi for reproducible environment management.

Install pixi (once, per user)¶

curl -fsSL https://pixi.sh/install.sh | bash
# Restart your shell or: source ~/.bashrc

Install environments¶

cd <repo-path>   # e.g. /path/to/Cytotoxicity-Pipeline

# Default CPU environment (notebooks, pipeline, docs)
pixi install

# GPU environment — Cellpose segmentation (PyTorch + CUDA 11.8)
pixi install -e cellpose

# GPU environment — StarDist segmentation (TensorFlow)
pixi install -e stardist

# TrackMate / Fiji integration
pixi install -e imagej

# Cellpose + TrackMate combined
pixi install -e gpu

Available environments¶

Environment	Key extras	Replaces
`default`	JupyterLab, Dask, analysis stack	`env.yaml`
`cellpose`	PyTorch, Cellpose ≥ 4, CUDA 11.8	`env-pytorch.yaml` + `env-cuda.yaml`
`stardist`	TensorFlow 2.12, StarDist	`env-tensorflow.yaml`
`imagej`	PyImageJ, OpenJDK 11	`env-imagej.yaml`
`gpu`	Cellpose + TrackMate	combined
`rapids`	dask-cuda, RAPIDS	`env-cuda.yaml`

MPI note: mpi4py is installed from conda-forge (OpenMPI). For cluster jobs that require the system MPI, rebuild it after loading the cluster MPI module:
module load mpi
pixi run pip install --no-binary mpi4py mpi4py

Running Notebooks on the Cluster¶

Notebooks run on a compute node, not the login node. The flow is:

        graph LR
    A["Local machine"] -->|SSH tunnel| B["Login node"]
    B -->|SLURM srun| C["Compute node"]
    C -->|serves| D["JupyterLab :8888"]

    classDef clusterNode fill:#0d7377,color:#fff,stroke:#0a5c60
    class A,B,C,D clusterNode

Step 1 — Allocate a compute node¶

Note

Partition names (short, gpu_short) are cluster-specific. Check available partitions with sinfo -h -o "%P|%a|%l|%D|%t" and substitute your cluster’s names.

CPU interactive session:

srun -p short --cpus-per-task=4 --mem=16G --time=04:00:00 --pty bash

GPU interactive session:

srun -p gpu_short --gres=gpu:1 --cpus-per-task=4 --mem=16G --time=04:00:00 --pty bash

Step 2 — Start JupyterLab on the compute node¶

cd <repo-path>

# Default environment
pixi run jupyter

# GPU work (Cellpose)
pixi run -e cellpose jupyter

This prints a URL like:

http://127.0.0.1:8888/lab?token=abc123...

Note the compute node hostname shown in your shell prompt:

echo $HOSTNAME   # e.g. node0123.cluster.example.ac.uk

Step 3 — SSH tunnel from your local machine¶

Open a new terminal locally and run:

# Replace node0123, <username>, and <login-node> with your actual values
ssh -L 8888:node0123.cluster.example.ac.uk:8888 \
    -J <username>@<login-node> \
    <username>@node0123.cluster.example.ac.uk -N

The -J flag jumps through the login node. Your cluster may have multiple login nodes (e.g. cluster1.hpc.example.ac.uk through cluster4.hpc.example.ac.uk) — use any one.

Step 4 — Open in browser¶

Navigate to http://localhost:8888 and paste the token from Step 2.

VS Code — Jupyter Kernel Connection¶

Option A: Remote SSH (recommended)¶

Install the Remote - SSH extension in VS Code.
Add an SSH host (Ctrl+Shift+P → Remote-SSH: Add New SSH Host):
```
ssh <username>@<login-node>
```
Remote-SSH: Connect to Host → select the cluster.
Open the project folder: <repo-path>
In the VS Code terminal, run Steps 1–2 above to allocate a node and start JupyterLab.
Open any .ipynb notebook.
Select kernel (top-right): Select Another Kernel → Existing Jupyter Server → paste http://localhost:8888/?token=<token>.

VS Code forwards the port automatically when connected via Remote-SSH — no manual SSH tunnel needed.

Option B: Local pixi kernel (quick, CPU only)¶

cd <repo-path>
pixi run python -m ipykernel install --user --name cyto --display-name "pyCyto (pixi)"

Then in VS Code (connected via Remote-SSH), select kernel: pyCyto (pixi).

For GPU notebooks always use Option A (requires a SLURM compute node allocation).

Local Jupyter (no cluster)¶

To run notebooks entirely on your local machine (CPU only):

cd <repo-path>
pixi install
pixi run jupyter

Open the printed URL in your browser. GPU-accelerated steps (Cellpose, contact detection) will fall back to CPU or must be skipped.

For local GPU support, ensure CUDA drivers are installed and run:

pixi run -e cellpose jupyter