diff --git a/docs/source/programming/index.md b/docs/source/programming/index.md index c102202..c022ac7 100644 --- a/docs/source/programming/index.md +++ b/docs/source/programming/index.md @@ -14,5 +14,6 @@ vscode-with-slurm-job Mount-ceph-ubuntu Mount-ceph-ubuntu-temp Cookiecutter-cruft +port-fowarding Troubleshooting ``` diff --git a/docs/source/programming/port-fowarding.md b/docs/source/programming/port-fowarding.md new file mode 100644 index 0000000..492b667 --- /dev/null +++ b/docs/source/programming/port-fowarding.md @@ -0,0 +1,142 @@ +# Accessing HTTP servers running on the HPC with port forwarding + +This guide explains how to securely and effectively forward ports from a compute node on the SWC HPC cluster to your local machine, enabling access to services like Jupyter Lab. This is particularly useful when `code tunnel` is unreliable or you prefer using a terminal-based workflow. + +Port forwarding allows you to interact with services running on a compute node (e.g., a Jupyter server on port 8082) from your browser or other tools on your laptop. + +## Overview + +The technique described below **does not involve SSHing into unallocated nodes**, which could interfere with other users or violate HPC usage policies. Instead, you'll **only access a node you've been assigned by SLURM**, and will **forward ports from that node to your laptop**, enabling tools like Jupyter Lab to work as expected without disconnection issues. + +--- + +## Step-by-step Instructions + +### 1. Connect to the cluster and request an interactive job + +```bash +ssh @ssh.swc.ucl.ac.uk +ssh hpc-gw2 +``` + +Then request a SLURM interactive job. For example: + +```bash +srun -p gpu --gres=gpu:1 --mem=16G --pty bash -i +``` + +This will assign you a compute node using one GPU and give you an interactive shell there. + +--- + +### 2. Set up and launch Jupyter Lab + +On the assigned node, activate your environment and navigate to your project folder: + +```bash +conda activate my_env +cd /path/to/your/project +``` + +Then launch Jupyter Lab, specifying a port (e.g., 8082) and disabling the browser: + +```bash +jupyter lab --no-browser --port=8082 +``` + +Jupyter will start and display output similar to this: + +``` +[I 2024-01-01 12:00:00.000 ServerApp] Jupyter Server 2.14.2 is running at: +[I 2024-01-01 12:00:00.000 ServerApp] http://localhost:8082/lab?token=abc123def456... +[I 2024-01-01 12:00:00.000 ServerApp] http://127.0.0.1:8082/lab?token=abc123def456... +... + Or copy and paste one of these URLs: + http://localhost:8082/lab?token=abc123def456... + http://127.0.0.1:8082/lab?token=abc123def456... +``` + +**Look for the lines that say "Or copy and paste one of these URLs:"** - these contain the complete URLs with the authentication token that you'll need to access Jupyter Lab from your browser. + +--- + +### 3. Forward the port from the compute node to your local machine + +On **your local machine**, open a separate terminal and run: + +```bash +ssh -N @ -J @ssh.swc.ucl.ac.uk,@hpc-gw2 -L 8082:localhost:8082 +``` + +Replace `` with the actual name of the compute node assigned to you (e.g., `gpu-sr670-20`). This command establishes a secure tunnel between your laptop and the node. + +Then, **in your browser**, copy one of the complete URLs from the Jupyter Lab output. For example, from the output above, you would copy: + +``` +http://localhost:8082/lab?token=abc123def456... +``` + +**Note:** Either URL works, but you can change `127.0.0.1` to `localhost` if needed. The important part is to **copy the entire URL including the `?token=` portion** - this token is what authenticates you to the Jupyter server. + +--- +## Troubleshooting + +- **Port mismatch?** Ensure both commands use the same port number. +- **Connection drops?** Keep your SLURM session active on the assigned node. +- **Port in use?** Try a different port (e.g., `8888`, `8090`) in both commands. + +--- + +## Complementary tools + +If you prefer a fully integrated development environment see our guide on: + +[Using VSCode with Interactive SLURM Jobs →](./vscode-with-slurm-job.md) + +--- + +## Examples of Other Web Applications + +### Dash Applications + +For Dash applications, you can follow the same port forwarding approach: + +Create your `app.py`: + +```python +from dash import Dash, html, dcc +import dash_bootstrap_components as dbc + +app = Dash(__name__, external_stylesheets=[dbc.themes.BOOTSTRAP]) + +app.layout = html.Div([ + html.H1("My Dash App"), + dcc.Graph(id='example-graph') +]) + +if __name__ == '__main__': + app.run(debug=False, host='0.0.0.0', port=8050) +``` + +**On the compute node**, launch your Dash app: + +```bash +python app.py +``` + + +### Streamlit Applications + +For Streamlit applications: + +```python +import streamlit as st + +st.title("My Streamlit App") +st.write("This is a simple Streamlit app.") +``` +**On the compute node**, launch Streamlit with a specific port: + +```bash +streamlit run app.py --server.port 8501 --server.address 0.0.0.0 +``` diff --git a/docs/source/programming/vscode-with-slurm-job.md b/docs/source/programming/vscode-with-slurm-job.md index f3db1e5..7b18232 100644 --- a/docs/source/programming/vscode-with-slurm-job.md +++ b/docs/source/programming/vscode-with-slurm-job.md @@ -67,11 +67,12 @@ As explained in [VSCode docs](https://code.visualstudio.com/docs/remote/tunnels) >Once you connect from a remote VS Code instance, an SSH connection is created over the tunnel in order to provide end-to-end encryption. ::: -## Additional benefits of code tunnel +## Port forwarding -One advantage of using VSCode's code tunnel is that it forwards any HTTP servers launched from the same node, such as Dash-Plotly apps or Jupyter Notebook servers. To launch your additional server, request a separate slurm job for the same compute node, e.g.: +One advantage of using VSCode's code tunnel is that it automatically detects any HTTP servers you launch from the same node (such as Dash-Plotly apps or Jupyter Notebook servers) and shows them in the "Ports" view. Make sure to launch your server inside your compute node, for example by running: ```{code-block} console -$ srun -p cpu -w -n 4 --mem 8G --pty bash -i +$ jupyter notebook ``` -When these are initiated, VSCode will notify you with a link that you can follow to access the server's UI directly. + +You can then open the forwarded address from the Ports view in your browser or copy it. By default, forwarded ports are private and require GitHub authentication; to make a port public (no sign-in required), right-click the port and select "Port Visibility > Public." For more details, see the [VSCode port forwarding documentation](https://code.visualstudio.com/docs/debugtest/port-forwarding).