Skip to content

Commit 4a0bb56

Browse files
uralikuralik
committed
greene cluster guide added
1 parent 67f4909 commit 4a0bb56

File tree

6 files changed

+422
-1
lines changed

6 files changed

+422
-1
lines changed

‎client/config‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,4 +23,14 @@ Host prince
2323
User <USERNAME> # DIFFERENT USERNAME HERE GIVEN BY HPC NOT CIMS
2424
IdentityFile <PATH_TO_KEY>
2525
ServerAliveInterval 60
26-
ProxyCommand ssh cims -W %h:%p
26+
ProxyCommand ssh cims -W %h:%p
27+
28+
Host greene
29+
HostName log-2.hpc.nyu.edu
30+
Port 22
31+
Compression yes
32+
User <USERNAME> # DIFFERENT USERNAME HERE GIVEN BY HPC NOT CIMS
33+
IdentityFile <PATH_TO_KEY>
34+
ForwardAgent yes
35+
ServerAliveInterval 60
36+
ProxyCommand ssh cims -W %h:%p

‎greene/README.md‎

Lines changed: 296 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,296 @@
1+
# Greene tutorial
2+
3+
This tutorial aims to help with your migration from the Prince cluster (retires early 2021). It assumes some knowledge of Slurm, ssh and terminal. It assumes that one can connect to Greene ([same way](https://github.com/nyu-dl/cluster-support/tree/master/client) as to Prince).
4+
5+
Official Greene docs & overview:
6+
https://sites.google.com/a/nyu.edu/nyu-hpc/documentation/greene
7+
https://sites.google.com/a/nyu.edu/nyu-hpc/systems/greene-cluster
8+
9+
**Content:**
10+
1. Greene login nodes
11+
2. Cluster overview
12+
3. Singularity - running jobs within a container
13+
4. Running a batch job with singularity
14+
5. Setting up a simple sweep over a hyper-parameter using Slurm array job.
15+
6. Port forwarding to Jupyter Lab
16+
7. Using a python-based [Submitit](https://github.com/facebookincubator/submitit) framework on Greene
17+
18+
## Greene login nodes
19+
20+
* greene.hpc.nyu.edu (balancer)
21+
* log-1.hpc.nyu.edu
22+
* log-2.hpc.nyu.edu
23+
24+
You have to use NYU HPC account to login, username is the same as your NetID.
25+
26+
Login nodes are accessible from within NYU network, i.e. one can connect from CIMS `access1` node, check [ssh config](https://github.com/nyu-dl/cluster-support/blob/master/client/config) for details.
27+
28+
Login node should not be used to run anything related to your computations, use it for file management (`git`, `rsync`), jobs management (`srun`, `salloc`, `sbatch`).
29+
30+
## Cluster overview
31+
32+
|Number of nodes|CPU cores|GPU|RAM|
33+
|---------------|---------|---|---|
34+
|524|48|-|180G|
35+
|40|48|-|369G|
36+
|4|96|-|3014G **(3T)**|
37+
|10|48|4 x V100 32G (PCIe)|369G|
38+
|65|48|4 x **RTX8000 48G**|384G|
39+
40+
**quotas**:
41+
42+
|filesystem|env var|what for|flushed|quota|
43+
|----------|-------|--------|-------|-----|
44+
|/archive|$ARCHIVE|long term storage|NO|2TB/20K inodes|
45+
|/home|$HOME|probably nothing|NO|50GB/30K inodes|
46+
|/scratch|$SCRATCH|experiments/stuff|YES (60 days)|5TB/1M inodes|
47+
48+
```
49+
echo $SCRATCH
50+
/scratch/ik1147
51+
```
52+
53+
### Differences with Prince
54+
55+
* Strong preference **for Singularity** / not modules.
56+
* GPU specification through `--gres=gpu:{rtx8000|v100}:1` (Prince uses/used partitions)
57+
* Low inode quota on home fs (~30k files)
58+
* No beegfs
59+
60+
## Singularity
61+
62+
One can read white paper here: https://arxiv.org/pdf/1709.10140.pdf
63+
64+
The main idea of using a container is to provide an isolated user space on a compute node and to simplify the node management (security and updates).
65+
66+
### Getting a container image
67+
68+
Each singularity container has a definition file (`.def`) and the image file (`.sif`). Lets consider the following container:
69+
70+
`/scratch/work/public/singularity/cuda10.1-cudnn7-devel-ubuntu18.04-20201207.def`
71+
`/scratch/work/public/singularity/cuda10.1-cudnn7-devel-ubuntu18.04-20201207.sif`
72+
73+
Definiton file contains all commands which are executed along the way when the image OS is created, take a look!
74+
75+
This particular image will have CUDA 10.1 with cudnn 7 libs within Ubuntu 18.04 OS.
76+
77+
Lets execute this container with singularity:
78+
79+
```bash
80+
[ik1147@log-2 ~]$ singularity exec /scratch/work/public/singularity/cuda10.1-cudnn7-devel-ubuntu18.04-20201207.sif /bin/bash
81+
Singularity> uname -a
82+
Linux log-2.nyu.cluster 4.18.0-193.28.1.el8_2.x86_64 #1 SMP Fri Oct 16 13:38:49 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
83+
Singularity> lsb_release -a
84+
LSB Version: core-9.20170808ubuntu1-noarch:security-9.20170808ubuntu1-noarch
85+
Distributor ID: Ubuntu
86+
Description: Ubuntu 18.04.5 LTS
87+
Release: 18.04
88+
Codename: bionic
89+
```
90+
91+
Notice that we are still on the login node but within the Ubuntu container!
92+
93+
One can find more containers here:
94+
95+
`/scratch/work/public/singularity/`
96+
97+
Container files are read-only images, i.e. there is no need to copy over the container. Instead one may create a symlink for convenience. Please contact HPC if you need a container image with some specific software.
98+
99+
### Setting up an overlay filesystem with your computing environment
100+
101+
**Why?** The whole reason of using Singularity with overlay filesystem is *to reduce the impact on scratch filesystem* (remember how slow is it on Prince sometimes?).
102+
103+
**High-level idea**: make a read-only filesystem which will be used exclusively to host your conda environment and other static files which you constantly re-use for each job.
104+
105+
**How is it different from scratch?** overlayfs is a separate fs mounted on each compute node when you start the container while scratch is a shared GPFS accessed via network.
106+
107+
There are two different modes when mounting the overlayfs:
108+
* read-write: use this one when setting up env (installing conda, libs, other static files)
109+
* read-only: use this one when running your jobs. It has to be read-only since multiple processes will access the same image. It will crash if any job has already mounted it as read-write.
110+
111+
Setting up your fs image:
112+
1. Copy the empty fs gzip to your scratch path (e.g. `/scratch/<NETID>/` or `$SCRATCH` for your root scratch): `cp /scratch/work/public/overlay-fs-ext3/overlay-50G-10M.ext3.gz $SCRATCH/`
113+
2. Unzip the archive: `gunzip -v $SCRATCH/overlay-50G-10M.ext3.gz` (can take a while to unzip...)
114+
3. Execute container with overlayfs (check comment below about `rw` arg): `singularity exec --overlay $SCRATCH/overlay-50G-10M.ext3:rw /scratch/work/public/singularity/cuda10.1-cudnn7-devel-ubuntu18.04-20201207.sif /bin/bash`
115+
4. Check file systems: `df -h`. There will be a record: `overlay 53G 52M 50G 1% /`. The size equals to the filesystem image you chose. **The actual content of the image is mounted in `/ext3`.**
116+
5. Create a file in overlayfs: `touch /ext3/testfile`
117+
6. Exit from Singularity
118+
119+
One has permission for file creation since the fs was mounted with `rw` arg. In contrast `ro` will mount it as read-only.
120+
121+
Setting up conda environment:
122+
123+
1. Start a CPU (GPU if you want/need) job: `srun --nodes=1 --tasks-per-node=1 --cpus-per-task=1 --mem=32GB --time=1:00:00 --gres=gpu:1 --pty /bin/bash`
124+
2. Start singularity (notice `--nv` for GPU propagation): `singularity exec --nv --overlay $SCRATCH/overlay-50G-10M.ext3:rw /scratch/work/public/singularity/cuda10.1-cudnn7-devel-ubuntu18.04-20201207.sif /bin/bash`
125+
3. Install your conda env in `/ext3`: https://github.com/nyu-dl/cluster-support/tree/master/cassio#conda-environment.
126+
3.1. `wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh`
127+
3.2. `bash ./Miniconda3-latest-Linux-x86_64.sh -b -p /ext3/miniconda3`
128+
3.3. Install packages you are using (torch, jupyter, tensorflow etc.)
129+
4. Exit singularity container (*not the CPU/GPU job*)
130+
131+
By now you have a working conda environment located in a ext3 image filesystem here: `$SCRATCH/overlay-50G-10M.ext3`.
132+
133+
Let's run the container with **read-only** layerfs now (notice `ro` arg there):
134+
`singularity exec --nv --overlay $SCRATCH/overlay-50G-10M.ext3:ro /scratch/work/public/singularity/cuda10.1-cudnn7-devel-ubuntu18.04-20201207.sif /bin/bash`
135+
136+
```bash
137+
Singularity> conda activate
138+
(base) Singularity> python
139+
Python 3.8.5 (default, Sep 4 2020, 07:30:14)
140+
[GCC 7.3.0] :: Anaconda, Inc. on linux
141+
Type "help", "copyright", "credits" or "license" for more information.
142+
>>> import torch
143+
>>> torch.__version__
144+
'1.7.0'
145+
>>>
146+
(base) Singularity> touch /ext3/test.txt
147+
touch: cannot touch '/ext3/test.txt': Read-only file system
148+
```
149+
150+
As shown above, the conda env works fine while `/ext3` is not writable.
151+
152+
**Caution:** by default conda and python keep cache in home: `~/.conda`, `~/.cache`. Since home fs has small quota, move your cache folder to scratch:
153+
1. `mkdir $SCRATCH/python_cache`
154+
2. `cd`
155+
3. `ln -s $SCRATCH/python_cache/.cache`
156+
157+
```
158+
.cache -> /scratch/ik1147/cache/
159+
```
160+
161+
***From now on you may run your interactive coding/debugging sessions, congratulations!***
162+
163+
## Running a batch job with Singularity
164+
165+
files to use:
166+
* `gpu_job.slurm`
167+
* `test_gpu.py`
168+
169+
There is one major detail with a batch job submission: sbatch script starts the singularity container with a `/bin/bash -c "<COMMAND>"` tail, where `"<COMMAND>"` is whatever your job is running.
170+
171+
In addition, conda has to be sourced and activated.
172+
173+
Lets create helper script for conda activation: copy the code below in `/ext3/env.sh`
174+
175+
```bash=
176+
#!/bin/bash
177+
178+
source /ext3/miniconda3/etc/profile.d/conda.sh
179+
export PATH=/ext3/miniconda3/bin:$PATH
180+
```
181+
182+
This is an example batch job submission script (also as a file `gpu_job.slurm` in this repo folder):
183+
184+
```bash=
185+
#!/bin/bash
186+
#SBATCH --job-name=job_wgpu
187+
#SBATCH --open-mode=append
188+
#SBATCH --output=./%j_%x.out
189+
#SBATCH --error=./%j_%x.err
190+
#SBATCH --export=ALL
191+
#SBATCH --time=00:10:00
192+
#SBATCH --gres=gpu:1
193+
#SBATCH --mem=64G
194+
#SBATCH -c 4
195+
196+
singularity exec --nv --overlay $SCRATCH/overlay-50G-10M.ext3:ro /scratch/work/public/singularity/cuda10.1-cudnn7-devel-ubuntu18.04-20201207.sif /bin/bash -c "
197+
198+
source /ext3/env.sh
199+
conda activate
200+
201+
python ./test_gpu.py
202+
"
203+
```
204+
205+
the output:
206+
```
207+
Torch cuda available: True
208+
GPU name: Quadro RTX 8000
209+
210+
211+
CPU matmul elapsed: 0.438138484954834 sec.
212+
GPU matmul elapsed: 0.2669832706451416 sec.
213+
```
214+
215+
***From now on you may run your experiments as a batch job, congratulations!***
216+
217+
218+
## Setting up a simple sweep over a hyper-parameter using Slurm array job
219+
220+
files to use:
221+
* `sweep_job.slurm`
222+
* `test_sweep.py`
223+
224+
Usually we may want to do multiple runs of the same job using different hyper params or learning rates etc. There are many cool frameworks to do that like Pytorch Lightning etc. Now we will look on the simplest (imo) version of such sweep construction.
225+
226+
**High-level idea:** define a sweep over needed arguments and create a product as a list of all possible combinations (check `test_sweep.py` for details). In the end a specific config is mapped to its position in the list. The SLURM array id is used as a map to specific config in the sweep (check `sweep_job.slurm`).
227+
228+
Notes:
229+
* notice no `--nv` in singularity call because we allocate CPU-only resources.
230+
* notice `$SLURM_ARRAY_TASK_ID`. For each specific step job (in range `#SBATCH --array=1-20`) this env var will have the corresponding value assigned.
231+
232+
Submitting a sweep job:
233+
`sbatch sweep_job.slurm`
234+
235+
Outputs from all completed jobs:
236+
```
237+
cat *out
238+
{'sweep_step': 20, 'seed': 682, 'device': 'cpu', 'lr': 0.01, 'model_size': 1024, 'some_fixed_arg': 0}
239+
{'sweep_step': 1, 'seed': 936, 'device': 'cpu', 'lr': 0.1, 'model_size': 512, 'some_fixed_arg': 0}
240+
{'sweep_step': 2, 'seed': 492, 'device': 'cpu', 'lr': 0.1, 'model_size': 512, 'some_fixed_arg': 0}
241+
{'sweep_step': 3, 'seed': 52, 'device': 'cpu', 'lr': 0.1, 'model_size': 512, 'some_fixed_arg': 0}
242+
{'sweep_step': 4, 'seed': 691, 'device': 'cpu', 'lr': 0.1, 'model_size': 512, 'some_fixed_arg': 0}
243+
{'sweep_step': 5, 'seed': 826, 'device': 'cpu', 'lr': 0.1, 'model_size': 512, 'some_fixed_arg': 0}
244+
{'sweep_step': 6, 'seed': 152, 'device': 'cpu', 'lr': 0.1, 'model_size': 1024, 'some_fixed_arg': 0}
245+
{'sweep_step': 7, 'seed': 626, 'device': 'cpu', 'lr': 0.1, 'model_size': 1024, 'some_fixed_arg': 0}
246+
{'sweep_step': 8, 'seed': 495, 'device': 'cpu', 'lr': 0.1, 'model_size': 1024, 'some_fixed_arg': 0}
247+
{'sweep_step': 9, 'seed': 703, 'device': 'cpu', 'lr': 0.1, 'model_size': 1024, 'some_fixed_arg': 0}
248+
{'sweep_step': 10, 'seed': 911, 'device': 'cpu', 'lr': 0.1, 'model_size': 1024, 'some_fixed_arg': 0}
249+
{'sweep_step': 11, 'seed': 362, 'device': 'cpu', 'lr': 0.01, 'model_size': 512, 'some_fixed_arg': 0}
250+
{'sweep_step': 12, 'seed': 481, 'device': 'cpu', 'lr': 0.01, 'model_size': 512, 'some_fixed_arg': 0}
251+
{'sweep_step': 13, 'seed': 618, 'device': 'cpu', 'lr': 0.01, 'model_size': 512, 'some_fixed_arg': 0}
252+
{'sweep_step': 14, 'seed': 449, 'device': 'cpu', 'lr': 0.01, 'model_size': 512, 'some_fixed_arg': 0}
253+
{'sweep_step': 15, 'seed': 840, 'device': 'cpu', 'lr': 0.01, 'model_size': 512, 'some_fixed_arg': 0}
254+
{'sweep_step': 16, 'seed': 304, 'device': 'cpu', 'lr': 0.01, 'model_size': 1024, 'some_fixed_arg': 0}
255+
{'sweep_step': 17, 'seed': 499, 'device': 'cpu', 'lr': 0.01, 'model_size': 1024, 'some_fixed_arg': 0}
256+
{'sweep_step': 18, 'seed': 429, 'device': 'cpu', 'lr': 0.01, 'model_size': 1024, 'some_fixed_arg': 0}
257+
{'sweep_step': 19, 'seed': 932, 'device': 'cpu', 'lr': 0.01, 'model_size': 1024, 'some_fixed_arg': 0}
258+
```
259+
260+
**Caution:** be careful with checkpointing when running a sweep. Make sure that your checkpoints/logs have corresponding sweep step in the filename or so (to avoid file clashes).
261+
262+
***From now on you may run your own sweeps, congratulations!***
263+
264+
## Port forwarding to Jupyter Lab
265+
266+
### Part 1: launching JupyterLab on Greene.
267+
268+
1. Launch interactive job with a gpu: `srun --nodes=1 --tasks-per-node=1 --cpus-per-task=1 --mem=32GB --time=1:00:00 --gres=gpu:1 --pty /bin/bash`
269+
2. Execute singularity container: `singularity exec --nv --overlay $SCRATCH/overlay-50G-10M.ext3:ro /scratch/work/public/singularity/cuda10.1-cudnn7-devel-ubuntu18.04-20201207.sif /bin/bash`
270+
3. Activate conda (base env in this case): `conda activate`
271+
4. Start jupyter lab: `jupyter lab --ip 0.0.0.0 --port 8965 --no-browser`
272+
273+
Expected output:
274+
275+
```
276+
[I 15:51:47.644 LabApp] JupyterLab extension loaded from /ext3/miniconda3/lib/python3.8/site-packages/jupyterlab
277+
[I 15:51:47.644 LabApp] JupyterLab application directory is /ext3/miniconda3/share/jupyter/lab
278+
[I 15:51:47.646 LabApp] Serving notebooks from local directory: /home/ik1147
279+
[I 15:51:47.646 LabApp] Jupyter Notebook 6.1.4 is running at:
280+
[I 15:51:47.646 LabApp] http://gr031.nyu.cluster:8965/
281+
[I 15:51:47.646 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
282+
```
283+
284+
Note the hostname of the node we run from here: `http://gr031.nyu.cluster:8965/`
285+
286+
### Part 2: Forwarding the connection from you local client to Greene.
287+
288+
[Check here the ssh config](https://github.com/nyu-dl/cluster-support/blob/master/client/config) with the greene host to make the port forwarding easier to setup.
289+
290+
To start the tunnel, run: `ssh -L 8965:gr031.nyu.cluster:8965 greene -N`
291+
292+
**Note the hostname should be equal to the node where jupyter is running. Same with the port.**
293+
294+
Press Ctrl-C if you want to stop the port forwarding SSH connection.
295+
296+
***From now on you may run your own Jupyter Lab with RTX8000/V100, congratulations!***

‎greene/gpu_job.slurm‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
#!/bin/bash
2+
#SBATCH --job-name=job_wgpu
3+
#SBATCH --open-mode=append
4+
#SBATCH --output=./%j_%x.out
5+
#SBATCH --error=./%j_%x.err
6+
#SBATCH --export=ALL
7+
#SBATCH --time=00:10:00
8+
#SBATCH --gres=gpu:1
9+
#SBATCH --mem=64G
10+
#SBATCH -c 4
11+
12+
singularity exec --nv --overlay $SCRATCH/overlay-50G-10M.ext3:ro /scratch/work/public/singularity/cuda10.1-cudnn7-devel-ubuntu18.04-20201207.sif /bin/bash -c "
13+
14+
source /ext3/env.sh
15+
conda activate
16+
17+
python ./test_gpu.py
18+
"

‎greene/sweep_job.slurm‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
#!/bin/bash
2+
#SBATCH --job-name=job_wgpu
3+
#SBATCH --open-mode=append
4+
#SBATCH --output=./%j_%x.out
5+
#SBATCH --error=./%j_%x.err
6+
#SBATCH --export=ALL
7+
#SBATCH --time=00:10:00
8+
#SBATCH --mem=1G
9+
#SBATCH -c 4
10+
11+
#SBATCH --array=1-20
12+
13+
singularity exec --overlay $SCRATCH/overlay-50G-10M.ext3:ro /scratch/work/public/singularity/cuda10.1-cudnn7-devel-ubuntu18.04-20201207.sif /bin/bash -c "
14+
15+
source /ext3/env.sh
16+
conda activate
17+
18+
python ./test_sweep.py $SLURM_ARRAY_TASK_ID
19+
"

‎greene/test_gpu.py‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
import torch
2+
import time
3+
4+
if __name__ == "__main__":
5+
6+
print(f"Torch cuda available: {torch.cuda.is_available()}")
7+
print(f"GPU name: {torch.cuda.get_device_name()}\n\n")
8+
9+
t1 = torch.randn(100, 1000)
10+
t2 = torch.randn(1000, 10000)
11+
12+
cpu_start = time.time()
13+
14+
for i in range(100):
15+
t = t1 @ t2
16+
17+
cpu_end = time.time()
18+
19+
print(f"CPU matmul elapsed: {cpu_end-cpu_start} sec.")
20+
21+
t1 = t1.to("cuda")
22+
t2 = t2.to("cuda")
23+
24+
gpu_start = time.time()
25+
26+
for i in range(100):
27+
t = t1 @ t2
28+
29+
gpu_end = time.time()
30+
31+
print(f"GPU matmul elapsed: {gpu_end-gpu_start} sec.")

0 commit comments

Comments
 (0)