MLDAS Wrapper ¶
Making things easier ¶
In order to ease the data analysis workflow, a shell script wrapper was written which can handle a variety of operations (e.g. software installation, file manipulation, quick plotting of results, custom phase-weighted stack). Most of the operations can also be run in parallel using the
--mpi
option which will automatically execute an
sbatch
job with the appropriate SLURM directives and submit it to the supercomputer of your choice. In this page we will explain in details how the wrapper works and to execute it.
weight not done through mpi4py because hdf5 issue opening multiple file within same memory process
SLURM Scripts ¶
Below:
#!/bin/bash
#SBATCH -n 60
#SBATCH -p lr3
#SBATCH -q lr_normal
#SBATCH -A ac_ciftres
#SBATCH -t 04:00:00
#SBATCH -J probmpi
#SBATCH -o logs/%x-%j.out
mkdir -p /global/scratch/vdumont/probmaps
cd /global/scratch/vdumont/probmaps
module load gcc/6.3.0
module load openmpi/3.0.1-gcc
module load python/3.6
export PYTHONPATH=/global/scratch/vdumont/myenv/lib/python3.6/site-packages/:$PYTHONPATH
mpirun -np 60 python $HOME/mldas/mldas/assess.py $HOME/mldas/configs/assess.yaml --mpi --version lrc
~/mldas/bin/quickrun.py weighting -i mat ../probmaps/ch4650_4850/50_1_10/ -o weight/50_1_10
~/mldas/scripts/mldas.sh -a weight -d mat -w ../probmaps/ch4650_4850/50_1_10/ -o weight/50_1_10
~/mldas/scripts/mldas.sh -a weight -d mat -w ../probmaps/ch4650_4850/50_1_10/ -o weight/50_1_10 -m 60