Installation

Installation#

EMU has been pre-installed on the P-Cluster for ECCO Hackathon participants to use. Therefore, readers can skip this section. Nevertheless, we provide a detailed description of how to install EMU on the P-Cluster. These instructions can serve as guidance for users to install EMU on other P-Clusters and High-Performance Computing (HPC) systems. It is expected that some level of customization will be required on top of the steps described here.

Attention

If users want to install EMU themselves, please follow the instructions provided in README_setup. Specifically, obtain emu_setup.sh from the source described in README_setup. Do not obtain files from an installed version, such as the one installed on the P-Cluster, as the file contents may differ.

Obtain Installation Script#

First, we obtain the installation script from GitHub and prepare for the installation.

cd /efs_ecco/USERNAME
mkdir EMU
cd EMU
wget https://raw.githubusercontent.com/ECCO-GROUP/ECCO-EIS/refs/heads/main/emu/emu_setup.sh

Execute Installation Script#

Then, we add executable permission to the script and run it.

chmod u+x emu_setup.sh
./emu_setup.sh

Choose Installing EMU Using Singularity Image#

There are two options to install EMU: a native installation that compiles the source code on the local machine, and a Singularity installation that uses a Singularity image. The native installation requires a TAF license to derive the MITgcm adjoint used by EMU’s Adjoint Tool, while the Singularity installation has pre-compiled executables containerized in a Singularity image and does not require a separate TAF license.

----------------------
EMU's Programs can be installed in two different ways;
   1) Compiling source code on host (native) 
   2) Using Singularity image (singularity)

Option 1) requires a TAF license to derive the MITgcm adjoint used by EMU's 
Adjoint Tool. Option 2) has compiled versions of the code in 
containerized form that do not require a separate TAF license to use.

Enter choice for type of EMU implementation ... (1 or 2)?
2

Implementation type choice is 2

Specify `sbatch` as the Command for Submitting Batch Jobs#

EMU uses batch scripts to run some of its tools and requires users to specify the command for submitting batch jobs. As described in the Getting Started with the P-Cluster tutorial, the P-Cluster uses SLURM as its batch system, and sbatch is the command we will use for submitting batch jobs.”

----------------------
EMU uses batch scripts to run some of its tools in PBS (Portable 
Batch System). The PBS commands in these shell scripts (pbs_*.sh),
installed in EMU's User Interface directory (emu_userinterface_dir)
/efs_ecco/USERNAME/EMU/emu_userinterface_dir
may need to be revised for different batch systems and/or different hosts. 
Alternatively, these shell scripts can be run interactively if sufficient 
resources are available.

Enter the command for submitting batch jobs (e.g., qsub, sbatch, 
bsub <, condor_submit, msub) or press the ENTER key to have EMU 
run its batch scripts interactively ... ?
sbatch

Command to submit EMU's batch job scripts will be: sbatch

Downloading EMU Input Files#

The next step is to choose what input files we want to download to the P-Cluster disk. The EMU menu provides a list of options for various subsets of the input files, with the complete set totaling 1.1 TB. Below is an example for downloading all input files. If you had already downloaded the input files and want to re-install EMU (e.g., due to updates in the EMU code on its GitHub repository), you can skip downloading the input files by pressing ‘Enter’.

----------------------
EMU input files will be placed in ... 
/efs_ecco/USERNAME/EMU/emu_input_dir/

----------------------
EMU's Input Files total 1.1 TB, of which (directory)
   175 GB (emu_ref) is needed by Sampling, Forward Gradient, Adjoint, Tracer, Budget, and Attribution
   195 GB (forcing) is needed by Forward Gradient, Adjoint, Modified Simultion
   380 GB (state_weekly) is needed by Tracer
   290 GB (emu_msim) is needed by Attribution
   (Convolution Tool uses results of the Adjoint Tool and files downloaded by default.)

Choose among the following to download ... 
   1) All Input Files (1.1 TB) 
   2) Files (~175 GB) needed for Sampling and Budget Tools
   3) Files (~195 GB) needed for Modified Simultion Tools
   4) Files (~370 GB) needed for Adjoint and Forward Gradient Tool
   5) Files (~465 GB) needed for Attribution Tool
   6) Files (~555 GB) needed for Tracer Tool
or press the ENTER key to skip this step, which can take a while
(~13 hours if downloading all input files.) 

EMU's Input Files can be downloaded later with shell script
   /efs_ecco/USERNAME/EMU/emu_userinterface_dir/emu_input_setup.sh 
See 
   /efs_ecco/USERNAME/EMU/emu_userinterface_dir/README_input_setup 
for additional detail, including options to download the input
in batch mode.

Enter Input Files download choice ... ?
0

Choice is 0

----------------------
Downloading EMU input interactively ... 
--2024-10-01 23:11:56--  https://ecco.jpl.nasa.gov/drive/files/Version4/Release4/input_init
...

Installing EMU Programs and Shell Scripts from GitHub#

The installation script will download EMU programs and shell scripts from GitHub and save them to the local disk (emu_dir or the directory name we entered earlier). There is no keyboard input required for this step.

----------------------
Installing EMU programs and shell scripts from GitHub to directory 
/efs_ecco/USERNAME/EMU/emu_dir

Installing EMU Singularity Image#

The script will download the EMU Singularity image (about 400 MB). Using the EMU Singularity image requires compiling a compatible version of OpenMPI, which can take roughly 30 minutes. When prompted (see below), we enter ‘1’ to compile OpenMPI.

----------------------
Installing EMU Singularity image (emu.sif) in directory 
/efs_ecco/USERNAME/EMU/emu_dir

--2024-09-30 23:08:23--  https://ecco.jpl.nasa.gov/drive/files/Version4/Release4/other/flux-forced/emu_input/emu_misc/emu.sif
Resolving ecco.jpl.nasa.gov (ecco.jpl.nasa.gov)... 128.149.52.112
Connecting to ecco.jpl.nasa.gov (ecco.jpl.nasa.gov)|128.149.52.112|:443... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Authentication selected: Basic realm="ECCO_Drive"
Reusing existing connection to ecco.jpl.nasa.gov:443.
HTTP request sent, awaiting response... 200 OK
Length: 387813376 (370M)
Saving to: ‘/efs_ecco/USERNAME/EMU/emu_dir/emu.sif’

emu.sif                                        100%[==================================================================================================>] 369.85M   109MB/s    in 3.6s    

2024-09-30 23:08:30 (103 MB/s) - ‘/efs_ecco/USERNAME/EMU/emu_dir/emu.sif’ saved [387813376/387813376]

FINISHED --2024-09-30 23:08:31--
Total wall clock time: 8.4s
Downloaded: 1 files, 370M in 3.6s (103 MB/s)

----------------------
EMU Singularity (emu.sif) requires compilation of compatible OpenMPI 
which can take ~30min. Enter 1 to compile OpenMPI now as a background 
job or press the ENTER key to skip this step. 

EMU compatible OpenMPI can be compiled later with shell script 
 /efs_ecco/USERNAME/EMU/emu_userinterface_dir/emu_openmpi_setup.sh 
See 
   /efs_ecco/USERNAME/EMU/emu_userinterface_dir/README_openmpi_setup 
for additional detail, including options to install OpenMPI
in batch mode.

Enter 1 to compile now or press the ENTER key to skip ... ?
1

OpenMPI will be downloaded and compiled.

Choose Number of CPU Cores#

EMU provides a list of options for using different numbers of CPU cores for MITgcm employed by EMU. We will choose the option of 96 CPUs by entering ‘96’.

----------------------
Enter number of CPU cores (nproc) to use for MITgcm employed by EMU.
Available options are ... 
13
36
48
68
72
96
192
360
 
Enter choice for nproc ... ?
96

Number of CPU cores to be used for MITgcm: 96

**********************
 End of user input for EMU setup 
 Rest of this script is conducted without user input.

Finishing up Installing#

We are now at the end of the user input for EMU setup. The script is finishing up by installing EMU’s user interface and compiling the EMU-compatible OpenMPI. This step takes roughly 30 minutes to complete.

----------------------
Installing EMU's User Interface in 
/efs_ecco/USERNAME/EMU/emu_userinterface_dir
Progress can be monitored in file  /efs_ecco/USERNAME/EMU/emu_dir/temp_setup/install_emu_access.log

----------------------
Downloading EMU's Input Files pid is 1234567

----------------------
Download and compiling EMU compatible OpenMPI in the background in 
/efs_ecco/USERNAME/EMU/emu_dir/ompi

This can take a while (~30 minutes). 
Progress can be monitored in file  /efs_ecco/USERNAME/EMU/emu_dir/temp_setup/emu_openmpi_setup.log
  tail /efs_ecco/USERNAME/EMU/emu_dir/temp_setup/emu_openmpi_setup.log 

Download and compiling OpenMPI pid is  1315569

----------------------
 Waiting for completion of background jobs ... 
  pid 1315569

***********************************
 Do not terminate this script until it completes on its own. 
***********************************

EMU compatible OpenMPI setup is still running.
pid is 1315569
Check progress in log file /efs_ecco/USERNAME/EMU/emu_dir/temp_setup/emu_openmpi_setup.log
Waiting for this job to finish...

EMU compatible OpenMPI setup has finished.

***********************************
Download EMU Input Files skipped during present setup with shell script
 /efs_ecco/USERNAME/EMU/emu_userinterface_dir/emu_input_setup.sh

Upon completion, EMU can be run by entering command
   /efs_ecco/USERNAME/EMU/emu_userinterface_dir/emu

See
   /efs_ecco/USERNAME/EMU/emu_userinterface_dir/README
for a brief description, including tools for interactively
reading and plotting EMU's results.
***********************************

----------------------
emu_setup.sh execution complete. Mon Sep 30 23:35:22 UTC 2024
Elapsed time: 0:30:23
----------------------