Jade has 2.1‑GHz quad‑core AMD Opteron processors on all 2152 compute nodes (8604 cores), and 2.8‑GHz dual‑core AMD Opteron processors on its login and other service nodes.
Compute nodes on jade use Compute Node Linux (CNL) as the OS, instead of Catamount. CNL allows the compute nodes to spawn threads, making OpenMP a valid parallel programming model for use on jade. Since the shared memory on jade is on each node, the most efficient approach to using OpenMP is to use four threads.
The PBS option for the number of cores requested changed from "PBS -l size=#" on sapphire, to "PBS -l ncpus=#" on jade.
CRAY's XT4 PBS options "mppwidth", "mppdepth", and "mppnppn" have been replaced with "ncpus" on jade to make the PBS options consistent with other HPC architectures' PBS batch schedulers.
Below is a quick list of steps to get up and running on the Cray XT4 (jade).
Jade can be accessed via Kerberized ssh as follows:
ssh login_node.erdc.hpc.mil
where login_node is one of jade01 through jade06.
You may also connect using telnet or rlogin. For security purposes, you must have a current Kerberos ticket on your computer before attempting to connect.
Each user has file space available in $HOME and $WORKDIR (these are environment variables that point to the appropriate locations in the corresponding file systems). Users are strongly encouraged to use the predefined environment variables $HOME and $WORKDIR in their scripts.
Even though the working directory on jade is 379 TBytes, that space can be consumed very quickly. Users are urged to keep the amount of data in their $WORKDIR directory to a minimum.
Use Kerberized ftp to transfer necessary files to and from jade. Since available storage is limited in your home directory, large data files should be placed in the $WORKDIR directory.
If you need to move data between machines at the ERDC DSRC, please use the ftp command.
The compilers available on jade are PGI (the default compiler), PathScale, and GNU. These are invoked using the familiar ftn, f77, cc, and CC commands. These commands are actually scripts that automatically use the compiler specified by the modules that are loaded. They automatically access the correct system include files and libraries such as MPI, LibSci, and Lustre. We highly recommend that you do not try to include any other libraries except for those that you build yourself or the DSRC‑installed libraries in /usr/local/usp or /usr/local/applic.
Executables compiled using these scripts will execute on the compute nodes. For codes using SHMEM, add the "-lsma" option to the compile command. For codes using OpenMP, add the "-mp=nonuma" option for PGI compiles, the "-mp" option for PathScale compiles, and the "-fopenmp" option for GNU compiles.
For small applications that run on the login nodes, you can use the pgf90, pgf77, pgcc, or pgCC commands. These applications must complete in less than 10 minutes and use less than 2 GBytes of memory unless you use PBS to schedule one of the interactive batch nodes.
The Portable Batch System (PBS) is the batch system for jade.
To submit a batch job, use the following command:
qsub scriptname
where scriptname is the name of the file containing the batch script. For more information, see the PBS Quick Reference Guide.
Your job script must contain a subproject ID against which the run time is to be charged. Use the show_usage command to generate a list of your current subprojects. For more information on qsub or show_usage, see their respective man pages.
Jade has a batch queue structure essentially identical to sapphire.
| Queue Name | Max Wall Time | Max Cores per Job |
Priority |
|---|---|---|---|
| urgent | 168:00:00 | 2048 | 50 |
| debug | 1:00:00 | 256 | 40 |
| high | 168:00:00 | 2048 | 35 |
| challenge | 168:00:00 | 2048 | 30 |
| standard | 48:00:00 | 2048 | 20 |
| special | 168:00:00 | 256 | 15 |
| background | 4:00:00 | 2048 | 10 |
You can monitor batch jobs on jade by using the qstat, qview, and qhist commands.
You can cancel a job by using "qdel ID", where ID is obtained from the qstat command.
The commands qstat, qview, and qlim let you to see the status of the queues.
When your job is finished running, you will want to save any important data. Files in $WORKDIR may be purged, so any important data should be copied to the Data Management System (DMS).
Files can be copied to the DMS using the following command:
archive put -C $ARCHIVE_HOME/subproject filename
and can be retrieved from the DMS using the following command:
archive get -C $ARCHIVE_HOME/subproject filename
For more information, see the archive man page or go to the PST Web site 
.
For available software packages on jade, see the
Software Version Matrix.
Last update: January 04, 2010
You are accessing a U.S. Government (USG) Information System (IS) that is provided for USG-authorized use only. By using this IS (which includes any device attached to this IS), you consent to the following conditions: * The USG routinely intercepts and monitors communications on this IS for purposes including, but not limited to, penetration testing, COMSEC monitoring, network operations and defense, personnel misconduct (PM), law enforcement (LE), and counterintelligence (CI) investigations. * At any time, the USG may inspect and seize data stored on this IS. * Communications using, or data stored on, this IS are not private, are subject to routine monitoring, interception, and search, and may be disclosed or used for any USG- authorized purpose. * This IS includes security measures (e.g., authentication and access controls) to protect USG interests--not for your personal benefit or privacy. * Not withstanding the above, using this IS does not constitute consent to PM, LE or CI investigative searching or monitoring of the content of privileged communications, or work product, related to personal representation or services by attorneys, psychotherapists, or clergy, and their assistants. Such communications and work product are private and confidential.
