GipsyX

Important License Information

The license for using this software is restrictive. As part of the terms of the license it is only available to select HPC projects on Comet. If you need to use this software and you are not a member of a project which already has access, then please contact us first.

GipsyX is developed by the Jet Propulsion Laboratory (JPL), and maintained by the Near Earth Tracking Applications and Systems groups. It replaces the GIPSY-OASIS (GNSS-Inferred Positioning System and Orbit Analysis Simulation Software) software.

Features:

Analysis of data from Global Navigation and Satellite Systems (GNSS):
- Global Positioning System (GPS)
- Russian GLONASS
- French DORIS
- Satellite Laser Ranging (SLR) system
Precise centimeter-level GNSS-based positioning and timing:
- Space platforms, including low-Earth orbiters and GNSS constellations
- Airplanes
- Terrestrial (ground) stations, static or moving
- Time transfer
Single high-level user interface supports majority of precise positioning applications
Single-receiver ambiguity resolution using JPL's GPS orbit and clock products

Applications:

Terrestrial positioning for geophysical research:
- Earth deformation, plate tectonics
- GNSS Networks (e.g. PBO and SCIGN)
- Ice flow
- Studying troposphere and ionosphere
- Reference frame (geocenter and scale) and Earth rotation parameters
Airplane positioning
Precise orbit determination
- Low-Earth orbiters: Topex/Poseidon, Jason-1, Jason-2/OSTM, Jason-3, Sentinel-6, NISAR, GRACE, GRACE-FO, Champ, SAC-C, COSMIC, COSMIC-2, DSAC, Space Shuttle (e.g. SRTM), Commercial Satellites
- GNSS constellations including GPS, GLONASS, Galileo, Beidou, QZSS, and IRNSS

For further information: https://gipsyx.jpl.nasa.gov/

Running GipsyX on Comet

GipsyX has been installed on Comet as an application container. This allows the software to be fully restricted to the HPC project group(s) who are allowed access as part of the terms of the license. It also allows the staff in those groups to update the container to new software versions as-and-when required without any superuser intervention.

The GipsyX container (by default named as gipsyx.sif) should be located in the project folder of only those HPC project groups which are licensed to use it.

All of the GipsyX tools are installed to the location /opt/gipsyx inside the container and all of the correct environment variables (PATH, GIPSYX, GOA_VAR, etc) have been set to point to this installation location.

To make running the GipsyX tools easier, we have provided a runtime helper script (by default named gipsyx.sh) which aims to simplify accessing the tools, allows mapping of an external goa-var configuration directory as well as reading/writing to your $HOME directory and /nobackup project folders. After sourcing the script, you will have a new command container.run that will run any tool or script from inside the container.

For example, to run the g2de tool from GipsyX, assuming that your project folder is named /nobackup/proj/PROJECT_NAME/, run the following:

$ source /nobackup/proj/PROJECT_NAME/gipsyx.sh
$ container.run gd2e.py --help
INFO:    gocryptfs not found, will not be able to use gocryptfs
usage: gd2e.py [-h] [-treeSequenceDir dir_of_trees] [-subIterations n [n ...]] [-addIteration n] [-runType {LEO,PPP,GNSSPRTL}] [-xmtPos] [-geop GEOP] [-gdCov]
               [-startTime startTime [startTime ...]] [-endTime endTime [endTime ...]] [-dataEditor "cmd dataRecordFile.gz"] [-drEditedFile edited.dr.gz] [-quatLeo leo.quat] [-initLeo leo.pos]
               [-trajTree trajedy.tree] [-noTrajFit] [-GNSSproducts dir|url] [-prodTypeGNSS nf|nnr|fid] [-HighRateProducts] [-orbClkDir [GNSSinitValues]] [-selectGnss pattern]
               [-rnxFile rinexFile] [-staDb StaDb] [-tdpInput tdpInput] [-antexFile f.atx] [-requireAntexPhaseCal {G,E,R,C} [{G,E,R,C} ...]] [-xyzFile f.xyz] [-ammFile AntennaMapModel]
               [-ammXmtFile AntennaMapModel_transmitters] [-gnssAdjustedProducts dir] [-selectGnssAdjusted pattern] [-recList [RecNm SAT|STA ...]] [-nProcessors np] [-keepIterFiles]
               [-mkTreeSeq [treeSeqDir]]

gd2e = Geodetic Data To Estimate

Beginnings of a script to iterate through geodetic (nominally GNSS) data (e.g. GPS) and estimate
various parameters. Initial use case includes Low Earth Orbiter data and static
point positioning.
...
...

As a reminder:

gipsyx.sif - the bare operating system container (Ubuntu LTS) with GipsyX, Python and all dependencies
gipsyx.sh - a runtime helper script which provides the container.run command to run any of the tools inside the container
goa-var - an external directory in your project folder which contains the goa-var configuration data necessary to run many of the GipsyX tools; this is intended to be maintained/updated by your group.

Running GipsyX verification

By using the runtime helper wrapper script as provided, you can run the GipsyX verification tests by a source of the runtime script and then calling container.test; a simple macro which in turn calls the GipsyX verify.py utility.

Note: Replace PROJECT_NAME in the path below with name of the project folder that your gipsyx.sh runtime helper, gipsyx.sif container file and goa-var directory are located in.

$ source /nobackup/proj/PROJECT_NAME/gipsyx.sh
$ container.test
Found overlay.img
INFO:    gocryptfs not found, will not be able to use gocryptfs
GipsyX VERIFY TESTS
Determining system setup....

Platform:
 Linux-5.14.0-427.91.1.el9_4.x86_64-x86_64-with-glibc2.35

Python executables
 python          : not found 
 python3         : /usr/bin/python3 
                 : Python 3.10.12 

Environment Variables:
 LANG            : en_GB.UTF-8
 PATH            : /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gipsyx/bin
 LD_LIBRARY_PATH : /.singularity.d/libs
 GOA_VAR         : /opt/goa-var
 GCORE           : /opt/gipsyx
 GNSSTESTDATA    : /opt/gipsyx/testData
 GCOREBUILD      : /opt/gipsyx
 OPAL_PREFIX     : /opt/gipsyx/lib/openmpi
 PYTHONPATH      : /opt/gipsyx/lib/python:

Running unit Tests...
STARTING G+E gd2e point positioning
SUCCESS  G+E gd2e point positioning     Runtime: 0:00:36.986917
STARTING gd2e LEO orbit determination
SUCCESS  gd2e LEO orbit determination   Runtime: 0:01:50.242200
STARTING GPS-only gd2e point positioning
SUCCESS  GPS-only gd2e point positioning Runtime: 0:00:08.342260
STARTING net*.py stacov manipulation
SUCCESS  net*.py stacov manipulation    Runtime: 0:00:19.584694
 4 of  4 tests pass.
All tests passed! May your data be good!

Sample GipsyX Slurm Job

This is an example of a very simple batch file which can be used to run the GipsyX container application under Slurm.

In the example below we:

Use the account code comet_abc123 - you should substitute with your own account code
Use the default_free partition - you should choose the correct partition for your needs
Allocate 8 CPU cores
Allocate 32 gigabytes of RAM
Allow up to 1 hour and 30 minutes of runtime

Run two different GipsyX tools (gd2e.py and tdpMerge.py) - this is to demonstrate that we can call the tools as many times as we want, after the initial source of the runtime helper script.

#!/bin/bash

#SBATCH --account=comet_abc123
#SBATCH --partition=default_free
#SBATCH -c 8
#SBATCH --mem=32G
#SBATCH --time 01:30:00

source /nobackup/proj/PROJECT_NAME/gipsyx.sh
container.run gd2e.py arg1 arg2 arg3

container.run tdpMerge.py -infFiles file1.tdp -outFile out

Save the script as gipsyx_sbatch.sh and submit as:

$ sbatch gipsyx_sbatch.sh

Running GipsyX graphical tools on Comet

To be written

Accessing Data

While running GipsyX you have two different filesystems to consider: inside the container and outside the container. As long as you use the container.run helper you will automatically get access to most of the directories you need while GipsyX is running and will usually not need to do anything special to access your input and output files.

If one of the GipsyX commands needs to import or export data, you must remember that paths such as '/, /usr, /usr/local, /bin, or similar all refer to directories inside the container. The three directories:

/scratch
/nobackup
$HOME (or /mnt/nfs/home)

… all refer to real directories outside the container on Comet, so restrict your reading/writing data to those three locations only.

In addition the directory inside the container:

/opt/goa-var

Is actually a pointer to a real directory within your HPC group's project directory (i.e. under /nobackup/proj/PROJECT_NAME) - this is dependent on who set up your GipsyX installation and runtime helper script.

If you use any of the graphical tools in GipsyX, and they offer to browse/open files or output directories on your behalf, remember that only those three locations actually refer to real Comet filesystem directories, anything else it shows you in the file select dialogue window refers to folders inside the container.

Building GipsyX on Comet

Important!

This section is only relevant to RSE HPC support staff or users who want to understand how GipsyX has been installed. If you only want to use the software, stop reading now.

This section is still in development!!!

You will need:

GipsyX container definition file (named gipsyx.def), listed below - only needed to build the container, not to run it
GipsyX runtime helper (named gipsyx.sh), listed below - your users source this file to run GipsyX
GipsyX build script (named gipsyx.build.sh), listed below - only needed to build the container, not to run it
GipsyX installation tarball (named gipsyx.tar.gz) - this must be provided by a user licensed to download the software

The output will be a single file named gipsyx.sif, containing the GipsyX software and all runtime dependencies. This should be copied to a location only accessible to members of the allowed HPC project group. We therefore suggest the use of your /nobackup/proj/PROJECT_NAME folder.

You must have also already extracted the goa-var-gipsyx.tgz file to your project folder (again, this must be provided by a user licensed to download the software). The build script and runtime helper will refuse to run if this is not found.

Container Definition:

This file should be named gipsyx.def.

Bootstrap: docker
From: ubuntu:jammy

####################################################################
# GipsyX Container
# ==================
# This is a runtime environment for GipsyX
# Please see: 
#	https://hpc.researchcomputing.ncl.ac.uk/dokuwiki/dokuwiki/doku.php?id=advanced:software:gipsyx
#
####################################################################
%files
	gipsyx.tgz /opt

%post
    # Prevent interactive prompts
    export DEBIAN_FRONTEND=noninteractive

####################################################################
# Basic system packages
####################################################################

	# Update & install only necessary packages
	apt-get update
	apt-get install -y aptitude wget unzip 

	# GipsyX deps
	apt-get install -y \
		python3 \
		python3-pip \
		libpciaccess0 \
		libnl-route-3-200 \
		libgfortran5 \
		libquadmath0 \
		libibverbs1 \
		python3-numpy \
		python3-bs4 \
		python3-requests \
		python3-matplotlib \
		python3-scipy \
		python3-boto \
		python3-cartopy \
		python3-distro \
		python3-h5py \
		python3-pandas
	
	# Clean up APT cache to save space
	apt-get clean 

	# Unpack GipsyX tarball
	# =====================
	# If the version of GipsyX changes, then this is the only
	# place you need to change it - referencing the directory
	# name inside the tarball itself.

	cd /opt
	tar -zxf /opt/gipsyx.tgz
	ln -s /opt/GipsyX-2.5 /opt/gipsyx
	
	# This is an empty directory where the externally mounted
	# GOA_VAR tarfile is unpacked to... those files *do not*
	# live inside the container, so as to allow them to be
	# updated as-and-when required.
	mkdir -p /opt/goa-var

	# Fix GOA_VAR in various GipsyX scripts
	# as we are *not* installing in a home directory.
	export GOA_VAR=/opt/goa-var
    export GIPSYX=/opt/gipsyx
    export GCORE=$GIPSYX
    export GCOREBUILD=$GIPSYX
    export GNSSTESTDATA=$GIPSYX/testData
    export GDATA=$GNSSTESTDATA/gcore
    export PYTHONPATH=$GIPSYX/lib/python:$PYTHONPATH
	export OPAL_PREFIX=$GIPSYX/lib/openmpi
	cd /opt/gipsyx
	./fix_rc_GipsyX.py -v $GOA_VAR

	# Remove all src packages and tarballs
	cd /
	rm -rf /src
	rm -f /opt/*.tgz

####################################################################
# Additional environment variables
####################################################################
%environment
	export PATH=$PATH:/opt/gipsyx/bin

	# GOA_VAR should be mapped to a directory using the 'bind' 
	# flag at runtime so that it is mounted outside the container 
	# and can be kept updated independent to the GipsyX binaries.
	export GOA_VAR=/opt/goa-var

	# GIPSYX points to the actual install, and everything else
	# is based on that location
	export GIPSYX=/opt/gipsyx
	export GCORE=$GIPSYX
	export GCOREBUILD=$GIPSYX
	export GNSSTESTDATA=$GIPSYX/testData
	export GDATA=$GNSSTESTDATA/gcore
	export PYTHONPATH=$GIPSYX/lib/python:$PYTHONPATH
	export OPAL_PREFIX=$GIPSYX/lib/openmpi

%runscript

Runtime Helper:

This file should be named gipsyx.sh, and should be set executable with chmod 750 gipsyx.sh.

#!/bin/bash

module load apptainer

IMAGE_NAME=/nobackup/proj/PROJECT_NAME/gipsyx.sif
GOA_VAR=/nobackup/proj/PROJECT_NAME/goa-var

if [ -s "$IMAGE_NAME" ]
then
	echo "Found $IMAGE_NAME"
else
	echo "Unable to find container image: $IMAGE_NAME"
	exit 1
fi

if [ -d "$GOA_VAR" ]
then
	echo "Found $GOA_VAR"
else
	echo "Unable to find goa-var dir: $GOA_VAR"
	exit 1
fi

container.run() {
	# Run a command inside the container...
	# automatically bind the /scratch and /nobackup dirs
	# pass through any additional parameters given on the command line
	apptainer exec --bind $GOA_VAR:/opt/goa-var --bind /scratch:/scratch --bind /nobackup:/nobackup ${IMAGE_NAME} $@
}

container.test() {
	# Run the GipsyX verification pass 
	# This creates a log file within the GIpsyX install tree, so
	# we temporarily create a (small) overlay filesystem to hold the changes.
	if [ -s overlay.img ]
	then
		echo "Found overlay.img"
	else
		echo "Creating test overlay.img"
		apptainer overlay create -s 64 overlay.img
	fi
	apptainer exec --overlay overlay.img --bind $GOA_VAR:/opt/goa-var --bind /scratch:/scratch --bind /nobackup:/nobackup ${IMAGE_NAME} /opt/gipsyx/verify.py
	apptainer exec --overlay overlay.img --bind $GOA_VAR:/opt/goa-var --bind /scratch:/scratch --bind /nobackup:/nobackup ${IMAGE_NAME} bash
}

Steps to building a new container

Put the files gipsyx.def and gipsyx.tgz in the same directory which also contains your goa-var folder, save the following script as gipsyx.build.sh in the same directory and then run it:

#!/bin/bash

###################################################################
# Container build script for GipsyX
# =================================
#
# Should not need to edit any of this, just make sure
# that you have a gipsyx.tgz and a goa-var directory
# in the same location as this script.
#
# For more information:
#	https://hpc.researchcomputing.ncl.ac.uk/dokuwiki/dokuwiki/doku.php?id=advanced:software:gipsyx

# Need a scratch local to create the container - always use /scratch/ - not a NFS/Lustre location
APPTAINER_TMPDIR=/scratch/

# Name of the GipsyX source tarball, should be named gipsyx.tgz
GIPSYX_TAR="gipsyx.tgz"

# Name of the GOA_VAR directory, should have already been created
# and had goa-var-gipsyx.tgz extracted in to it
GOA_VAR="goa-var"

echo "GipsyX Container Build"
echo "======================"

if [ -s "./$GIPSY_TAR" ]
then
	echo "OK: Found GIPSYX_TAR file: $GIPSYX_TAR"
else
	echo "ERROR: You have not provided $GIPSYX_TAR in the same directory as this script"
	exit 1
fi

if [ -d "./$GOA_VAR" ]
then
	echo "OK: Found GOA_VAR dir: $GOA_VAR"
else
	echo "ERROR: You have not created the GOA_VAR dir: $GOA_VAR"
	exit 1
fi

# Load the apptainer module
echo "Loading apptainer..."
module load apptainer

# Build the container image
echo "Building container..."
apptainer build gipsyx.sif gipsyx.def

# Set the container to be readable by self and group only
echo "Set container permissions..."
chmod 440 gipsyx.sif

echo "All done"

Once complete, you can move the gipsyx.sh, gipsyx.sif and goa-var directories to wherever you want - just be sure to make sure that the IMAGE_NAME and GOA_VAR variables in the runtime helper (gipsyx.sh) point to their new locations.

Users will run GipsyX by simply doing a source of the gipsyx.sh file and then using container.run to run any of the GipsyX tools as explained above.

Back to software