====== GipsyX ====== **Important License Information** The license for using this software is restrictive. As part of the terms of the license it is **only** available to select HPC projects on Comet. If you need to use this software and you are //not// a member of a project which already has access, then please [[:contact:index|contact us]] first. > GipsyX is developed by the Jet Propulsion Laboratory (JPL), and maintained by the Near Earth Tracking Applications and Systems groups. It replaces the GIPSY-OASIS (GNSS-Inferred Positioning System and Orbit Analysis Simulation Software) software. **Features:** * Analysis of data from Global Navigation and Satellite Systems (GNSS): * Global Positioning System (GPS) * Russian GLONASS * French DORIS * Satellite Laser Ranging (SLR) system * Precise centimeter-level GNSS-based positioning and timing: * Space platforms, including low-Earth orbiters and GNSS constellations * Airplanes * Terrestrial (ground) stations, static or moving * Time transfer * Single high-level user interface supports majority of precise positioning applications * Single-receiver ambiguity resolution using JPL's GPS orbit and clock products **Applications:** * Terrestrial positioning for geophysical research: * Earth deformation, plate tectonics * GNSS Networks (e.g. PBO and SCIGN) * Ice flow * Studying troposphere and ionosphere * Reference frame (geocenter and scale) and Earth rotation parameters * Airplane positioning * Precise orbit determination * Low-Earth orbiters: Topex/Poseidon, Jason-1, Jason-2/OSTM, Jason-3, Sentinel-6, NISAR, GRACE, GRACE-FO, Champ, SAC-C, COSMIC, COSMIC-2, DSAC, Space Shuttle (e.g. SRTM), Commercial Satellites * GNSS constellations including GPS, GLONASS, Galileo, Beidou, QZSS, and IRNSS For further information: https://gipsyx.jpl.nasa.gov/ ---- ===== Running GipsyX on Comet ===== **GipsyX** has been installed on Comet as an **application container**. This allows the software to be fully restricted to the HPC project group(s) who are allowed access as part of the terms of the license. It also allows the staff in those groups to update the container to new software versions as-and-when required //without// any superuser intervention. The GipsyX container (by default named as ''gipsyx.sif'') should be located in the project folder of only those HPC project groups __which are licensed to use it__. All of the GipsyX tools are installed to the location ''/opt/gipsyx'' inside the container and all of the correct environment variables (''PATH'', ''GIPSYX'', ''GOA_VAR'', etc) have been set to point to this installation location. To make running the GipsyX tools easier, we have provided a runtime helper script (by default named ''gipsyx.sh'') which aims to simplify accessing the tools, allows mapping of an external ''goa-var'' configuration directory as well as reading/writing to your ''$HOME'' directory and ''/nobackup'' project folders. After sourcing the script, you will have a new command ''container.run'' that will run any tool or script from inside the container. For example, to run the ''g2de'' tool from GipsyX, assuming that your project folder is named ''/nobackup/proj/PROJECT_NAME/'', run the following: $ source /nobackup/proj/PROJECT_NAME/gipsyx.sh $ container.run gd2e.py --help INFO: gocryptfs not found, will not be able to use gocryptfs usage: gd2e.py [-h] [-treeSequenceDir dir_of_trees] [-subIterations n [n ...]] [-addIteration n] [-runType {LEO,PPP,GNSSPRTL}] [-xmtPos] [-geop GEOP] [-gdCov] [-startTime startTime [startTime ...]] [-endTime endTime [endTime ...]] [-dataEditor "cmd dataRecordFile.gz"] [-drEditedFile edited.dr.gz] [-quatLeo leo.quat] [-initLeo leo.pos] [-trajTree trajedy.tree] [-noTrajFit] [-GNSSproducts dir|url] [-prodTypeGNSS nf|nnr|fid] [-HighRateProducts] [-orbClkDir [GNSSinitValues]] [-selectGnss pattern] [-rnxFile rinexFile] [-staDb StaDb] [-tdpInput tdpInput] [-antexFile f.atx] [-requireAntexPhaseCal {G,E,R,C} [{G,E,R,C} ...]] [-xyzFile f.xyz] [-ammFile AntennaMapModel] [-ammXmtFile AntennaMapModel_transmitters] [-gnssAdjustedProducts dir] [-selectGnssAdjusted pattern] [-recList [RecNm SAT|STA ...]] [-nProcessors np] [-keepIterFiles] [-mkTreeSeq [treeSeqDir]] gd2e = Geodetic Data To Estimate Beginnings of a script to iterate through geodetic (nominally GNSS) data (e.g. GPS) and estimate various parameters. Initial use case includes Low Earth Orbiter data and static point positioning. ... ... As a reminder: * ''gipsyx.sif'' - the bare operating system container (Ubuntu LTS) with GipsyX, Python and all dependencies * ''gipsyx.sh'' - a runtime helper script which provides the ''container.run'' command to run any of the tools inside the container * ''goa-var'' - an external directory **in your project folder** which contains the goa-var configuration data necessary to run many of the GipsyX tools; this is intended to be maintained/updated //by your group//. ==== Running GipsyX verification ==== By using the runtime helper wrapper script as provided, you can run the GipsyX verification tests by a ''source'' of the runtime script and then calling ''container.test''; a simple macro which in turn calls the GipsyX ''verify.py'' utility. Note: Replace //PROJECT_NAME// in the path below with name of the project folder that your ''gipsyx.sh'' runtime helper, ''gipsyx.sif'' container file and ''goa-var'' directory are located in. $ source /nobackup/proj/PROJECT_NAME/gipsyx.sh $ container.test Found overlay.img INFO: gocryptfs not found, will not be able to use gocryptfs GipsyX VERIFY TESTS Determining system setup.... Platform: Linux-5.14.0-427.91.1.el9_4.x86_64-x86_64-with-glibc2.35 Python executables python : not found python3 : /usr/bin/python3 : Python 3.10.12 Environment Variables: LANG : en_GB.UTF-8 PATH : /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gipsyx/bin LD_LIBRARY_PATH : /.singularity.d/libs GOA_VAR : /opt/goa-var GCORE : /opt/gipsyx GNSSTESTDATA : /opt/gipsyx/testData GCOREBUILD : /opt/gipsyx OPAL_PREFIX : /opt/gipsyx/lib/openmpi PYTHONPATH : /opt/gipsyx/lib/python: Running unit Tests... STARTING G+E gd2e point positioning SUCCESS G+E gd2e point positioning Runtime: 0:00:36.986917 STARTING gd2e LEO orbit determination SUCCESS gd2e LEO orbit determination Runtime: 0:01:50.242200 STARTING GPS-only gd2e point positioning SUCCESS GPS-only gd2e point positioning Runtime: 0:00:08.342260 STARTING net*.py stacov manipulation SUCCESS net*.py stacov manipulation Runtime: 0:00:19.584694 4 of 4 tests pass. All tests passed! May your data be good! ==== Sample GipsyX Slurm Job ==== This is an example of a //very simple// batch file which can be used to run the GipsyX container application under Slurm. In the example below we: * Use the account code ''comet_abc123'' - you should substitute with your own account code * Use the ''default_free'' partition - you should choose the [[:started:comet_resources|correct partition]] for your needs * Allocate ''8'' CPU cores * Allocate ''32'' gigabytes of RAM * Allow up to ''1'' hour and ''30'' minutes of runtime Run two different GipsyX tools (''gd2e.py'' and ''tdpMerge.py'') - this is to demonstrate that we can call the tools as many times as we want, after the initial ''source'' of the runtime helper script. #!/bin/bash #SBATCH --account=comet_abc123 #SBATCH --partition=default_free #SBATCH -c 8 #SBATCH --mem=32G #SBATCH --time 01:30:00 source /nobackup/proj/PROJECT_NAME/gipsyx.sh container.run gd2e.py arg1 arg2 arg3 container.run tdpMerge.py -infFiles file1.tdp -outFile out Save the script as ''gipsyx_sbatch.sh'' and submit as: $ sbatch gipsyx_sbatch.sh ---- ===== Running GipsyX graphical tools on Comet ===== To be written ---- ===== Accessing Data ===== While running GipsyX you have two different filesystems to consider: //inside// the container and //outside// the container. As long as you use the ''container.run'' helper you will automatically get access to most of the directories you need while GipsyX is running and will usually not need to do anything special to access your input and output files. If one of the GipsyX commands needs to import or export data, you must remember that paths such as '''/'', ''/usr'', ''/usr/local'', ''/bin'', or similar all refer to directories //inside// the container. The three directories: * ''/scratch'' * ''/nobackup'' * ''$HOME'' (or ''/mnt/nfs/home'') ... all refer to //real// directories //outside// the container on Comet, so restrict your reading/writing data to //those// three locations only. In addition the directory //inside// the container: * ''/opt/goa-var'' Is actually a pointer to a //real// directory within your HPC group's project directory (i.e. under ''/nobackup/proj/PROJECT_NAME'') - this is dependent on who set up your GipsyX installation and runtime helper script. If you use any of the graphical tools in GipsyX, and they offer to browse/open files or output directories on your behalf, remember that only those three locations actually refer to real Comet filesystem directories, anything else it shows you in the file select dialogue window refers to folders //inside// the container. ---- ===== Building GipsyX on Comet ===== **Important!** This section is only relevant to RSE HPC support staff or users who want to understand how GipsyX has been installed. If you only want to //use// the software, stop reading now. This section is still in development!!! You will need: * GipsyX container definition file (named ''gipsyx.def''), listed below - //only needed to build the container, not to run it// * GipsyX runtime helper (named ''gipsyx.sh''), listed below - your users ''source'' this file to run GipsyX * GipsyX build script (named ''gipsyx.build.sh''), listed below - //only needed to build the container, not to run it// * GipsyX installation tarball (named ''gipsyx.tar.gz'') - this __must__ be provided by a user //licensed to download the software// The output will be a single file named ''gipsyx.sif'', containing the GipsyX software and all runtime dependencies. This should be copied to a location only accessible to members of the allowed HPC project group. We therefore suggest the use of your ''/nobackup/proj/PROJECT_NAME'' folder. You must **have also** already extracted the ''goa-var-gipsyx.tgz'' file to your project folder (again, this __must__ be provided by a user //licensed to download the software//). The build script and runtime helper **will refuse to run** if this is not found. **Container Definition:** This file should be named ''gipsyx.def''. Bootstrap: docker From: ubuntu:jammy #################################################################### # GipsyX Container # ================== # This is a runtime environment for GipsyX # Please see: # https://hpc.researchcomputing.ncl.ac.uk/dokuwiki/dokuwiki/doku.php?id=advanced:software:gipsyx # #################################################################### %files gipsyx.tgz /opt %post # Prevent interactive prompts export DEBIAN_FRONTEND=noninteractive #################################################################### # Basic system packages #################################################################### # Update & install only necessary packages apt-get update apt-get install -y aptitude wget unzip # GipsyX deps apt-get install -y \ python3 \ python3-pip \ libpciaccess0 \ libnl-route-3-200 \ libgfortran5 \ libquadmath0 \ libibverbs1 \ python3-numpy \ python3-bs4 \ python3-requests \ python3-matplotlib \ python3-scipy \ python3-boto \ python3-cartopy \ python3-distro \ python3-h5py \ python3-pandas # Clean up APT cache to save space apt-get clean # Unpack GipsyX tarball # ===================== # If the version of GipsyX changes, then this is the only # place you need to change it - referencing the directory # name inside the tarball itself. cd /opt tar -zxf /opt/gipsyx.tgz ln -s /opt/GipsyX-2.5 /opt/gipsyx # This is an empty directory where the externally mounted # GOA_VAR tarfile is unpacked to... those files *do not* # live inside the container, so as to allow them to be # updated as-and-when required. mkdir -p /opt/goa-var # Fix GOA_VAR in various GipsyX scripts # as we are *not* installing in a home directory. export GOA_VAR=/opt/goa-var export GIPSYX=/opt/gipsyx export GCORE=$GIPSYX export GCOREBUILD=$GIPSYX export GNSSTESTDATA=$GIPSYX/testData export GDATA=$GNSSTESTDATA/gcore export PYTHONPATH=$GIPSYX/lib/python:$PYTHONPATH export OPAL_PREFIX=$GIPSYX/lib/openmpi cd /opt/gipsyx ./fix_rc_GipsyX.py -v $GOA_VAR # Remove all src packages and tarballs cd / rm -rf /src rm -f /opt/*.tgz #################################################################### # Additional environment variables #################################################################### %environment export PATH=$PATH:/opt/gipsyx/bin # GOA_VAR should be mapped to a directory using the 'bind' # flag at runtime so that it is mounted outside the container # and can be kept updated independent to the GipsyX binaries. export GOA_VAR=/opt/goa-var # GIPSYX points to the actual install, and everything else # is based on that location export GIPSYX=/opt/gipsyx export GCORE=$GIPSYX export GCOREBUILD=$GIPSYX export GNSSTESTDATA=$GIPSYX/testData export GDATA=$GNSSTESTDATA/gcore export PYTHONPATH=$GIPSYX/lib/python:$PYTHONPATH export OPAL_PREFIX=$GIPSYX/lib/openmpi %runscript **Runtime Helper:** This file should be named ''gipsyx.sh'', and should be set executable with ''chmod 750 gipsyx.sh''. #!/bin/bash module load apptainer IMAGE_NAME=/nobackup/proj/PROJECT_NAME/gipsyx.sif GOA_VAR=/nobackup/proj/PROJECT_NAME/goa-var if [ -s "$IMAGE_NAME" ] then echo "Found $IMAGE_NAME" else echo "Unable to find container image: $IMAGE_NAME" exit 1 fi if [ -d "$GOA_VAR" ] then echo "Found $GOA_VAR" else echo "Unable to find goa-var dir: $GOA_VAR" exit 1 fi container.run() { # Run a command inside the container... # automatically bind the /scratch and /nobackup dirs # pass through any additional parameters given on the command line apptainer exec --bind $GOA_VAR:/opt/goa-var --bind /scratch:/scratch --bind /nobackup:/nobackup ${IMAGE_NAME} $@ } container.test() { # Run the GipsyX verification pass # This creates a log file within the GIpsyX install tree, so # we temporarily create a (small) overlay filesystem to hold the changes. if [ -s overlay.img ] then echo "Found overlay.img" else echo "Creating test overlay.img" apptainer overlay create -s 64 overlay.img fi apptainer exec --overlay overlay.img --bind $GOA_VAR:/opt/goa-var --bind /scratch:/scratch --bind /nobackup:/nobackup ${IMAGE_NAME} /opt/gipsyx/verify.py apptainer exec --overlay overlay.img --bind $GOA_VAR:/opt/goa-var --bind /scratch:/scratch --bind /nobackup:/nobackup ${IMAGE_NAME} bash } ==== Steps to building a new container ==== Put the files ''gipsyx.def'' and ''gipsyx.tgz'' in the same directory which also contains your ''goa-var'' folder, save the following script as ''gipsyx.build.sh'' in the same directory and then run it: #!/bin/bash ################################################################### # Container build script for GipsyX # ================================= # # Should not need to edit any of this, just make sure # that you have a gipsyx.tgz and a goa-var directory # in the same location as this script. # # For more information: # https://hpc.researchcomputing.ncl.ac.uk/dokuwiki/dokuwiki/doku.php?id=advanced:software:gipsyx # Need a scratch local to create the container - always use /scratch/ - not a NFS/Lustre location APPTAINER_TMPDIR=/scratch/ # Name of the GipsyX source tarball, should be named gipsyx.tgz GIPSYX_TAR="gipsyx.tgz" # Name of the GOA_VAR directory, should have already been created # and had goa-var-gipsyx.tgz extracted in to it GOA_VAR="goa-var" echo "GipsyX Container Build" echo "======================" if [ -s "./$GIPSY_TAR" ] then echo "OK: Found GIPSYX_TAR file: $GIPSYX_TAR" else echo "ERROR: You have not provided $GIPSYX_TAR in the same directory as this script" exit 1 fi if [ -d "./$GOA_VAR" ] then echo "OK: Found GOA_VAR dir: $GOA_VAR" else echo "ERROR: You have not created the GOA_VAR dir: $GOA_VAR" exit 1 fi # Load the apptainer module echo "Loading apptainer..." module load apptainer # Build the container image echo "Building container..." apptainer build gipsyx.sif gipsyx.def # Set the container to be readable by self and group only echo "Set container permissions..." chmod 440 gipsyx.sif echo "All done" Once complete, you can move the ''gipsyx.sh'', ''gipsyx.sif'' and ''goa-var'' directories to wherever you want - just be sure to make sure that the ''IMAGE_NAME'' and ''GOA_VAR'' variables in the runtime helper (''gipsyx.sh'') point to their new locations. Users will run GipsyX by simply doing a ''source'' of the ''gipsyx.sh'' file and then using ''container.run'' to run any of the GipsyX tools as explained above. ---- [[:advanced:software|Back to software]]