This is a project which is currently making use of HPC facilities at Newcastle University. It is active.
For further information about this project, please contact:
At Imago, our mission is to make satellite imagery more useful, usable, and used across social research, public health, and policy. We work with academics, data providers, practitioners and policymakers to create data products that can help solve the UK’s most pressing challenges, from wellbeing to prosperity and sustainability. We are a Smart Data Research UK Data Service, led by researchers from University of Liverpool, Newcastle University, University of Manchester, and Harvard University.
Required Tools:
- Anaconda (Python)
- Dask
- Apptainer
- Apache Spark
- Java (required for Spark)
Our workflow primarily relies on Python and Dask. However, in some cases, we may need Apache Spark for large-scale data processing. I’ve included both Dask and Spark because, based on experience, containers alone cannot be used to set up a distributed cluster. While we can configure a Dask cluster using Anaconda, installing Apache Spark is still necessary for certain big data tasks.