This is a project which is currently making use of HPC facilities at Newcastle University. It is active.
For further information about this project, please contact:
This project investigates reasoning-aware Mixture-of-Experts (MoE) architectures for Large Language Models (LLMs). The research focuses on improving reasoning performance by studying expert specialisation, routing strategies, and architectural modifications for complex reasoning tasks. Experiments involve training and evaluating transformer-based MoE models on public reasoning benchmarks, including mathematical, commonsense, and multi-step reasoning datasets. The project also analyses expert routing behaviour and model efficiency to better understand the relationship between routing decisions and reasoning capability.
The project uses Python with the PyTorch deep learning framework, Hugging Face Transformers and Datasets, CUDA-enabled GPU computing, and supporting scientific Python libraries. Compute workloads include distributed and GPU-accelerated model training, fine-tuning, inference, hyperparameter optimisation, and large-scale evaluation of language models on reasoning benchmarks. Additional workflows include experiment logging, routing analysis, and processing of large benchmark datasets.