You'll build the distributed systems that train Luma's large-scale multimodal models across thousands of GPUs, so researchers can focus on innovation on top of reliable, efficient, scalable infrastructure.
This is hard PyTorch, CUDA, and distributed-systems work — advanced parallelism, training stability, and utilization across massive clusters. It fits an engineer who's solved real problems training foundation models at scale. If you haven't worked at the level of FSDP and multi-node training, this is the wrong depth.
What You'll Own
Design, implement, and optimize efficient distributed training systems for models across thousands of GPUs.
Research and implement advanced parallelization (FSDP, Tensor Parallel, Pipeline Parallel, Expert Parallel).
Build monitoring, visualization, and debugging tools for large-scale training runs.
Optimize training stability, convergence, and resource utilization across massive clusters.
First 90 Days
One way the first 90 could unfold.
Days 1–30 — Immerse & Diagnose: Learn the current training stack and where stability and utilization hurt at scale.
Days 30–60 — Ship & Validate: Land a parallelization or stability improvement that measurably helps a real training run.
Days 60–90 — Scale & Systemize: Build the monitoring and tooling that keeps large runs reliable and efficient.
What You Bring
Extensive distributed PyTorch training and parallelisms in foundation-model training.
Deep understanding of GPU clusters, networking, and storage systems.
Familiarity with communication libraries (NCCL, MPI) and distributed-system optimization.
Nice to Have
Strong Linux systems administration and scripting.
Experience managing training runs across 100+ GPUs.
Experience with containerization, orchestration, and cloud infrastructure.
About Luma: Luma's mission is to build unified general intelligence that can generate, understand, and operate in the physical world. We believe multimodality is critical for intelligence — the next step beyond language models comes from vision. Luma is an equal opportunity employer.
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Research Scientist / Engineer - Training Infrastructure in London vacancy
$230k - $450k per annum
...legal work. We recently post-trained our first model built for... ...legal tasks. You’ll help set the research direction and carry ideas... ...small, high-bar team of hands-on engineers and researchers, where each... ...for research engineers and scientists who have gotten training runs...TrainingFull-time$230k - $450k per annum
...for legal work. We recently post-trained our first model built for legal work,... ...post-training efforts. You’ll own the infrastructure behind that effort: distributed training... ...a small, high-bar team of hands-on engineers and researchers, where each person owns a large part...TrainingFull-time- £103k - £140k per annumEstimated...something to contribute. About the Research & Models Team The Models team builds... ...-edge agentic technology. We focus on training techniques to optimize model capabilities... ...you: You have a strong research engineer / scientist mindset with experience training and improving...TrainingFull-timeHybrid workingOn-site
£260k - £370k per annum
...a quickly growing group of committed researchers, engineers, policy experts, and business leaders... ...You could describe yourself as both a scientist and an engineer. As a Research Engineer... ...robustness of our safety techniques by training language models to subvert our safety...TrainingLong-term contractHybrid workingVisa sponsorshipOn-siteFlexible hours- £126k - £171k per annumEstimated...With OpenAI predicting an automated researcher by March 2028, we don't have time to... ...ambitious and we’re looking for Research Scientists and Research Engineers who are excited to build a new hard... ...misaligned preferences. Design and train model organisms, and scale your insights...TrainingLong-term contractFull-timeRelocation packageVisa sponsorshipOn-siteRemoteWork from homeFlexible hours
- £81k - £111k per annumEstimated...Research Engineer, Model Inference & Serving About H: H exists to push the boundaries of superintelligence... ...co-designing with the Models team on training-time choices and with the agent teams... ...environment and with production ML infrastructure (Kubernetes, etc.) Working knowledge...TrainingFull-timeInternshipHybrid workingOn-site
£260k per annum
...a quickly growing group of committed researchers, engineers, policy experts, and business leaders... ...Anthropic's ML Performance and Scaling team trains our production pretrained models, work... ...to training dynamics and evaluation infrastructure Design and run experiments to...TrainingLong-term contractHybrid workingVisa sponsorshipOn-site5 days/weekFlexible hoursWeekend workAfternoon shift- £74k - £98k per annumEstimated...Helical, we’re focused on leveraging research to transform the future of drug discovery... ...We are seeking an Applied Research Engineer - Post-Training to join our team, focusing on... ...You'll collaborate closely with our ML infrastructure and biology teams, but you'll be the...TrainingFull-time
- £39k - £52k per annumEstimated...group of visionaries, we want to hear from you. We're seeking a Research Engineer to develop cutting-edge generative methods that create physically-grounded 3D environments. You will work on building, training, evaluating, and optimizing models that generate high-quality 3D...Training
£375k per annum
...a quickly growing group of committed researchers, engineers, policy experts, and business leaders... ...and turns that understanding into the training recipes behind our frontier models. As... ...issues at the seam where research meets infrastructure - failures that only appear at scale...TrainingLong-term contractHybrid workingVisa sponsorshipOn-siteFlexible hours£260k per annum
...whole. Our team is a quickly growing group of committed researchers, engineers, policy experts, and business leaders working together to... ...understanding of large language models Optimize and scale our training infrastructure to improve efficiency and reliability Develop and...TrainingLong-term contractHybrid workingVisa sponsorshipOn-siteFlexible hours- £63k - £85k per annumEstimated...a world-leading drug design engine comprising AI models that are... ...you to contribute to frontier research at the intersection of AI and... ...where you will partner with scientists and engineers to scale foundational... ...and optimize LLM post-training methods at scale on frontier...TrainingHybrid workingOn-site1 day/week
- £73k - £99k per annumEstimated...Flow Traders is looking for a Senior Research Engineer to join us in our London office. This is a unique opportunity to join a leading proprietary... ...You'll expand the framework to become global standard way of training, consuming, combining, and transforming any data source in a...TrainingFull-timeOn-site
£370k per annum
...is a quickly growing group of committed researchers, engineers, policy experts, and business leaders working... ...of our RL Science stack - the infrastructure, tooling, and systems that let researchers iterate quickly on training runs. As a Research Engineer on the team,...TrainingLong-term contractHybrid workingVisa sponsorshipOn-siteFlexible hours- £82k - £108k per annumEstimated...innovation and a willingness to challenge consensus. As a Research Engineer , you will be an integral member of a systematic research team... ...technology, mentored by industry-leading engineers, offered training and continuing education opportunities, and given immediate responsibility...TrainingLong-term contractImmediate start
£260k per annum
...whole. Our team is a quickly growing group of committed researchers, engineers, policy experts, and business leaders working together to... ...for large language models Building scalable RL infrastructure and training methodologies Enhancing model reasoning capabilities...TrainingLong-term contractHybrid workingVisa sponsorshipOn-siteFlexible hours- £50k - £68k per annumEstimated...language models (LLMs) with sound software engineering to build useful tools and automate... ...including user feedback, documentation and training Maintaining solutions and measuring... ...time saved, fewer errors and usage Researching and testing new approaches, share useful...TrainingPart-timeHybrid workingFlexible hours
£180k - £220k per annum
...About Nominal Engineers today are building some of the most ambitious machines... ...them. The role As a Staff Research Engineer supporting agent evals & post-training, you'll define how Nominal... ...why ours win. Build the eval infrastructure: datasets, model-based and human...TrainingFull-timeOn-site5 days/week- £61k - £82k per annumEstimated...About us Re-architecting Engineering for the Age of Intelligence PhysicsX is the... ...closely with our simulation engineers and research scientists to develop an understanding of the... ...GPU/multi-node distributed systems for training and inference at the scale of PBs of data...TrainingLong-term contractHybrid workingOn-siteWork from home
- £65k - £88k per annumEstimated...a world-leading drug design engine comprising AI models that are... ...disease with the help of AI. Research Engineering (Machine Learning... ...you will be partnering with scientists and engineers to advance... ...codebases, data pipelines, and infrastructure for training and evaluation....TrainingHybrid workingOn-site1 day/week
- £98k - £133k per annumEstimated...Trading Group is committed to world class research. We empower exceptional talents in... ...of traders, quantitative researchers, and engineers who work together to examine the global markets... ...quantitative finance. Whether optimizing training pipelines on high-performance computing...TrainingFull-timeVisa sponsorshipWork visaFlexible hours
- £52k - £71k per annumEstimated...Basecamp Research seeks a Senior Research Scientist to advance AI safety and biosecurity for biological AI systems. The role involves designing evaluations... ...creating safeguards for frontier therapeutic AI models trained on proprietary genomic data. Responsibilities Design...Training
- £49k - £64k per annumEstimated...robotics and AR/VR to gaming and cinema, generate and interact with physically-grounded 3D environments. We're seeking a Research Scientist to train the policies that turn a world model into a robot that acts. You will own vision-language-action (VLA) and world-action...Training
- £42k - £53k per annumEstimated...collaborate with a talented group of visionaries, we want to hear from you. We're seeking a Research Scientist focused on 3D diffusion . You will lead research to design, build, train, evaluate, and optimize diffusion-based generative models that produce high-quality 3D...Training
- £69k - £93k per annumEstimated...Team We are the central research and development team within Faculty... ...As a Senior Research Scientist at Faculty, you will lead the... ...generative AI architectures, training methodologies, and safety mitigation... ...combined with the engineering discipline required to build...TrainingFull-timePart-time
$150 - $200 per hour
...technical talent with leading AI research labs. Headquartered in San... ...machine learning engineering tasks in a long-horizon AI research... ...systems work over heavy model training. Balance tasks between GPU... ...research engineer , research scientist , or applied scientist ....TrainingRemote jobSummer workMercor
London17 days ago£123k - £129k per annum
...also initiate and foster collaborations with other research teams in Alphabet. AI Foundations areas that we... ...role, you will report to a Senior Staff Software Engineer. You will: Partner with foundation model training teams to determine optimal data mixtures, training...TrainingFull-timeHybrid working- £49k - £66k per annumEstimated...Basecamp Research London, United Kingdom The Role You will advance our work... ...evaluate them and make them safer. We train frontier biological models on BaseData,... ...in biology, and working with experimental scientists to establish when computational results support...TrainingImmediate start
$110 - $145 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include... ...Provide clear, structured feedback to AI research teams to improve training data quality and downstream performance. Develop...TrainingRemote jobSummer workMercor
Londonmore than 2 months ago£110k - £130k per annum
...approach stems from years of scientific research and a series of breakthroughs in... ...a small team of exceptional engineers to help define what comes next. This... ...the company. Take ownership of the infrastructure that powers the AI training. This is not a conventional infrastructure...TrainingPermanent
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Research Scientist / Engineer - Training Infrastructure. Be the first to apply!
