RadixArk logo

Member of Technical Staff — Training

RadixArk

Palo Alto, CAFull-time$200–400K/yrPosted todayStill listed today

Most applications go out cold — see where you stand first. No sign-up to start.

You apply on the employer's site, with no Olive account needed.

Watch jobs like this.

Don't just apply. Show up ready.

Olive works from this exact posting.

At a glance

Compensation
$200–400K/yr
Location
Palo Alto, CA
Schedule
Full-time
Work Authorization
Visa required • Visa sponsorship

Olive lists jobs from US employers, including remote roles you can work from the United States.

Job overview

The Member of Technical Staff, Training designs, builds, and operates distributed systems for large-scale model post-training across training, inference, and orchestration. The role focuses on performance, correctness, scalability, and reliability across GPU clusters, and works across modeling recipes, infrastructure, and low-level systems to turn experimental requirements into robust software.

Skills & qualifications

RequiredNice to have

Skills

Distributed ComputingMachine LearningPerformance TuningFault ToleranceOpen-Source SoftwareReinforcement LearningInfrastructureDistributed Machine Learning SystemsLarge-Scale Training InfrastructureHigh-Performance InferenceLLM Post-Training SystemsTraining BackendsInference SystemsMulti-GPU WorkloadsMulti-Node WorkloadsNumerical CorrectnessLow PrecisionPost-Training AlgorithmsOrchestration InfrastructureMultimodal TrainingMultimodal InferenceAgent InfrastructureOpen-Source DevelopmentPPOGRPOPost-Training FrameworksGPU Kernel DevelopmentCommunication-Layer OptimizationLarge-Scale Model TrainingLarge-Scale Model Serving

Qualifications

3+ Years Distributed ML Systems ExperienceTop-Tier Systems Publications

Benefits

401(k) Match
Paid Time Off

Full job description

About the Role As a Member of Technical Staff, Training, you will design, build, and operate the distributed systems behind large-scale model post-training — spanning training, inference, and orchestration, with a focus on the performance, correctness, scalability, and reliability of workloads running across large GPU clusters. This role suits engineers who move fluidly across modeling recipes, complex infrastructure, and low-level systems, identify bottlenecks in distributed workloads, and translate experimental requirements into robust software. In This Role, You Will

  • Design, build, and operate distributed training, rollout, and orchestration systems for large-scale LLM and multimodal post-training across multi-GPU, multi-node environments.

  • Profile and optimize performance across the full-stack — model implementation, parallelism strategies, communication libraries, and GPU kernels — to improve throughput, latency, memory efficiency, hardware utilization, and cost.

  • Investigate numerical correctness and low-precision issues in distributed training and inference, including train–inference consistency for reinforcement learning.

  • Improve the reliability of long-running workloads through checkpointing, fault recovery, observability, and operational tooling.

  • Build supporting infrastructure for reinforcement learning and agentic post-training, including asynchronous rollout, trajectory collection, sandboxed execution, evaluation harnesses, and data pipelines.

  • Contribute to open-source training and inference systems, including Miles and SGLang, and partner with researchers to turn experimental requirements into production systems.

Minimum Qualifications

  • 3+ years of experience building or operating distributed machine learning systems, large-scale training infrastructure, or high-performance inference systems.

  • Hands-on experience with post-training systems, training backends, or inference systems for large language models (e.g., Megatron-LM, FSDP, SGLang, TensorRT-LLM, vLLM).

  • Experience in at least two of the following areas:

  • Performance, efficiency, and scalability of multi-GPU, multi-node workloads

  • Numerical correctness or low precision

  • Stability, reliability, or fault tolerance

  • Post-training algorithm recipes and orchestration infrastructure for large training runs

  • Multimodal training or inference, including vision-language models and multimodal generation

  • Agent infrastructure, including sandboxes, harnesses, and eval systems

  • Building and maintaining open-source projects widely adopted in industry and academia

Preferred Qualifications

  • Familiarity with RL algorithms such as PPO, GRPO, and their variants, and experience applying them in large-scale post-training.

  • Experience with modern post-training frameworks (e.g., Miles, slime, AReaL, verl, Prime-RL).

  • Key open-source contributions to training or inference frameworks (e.g., SGLang, vLLM, Megatron-LM).

  • GPU kernel development (e.g., CUDA, Triton, CUTLASS) or communication-layer optimization (e.g., NCCL, RDMA, NVLink/NVSwitch).

  • Experience training or serving models at very large scale (e.g., Mixture-of-Experts models on clusters of thousands of GPUs).

  • Top-tier publications in ML systems or other systems fields.

Even if you don't meet every qualification above, we encourage you to apply — we care most about demonstrated ability to build and reason about large-scale systems. About RadixArk RadixArk builds open-source and production infrastructure for large language models and multimodal post-training. Our systems — including Miles, an enterprise-grade reinforcement learning training framework, and SGLang, a widely deployed high-performance LLM inference engine — power distributed post-training across clusters of 10k–100k+ GPUs. Compensation Depending on background, skills, and experience, the expected annual salary range for this position is $200,000 to $400,000, plus equity.

Benefits include a 401(k) plan and unlimited PTO.

RadixArk sponsors employment visas (e.g., H-1B, O-1) for eligible candidates. Equal Opportunity RadixArk is an Equal Opportunity Employer and is proud to offer equal employment opportunity to everyone regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, veteran status, and more.

How this job compares

Set against the 865 open Distributed Systems Engineer jobs on Olive Jobs, counted November 8, 2026.

Pay
  • This posting$200–400K/yr
  • California median$180–262K/yr144 salary ranges
  • US median$115–235K/yr758 salary ranges

This range's midpoint, $300K, is 33% above the California median midpoint ($225K) — in the top quarter of Distributed Systems Engineer salary ranges in California.

Openings
Olive Jobs lists 194 open Distributed Systems Engineer jobs in California and 865 across the US. 3% of them nationwide are listed as remote or flexible.
Hiring the most in California
Experience
This posting asks for at least 3 years of experience — less than three in four Distributed Systems Engineer postings that state a minimum ask for (median 6 years, across 708 postings).
Education
81 of the 865 open Distributed Systems Engineer jobs (9%) ask for a bachelor's degree or higher.

Counted across the open, public US postings on Olive Jobs on November 8, 2026. Pay medians are the middle values of the low ends and of the high ends of postings that list both ends of a range in the same cadence (a yearly salary or an hourly rate); a range's midpoint is halfway between its ends. Pay and experience figures resting on fewer than 20 postings are left out.

Prepare for this role

You've read the whole posting — now see how you match it.