Skip to main content
Trainy logo

Member of Techincal Staff

Trainy

San Francisco, CA · HybridFull-time$140–250K/yrSeen todayStill listed · checked in the last 2 days

See where you stand before you apply. No sign-up to start.

You apply on the posting's application page, with no Olive account needed.

Watch jobs like this.

Don't just apply. Show up ready.

Olive works from this exact posting.

At a glance

Compensation
$140–250K/yr
Location
San Francisco, CAHybrid
Schedule
Full-time
Work Authorization
Not specified

Olive focuses on jobs in the United States, including remote roles you can work from there.

Job overview

Trainy operates infrastructure for foundation model labs, keeping more than 10,000 GPUs healthy across cloud and bare-metal clusters. The early-career engineer joins the operations team to support cluster operations and develop Konduktor, a Kubernetes-native orchestration platform for detecting hardware failures and automating remediation. The role includes twice-weekly visits to Bay Area customer sites and hands-on work with teams training frontier models.

Skills & qualifications

RequiredNice to have

Skills

Git, nice to haveGo, nice to haveKubernetes, nice to havePython, nice to haveTypeScript, nice to haveWeb Front-End Development, requiredReact, requiredGrafana, nice to havePrometheus, nice to haveLinux, requiredCommand Line, requiredSystem Troubleshooting, requiredNetworking Troubleshooting, requiredKubectl, requiredTroubleshooting, requiredWritten Communication, requiredCustomer Communication, requiredAgentic Software Development, nice to haveClaude Code, nice to haveCursor, nice to haveCodex, nice to havePyTorch, nice to haveJAX, nice to haveMegatron/NeMo, nice to haveSlurm, nice to haveLustre, nice to haveInfiniBand/RoCE, nice to haveNCCL, nice to haveGPU Infrastructure, nice to haveDCGM, nice to haveNVIDIA SMI, nice to haveXID Error Interpretation, nice to haveKubernetes Controllers, nice to haveController-Runtime, nice to haveKubebuilder, nice to haveD3, nice to haveRecharts, nice to haveECharts, nice to haveKueue, nice to haveMultiKueue, nice to haveVolcano, nice to have

Qualifications

BS in CS or Related or Equivalent, required

Full job description

About the role Trainy operates the infrastructure behind some of the world's leading foundation model labs. Every day, our platform keeps more than 10,000 GPUs healthy, scheduled, and productive across multiple clouds and bare-metal clusters, where a single bad node can stall a training run that costs thousands of dollars an hour. We're looking for an early-career engineer to join our operations team and help keep that fleet running. You'll work on Konduktor, our Kubernetes-native orchestration platform, with a focus on the systems that detect failing hardware, surface problems before customers notice them, and automate the fixes. You'll also spend time onsite with our customers in the Bay Area, working directly with the teams training frontier models. This is a rare chance to work hands-on with frontier-scale AI infrastructure from day one. What you'll be doing

  • Supporting day-to-day operations of large GPU clusters for foundation model labs, triaging node failures, job disruptions, and customer escalations alongside senior engineers.
  • Visiting customer sites in the Bay Area twice a week to work alongside their infrastructure and research teams, gather feedback, and resolve issues in person.
  • Building and extending Konduktor's health controller to detect GPU, network, and node faults, then automatically cordon, drain, and remediate affected machines.
  • Designing and shipping TypeScript front ends that give our team and our customers real-time visibility into cluster health, job status, and fleet utilization.
  • Turning recurring operational work into automation, replacing manual runbooks with tooling and controllers.
  • Writing clear incident reports and runbooks, and feeding what you learn from production back into the platform.

What we need to see

  • BS in Computer Science, Computer Engineering, or a related field, or equivalent practical experience (internships, research, open source, or a homelab count).
  • Working proficiency in TypeScript and at least one of Python or Go. You'll write front-end visualizations as well as backend tooling and automation.
  • Experience building web front ends with a modern framework (React or similar), and an instinct for turning raw telemetry into views that make problems obvious. Familiarity with Grafana or Prometheus is a plus.
  • Hands-on familiarity with Linux and the command line, including reading logs, inspecting processes, and diagnosing basic system and networking issues.
  • Foundational knowledge of Kubernetes concepts such as pods, nodes, deployments, taints, and labels, and comfort using kubectl across multiple clusters and contexts.
  • A methodical approach to troubleshooting: forming hypotheses, gathering evidence, and writing up clear findings and runbooks.
  • Strong written communication and the ability to work directly with customers' ML and infrastructure teams during incidents.
  • Ability to be onsite at customer locations in the San Francisco Bay Area twice a week.

Ways to stand out from the crowd

  • Fluency with agentic software development, such as using tools like Claude Code, Cursor, or Codex to plan, implement, test, and review changes, along with good judgment about when to trust, verify, or rewrite their output.
  • Familiarity with ML frameworks (PyTorch, JAX, Megatron/NeMo) and distributed HPC infrastructure (Slurm, Lustre, InfiniBand/RoCE), including common failure modes like NCCL hangs and straggler nodes.
  • Exposure to GPU infrastructure and NVIDIA tooling such as DCGM, nvidia-smi, and XID error interpretation.
  • Experience writing Kubernetes controllers or operators (controller-runtime, kubebuilder), especially for automated node health detection, cordoning, and remediation.
  • Experience with charting and data visualization libraries (D3, Recharts, ECharts, or similar) for real-time or time-series data.
  • Familiarity with job queueing and batch scheduling on Kubernetes (Kueue, MultiKueue, or Volcano).
  • Experience with GitOps workflows (FluxCD or Argo CD) and multi-cloud environments (GKE, OKE, EKS).
  • Contributions to open-source infrastructure projects or a portfolio of tools you've built to monitor or automate systems.

How this job compares

Set against the 4,722 open Infrastructure Engineer jobs on Olive Jobs, counted November 18, 2026.

Pay
  • This posting$140–250K/yr
  • California median$172–253K/yr729 salary ranges
  • US median$121–229K/yr4,030 salary ranges

This range's midpoint, $195K, is 9% below the California median midpoint ($214K) — in the middle half of Infrastructure Engineer salary ranges in California.

Openings
Olive Jobs lists 908 open Infrastructure Engineer jobs in California and 4,722 across the US. 9% of them nationwide are listed as remote or flexible.
Hiring the most in California
Experience
This posting asks for no prior experience — less than three in four Infrastructure Engineer postings that state a minimum ask for (median 6 years, across 3,778 postings).
Education
608 of the 4,722 open Infrastructure Engineer jobs (13%) ask for a bachelor's degree or higher.

Pay, requirements and demand for Software Engineer jobs

Counted across the open, public US postings on Olive Jobs on November 18, 2026. Figures based on too few postings are left out.

Prepare for this role

You've read the whole posting — now see how you match it.