Skip to main content
Open Relay logo

Member of Technical Staff, Inference

Open Relay

CAFull-timePosted todayStill listed · checked in the last 2 days

See where you stand before you apply. No sign-up to start.

You apply on the posting's application page, with no Olive account needed.

Watch jobs like this.

Don't just apply. Show up ready.

Olive works from this exact posting.

At a glance

Compensation
No compensation found
Location
CA
Schedule
Full-time
Work Authorization
Not specified

Olive focuses on jobs in the United States, including remote roles you can work from there.

Job overview

Open Relay is hiring a Member of Technical Staff to own the serving layer for open-weight models running on its GPUs. The role covers engine selection and configuration, request routing, and performance under changing traffic. The hire will work with the founders, set technical direction, and independently identify, ship, and own production improvements. The position is based in San Francisco or Seattle; no degree is required, and equivalent experience counts.

Skills & qualifications

RequiredNice to have

Skills

Python (Programming Language), nice to havePyTorch (Machine Learning Library), nice to haveNvidia CUDA, nice to haveTrident, nice to haveVLLM, requiredSGLang, requiredNVIDIA Dynamo, requiredLarge Codebases, requiredWritten Communication, requiredEnglish, requiredOpen-Source Inference Engines, nice to haveGPU Kernel Performance, nice to haveCUDA, nice to haveROCm, nice to haveTriton, nice to haveMixture-of-Experts Models, nice to haveMulti-Node Inference, nice to haveStartup Experience, nice to have

Qualifications

3+ Years Industry Experience, requiredProduction LLM Serving Experience, requiredMeasurable Performance Wins, requiredHigh Autonomy, required

Full job description

← CareersMember of Technical Staff, Inference San Francisco or Seattle · Full time · GPU compute and inference platform (YC S26) Before you apply: a demo video is optional. Enter application

About OpenRelay OpenRelay is GPU compute and LLM inference for people who ship. Customers rent GPU machines and call inference APIs; providers plug their hardware into our network and get paid for it. We run a distributed fleet on real hardware, not a reseller skin over someone else's cloud. We are a small team backed by Y Combinator. Everyone here owns large, important systems that are live in production.

About the role We serve open-weight models on GPUs we run, and throughput per GPU is our margin. You will own that serving layer: which engine runs a model, how it is configured, how requests reach it, and how it holds up when traffic spikes. This is a high-autonomy role. There is no ticket queue. You will find the most important problem, decide how to solve it, ship it to production, and own the result. You will work directly with the founders and help set our technical direction.

Responsibilities

  • Own the performance, cost, and reliability of every model we serve
  • Bring new open-weight models to production quickly, including ones that current engine releases do not yet support
  • Tune serving for each model and GPU: parallelism, batching, caching, and quantization
  • Design how requests are routed and scheduled across replicas and nodes
  • Profile and benchmark under realistic traffic, fix what the numbers point to, and send general fixes upstream
  • Decide what to build next, and say no to work that does not move the numbers

Minimum qualifications

  • 3+ years of industry experience, with production LLM serving a real part of it
  • Hands-on production experience with vLLM, SGLang, and NVIDIA Dynamo
  • Strong Python and PyTorch, and the ability to read and change large codebases you did not write
  • A track record of measurable wins in throughput, latency, or cost, with numbers you can defend
  • You do your best work with high autonomy: you scope your own projects, ship them, and own the outcome
  • Clear written communication in English

Strong candidates may also have

  • Contributions to an open-source inference engine
  • GPU kernel or low-level performance work (CUDA, ROCm, or Triton)
  • Experience serving mixture-of-experts models or running multi-node inference
  • Experience at an early-stage startup

Representative projects

  • Get a newly released open model serving in production within days of its release, then make it fast
  • Cut cost per token on a high-traffic model by retuning parallelism, batching, and caching, and prove it with benchmarks
  • Route repeat prompts to the replicas that already hold their KV cache
  • Trace a tail-latency regression from the request down to a scheduler or kernel, and fix it

The video Optional, five minutes or less, in English. If you record one: your face on camera introducing yourself, and a screen recording walking through a serving system you built or tuned and the numbers it hit. Casual is fine, content matters more than polish.

Logistics

  • Location: San Francisco or Seattle
  • Education: no degree required, equivalent experience counts
  • Compensation: competitive salary and equity

We encourage you to apply even if you do not meet every qualification listed. Strong candidates rarely match all of them.

Enter application

How this job compares

Set against the 6,317 open Machine Learning Engineer jobs on Olive Jobs, counted November 18, 2026.

Pay
  • This postingNot listed
  • California median$175–261K/yr1,424 salary ranges
  • California median$50–70/hr24 hourly ranges
Openings
Olive Jobs lists 1,841 open Machine Learning Engineer jobs in California and 6,317 across the US. 8% of them nationwide are listed as remote or flexible.
Hiring the most in California
Experience
This posting asks for at least 3 years of experience; Machine Learning Engineer postings that state a minimum ask for 5 years at the median (5,234 postings).
Education
2,178 of the 6,317 open Machine Learning Engineer jobs (34%) ask for a bachelor's degree or higher.

Pay, requirements and demand for Software Engineer jobs

Counted across the open, public US postings on Olive Jobs on November 18, 2026. Figures based on too few postings are left out.

Prepare for this role

You've read the whole posting — now see how you match it.