
AI/ML Engineer
Lemma
San Francisco, CAFull-time$120–200K/yrSeen 1mo agoStill listed · checked in the last 2 days
See where you stand before you apply. No sign-up to start.
You apply on the posting's application page, with no Olive account needed.
Watch jobs like this. New roles like this one near San Francisco, CA, by email.
Don't just apply. Show up ready.
Olive works from this exact posting.
At a glance
Olive focuses on jobs in the United States, including remote roles you can work from there.
Job overview
Lemma builds production monitoring for AI agents, detecting silent failures that observability tools miss, such as wrong tool calls, lost context, and infinite loops, to keep agents reliable and trustworthy for users.
Skills & qualifications
Skills
Full job description
Lemma is production monitoring for AI agents. We catch the silent failures your observability tools and evals miss (think bad tool calls, lost context, and infinite loops) before your users find them. Why this role exists Agents break silently. They call the wrong tool, forget what the user said three turns ago, and loop until someone pulls the plug. Soon they'll be responsible for the majority of the world’s economic work, and most teams won't even know when they fail. Making agents reliable is the problem Lemma exists to solve. That means catching the unknown unknowns, the failures nobody thought to write an eval for, and closing the loop end to end so they get fixed, not just flagged. It's the foundation for building agents people can actually trust and the future of self-improving systems. The hardest part of our product is deciding what counts as a failure.
There is no ground truth here, and no benchmark to climb. Every customer's agent is different, what "wrong" means changes from one to the next, and we have to get it right across production without anyone telling us what to look for. Being confidently wrong often costs us more trust than being right fifty times earns. This role owns the intelligence in the loop: what we flag, how sure we are, and whether the fix we propose actually fixes it. What you’ll do
- Own detection quality. Find the failures that don't look like failures: compliant but wrong, omissions, and patterns that only show up across thousands of traces
- Turn implicit signals into evidence. Rephrasing, abandonment, retries, and the other ways users tell you something broke without saying so
- Build the evals for our own system. If we can't measure precision on a problem with no labels, we can't improve it
- Make patch generation trustworthy. Reproduce the failure, verify the fix, and know when to stay quiet instead of opening a bad PR
- Keep it affordable. LLM-as-judge on every event is easy. Doing it at a cost per event that doesn't eat the business is the actual job
- Read real customer traces every week. The best ideas here come from staring at production, not papers
What we’re looking for
- High slope over years of experience. New grads and dropouts welcome
- A track record of shipping things people actually use
- Hands-on with LLMs in production: evals, LLM-as-judge, embeddings, and knowing when a smaller model or no model is the right call
- Real research taste. You can tell a real improvement from noise, even when there's no ground truth to check against
- Bonus: you were the customer once. You ran agents in production and got burned
Who you’ll work with You'll join a team of dropout founders and engineers from Amazon, Together, and Zoom. We've been founding operators at unicorns and at startups that went on to be acquired. Onsite in San Francisco. We don’t sponsor visas.
How this job compares
Set against the 320 open ML Engineer jobs on Olive Jobs, counted November 18, 2026.
- Pay
- This posting$120–200K/yr
- California median$163–261K/yr92 salary ranges
- US median$144–235K/yr248 salary ranges
This range's midpoint, $160K, is 27% below the California median midpoint ($220K) — in the lowest quarter of ML Engineer salary ranges in California.
- Openings
- Olive Jobs lists 129 open ML Engineer jobs in California and 320 across the US. 5% of them nationwide are listed as remote or flexible.
- Experience
- This posting asks for no prior experience — less than three in four ML Engineer postings that state a minimum ask for (median 5 years, across 255 postings).
- Education
- 178 of the 320 open ML Engineer jobs (56%) ask for a bachelor's degree or higher.
Counted across the open, public US postings on Olive Jobs on November 18, 2026. Figures based on too few postings are left out.
Prepare for this role
You've read the whole posting — now see how you match it.