Jarmin logo

Member of Technical Staff (Evals)

Jarmin

San Francisco, CAFull-timeSeen 1mo agoStill listed 2 days ago

Evaluate my fit

Most applications go out cold — see where you stand first. No sign-up to start.

Watch jobs like this.

Don't just apply. Show up ready.

Olive works from this exact posting.

At a glance

Compensation
No compensation found
Location
San Francisco, CA
Schedule
Full-time
Work Authorization
Not specified

Olive lists jobs from US employers, including remote roles you can work from the United States.

Job overview

As a Member of Technical Staff on Evals, you will own how AI employee performance is measured, developing evaluation systems, feedback loops, and failure analysis while defining criteria for good, degraded, and failed runs.

Skills & qualifications

RequiredNice to have

Skills

Machine LearningEvalsEvaluation SystemsDataset ManagementOffline ReplayScorers DevelopmentRegression AlertsFeedback LoopsAutonomous Failure AnalysisMetric DefinitionProduction Systems EngineeringLLM Systems DeploymentData AnalysisOpen Source Contribution

Full job description

As a Member of Technical Staff on Evals, you'll own how we measure AI Employee performance on the job. Areas you may work in:

  • Evaluation systems: datasets, offline replay, scorers, and regression alerts
  • Feedback loops for models and agents
  • Autonomous failure analysis
  • Defining what good, degraded, and failed runs look like, and alerting on them

You may be a good fit if you:

  • Have built evaluation or measurement systems, such as AI evals, experimentation, ranking, or search quality
  • Can turn ambiguous quality questions into concrete metrics and decisions
  • Have strong software engineering fundamentals and ship production systems
  • Are comfortable where there are few established patterns
  • Want to work in person in San Francisco

Even better:

  • You've shipped LLM systems to production and learned where they break
  • You have strong data instincts and work well with researchers
  • You contribute to open source projects

Similar jobs, posted recently

Open roles like this one, listed in the last 30 days.

You've read the whole posting — now see how you match it.

Evaluate my fit