Skip to main content
Ellipsis Health logo

Senior Backend Engineer, AI Evaluations

Ellipsis Health

Remote · USFull-time$160–210K/yrPosted 4mo agoStill listed 5 days ago

See where you stand before you apply. No sign-up to start.

You apply on the posting's application page, with no Olive account needed.

Watch jobs like this.

Don't just apply. Show up ready.

Olive works from this exact posting.

At a glance

Compensation
$160–210K/yr
Location
Remote · US
Schedule
Full-time
Work Authorization
Not specified

Olive focuses on jobs in the United States, including remote roles you can work from there.

Job overview

Ellipsis Health is hiring a Senior Backend Engineer, AI Evaluations. Ellipsis Health seeks a strong Senior Backend Engineer to join its AI Evaluation team, building infrastructure and internal tooling for reliable, repeatable AI system evaluations in production. The role involves developing APIs, data integrations, test configuration management, and reporting tools, as well as productionalizing evaluation frameworks for components like ASR, LLMs, and TTS.

Key focus areas include Build and maintain infrastructure and tooling for the AI evaluations platform, including automated testing and debugging tools, Develop and productionalize evaluation frameworks for system components such as ASR, LLMs, TTS, knowledge bases, and guardrails, and Partner with ML, engineering and QA teams to translate evaluation requirements into robust, maintainable infrastructure.

Successful candidates bring 5+ Years Professional Software Engineering Experience, Experience Building Backend Systems, and Experience Building Platforms. Important skills include Python, TypeScript, JavaScript, Java, Test Automation Frameworks, and Evaluation Pipelines. Preferred (not required): ASR, LLMs, TTS, and ML Powered Services.

Skills & qualifications

RequiredNice to have

Skills

Python, requiredTypeScript, requiredJavaScript, requiredJava, requiredTest Automation Frameworks, requiredEvaluation Pipelines, requiredCI/CD Integrated Testing Systems, requiredObservability Tools, requiredDebugging Tools, requiredLogging, requiredMetrics, requiredTracing, requiredBuilding Internal Tools, requiredMethodical Approach to Diagnosing Issues, requiredCollaboration, requiredProduct Minded Approach to Infrastructure, requiredUsability, requiredDocumentation, requiredLong Term Maintainability, requiredASR, nice to haveLLMs, nice to haveTTS, nice to haveML Powered Services, nice to haveHealthcare Industry Experience, nice to haveRegulated Environments, nice to haveHIPAA, nice to havePHI Handling, nice to haveConversational AI, nice to haveVoice Agents, nice to haveMulti Turn Dialogue, nice to haveLatency Constraints, nice to haveError Recovery, nice to haveLangfuse, nice to havePrompt Eval Frameworks, nice to haveDigital Health, nice to haveCare Coordination, nice to havePatient Facing Systems, nice to have

Qualifications

5+ Years Professional Software Engineering Experience, requiredExperience Building Backend Systems, requiredExperience Building Platforms, requiredExperience Building Developer Tooling, requiredExperience Designing Production Grade Infrastructure With Code, required

Benefits

401(k) Match
Medical Insurance
Vision Insurance
Dental Insurance
Paid Time Off

Full job description

About the Role We’re seeking a strong Senior Backend Engineer to join our AI Evaluation team, focused on building the infrastructure and internal tooling that enable reliable, repeatable evaluation of AI systems in production. In this role, you’ll develop the scaffolding around our evaluations platform - API/data integrations, test configuration management, and reporting tools that make evaluations easy to run, extend, and operate at scale. You’ll also build and maintain evaluation frameworks for individual system components such as ASR, LLMs, TTS, knowledge bases, and guardrails, productionalizing these frameworks to support regular analysis, regression detection, and continuous monitoring.

In addition, you’ll create debugging tools that make it easy to inspect end-to-end calls, trace failures, and surface all relevant signals in one place - empowering internal teams to diagnose issues quickly and confidently. The ideal candidate is a pragmatic, infrastructure-minded engineer who enjoys turning ad hoc analysis into durable systems, cares deeply about developer experience, holds a high bar for software-quality and takes pride in building tooling that makes complex AI systems observable, testable, and easier to operate.

Responsibilities:

  • Build and maintain infrastructure and tooling for the AI evaluations platform used by internal teams, including automated testing platform for AI voice agents, debugging and observability tools.

  • Develop and productionalize evaluation frameworks for individual system components such as ASR, LLMs, TTS, knowledge bases, and guardrails.

  • Partner with ML, engineering and QA teams to translate evaluation requirements into robust, maintainable infrastructure and tooling.

  • Improve developer experience by making evaluation systems easy to extend, well-documented, and reliable in day-to-day use.

  • Ensure evaluation tooling meets production standards for reliability, performance, and maintainability.

Qualifications:

  • 5+ years of professional software engineering experience, with a strong focus on building backend systems, platforms, or developer tooling.

  • Proven experience designing and maintaining production-grade infrastructure with code, including APIs, services, and data pipelines.

  • Strong proficiency in at least one general-purpose programming language (e.g., Python, Typescript/Javascript, Java, or similar).

  • Experience using test automation frameworks, evaluation pipelines, or CI/CD-integrated testing systems.

  • Familiarity with observability and debugging tools (logging, metrics, tracing) and building internal tools that improve developer and QA workflows.

  • Strong debugging skills and a methodical approach to diagnosing production and evaluation issues.

  • Ability to collaborate effectively across engineering, QA, and operations teams, translating requirements into reliable, maintainable systems.

  • Product-minded approach to infrastructure, with attention to usability, documentation, and long-term maintainability.

Preferred:

  • Experience working with complex, multi-component systems (e.g., ASR, LLMs, TTS, or other ML-powered services)

  • Experience working in healthcare or other regulated environments, including awareness of HIPAA and PHI handling.

  • Familiarity with conversational AI or voice agents, including multi-turn dialogue, latency constraints, and error recovery.

  • Familiarity with LLM observability or evaluation tools (e.g., Langfuse, prompt eval frameworks).

  • Background in digital health, care coordination, or patient-facing systems.

Salary and Benefits We offer competitive salary and benefits, including 401(k) matching, health, vision, and dental insurance, and very flexible paid time off.

The typical salary range for this role is $160,000 to $210,000 USD, depending on skills, qualifications, and relevant experience.

Background Checks As a health technology company, we reserve the right to run background checks on candidates to whom we extend offers, in compliance with applicable laws. We evaluate candidates holistically and comply with all “ban the box” regulations.

Assistance If you have a disability or require accommodations during the application or recruitment process, please contact [email protected].

How this job compares

Set against the 2,906 open Backend Engineer jobs on Olive Jobs, counted November 16, 2026.

Pay
  • This posting$160–210K/yr
  • California median$170–246K/yr742 salary ranges
  • US median$156–224K/yr2,352 salary ranges

This range's midpoint, $185K, is 12% below the California median midpoint ($210K) — in the middle half of Backend Engineer salary ranges in California.

Openings
Olive Jobs lists 944 open Backend Engineer jobs in California and 2,906 across the US. 14% of them nationwide are listed as remote or flexible.
Hiring the most in California
Experience
This posting asks for at least 5 years of experience; Backend Engineer postings that state a minimum ask for 5 years at the median (2,423 postings).
Education
561 of the 2,906 open Backend Engineer jobs (19%) ask for a bachelor's degree or higher.

Counted across the open, public US postings on Olive Jobs on November 16, 2026. Figures based on too few postings are left out.

Prepare for this role

You've read the whole posting — now see how you match it.