Skip to main content
Apple logo

AI Applied Scientist, AI Evaluation Science

Apple

Seattle, WAFull-timeSeen todayIn the employer's feed · checked in the last 2 days

See where you stand before you apply. No sign-up to start.

You apply on the posting's application page, with no Olive account needed.

Watch jobs like this.

Don't just apply. Show up ready.

Olive works from this exact posting.

At a glance

Compensation
No compensation found
Location
Seattle, WA
Schedule
Full-time
Work Authorization
Not specified

Olive focuses on jobs in the United States, including remote roles you can work from there.

Job overview

Apple’s AI Evaluation Science team develops rigorous, scalable methods and tools for evaluating LLMs, agentic systems, and human-AI interaction. The Applied Scientist will turn research into self-service evaluation capabilities, collaborate with measurement scientists and engineers, and help teams interpret results and make product decisions. The role also involves developing new methods when existing approaches fall short and sharing findings through publications and internal research reports.

Skills & qualifications

RequiredNice to have

Skills

AI Evaluation, requiredPython, requiredModern ML Ecosystem, requiredLarge-Scale Experimentation, requiredLarge Language Models, requiredFine-Tuning, requiredLLM Inference, requiredPrompting Pipelines, requiredLLM Applications, requiredResearch Publications, requiredResearch Application, requiredVersion Control, requiredSoftware Testing, requiredReproducibility, requiredWritten Communication, requiredVerbal Communication, requiredLLM-as-Judge, nice to haveCalibration, nice to haveRubric Design, nice to haveBias Mitigation, nice to haveBenchmark Design, nice to haveBenchmark Validation, nice to haveItem Response Theory, nice to haveContamination Detection, nice to haveHuman Evaluation, nice to haveProtocol Design, nice to haveQuality Control, nice to haveA/B Testing, nice to haveUncertainty Quantification, nice to haveAgentic System Evaluation, nice to haveFailure Diagnosis, nice to haveAutomated Red-Teaming, nice to haveMeasurement Theory, nice to haveValidity Frameworks, nice to haveStatistical Learning Theory, nice to haveReliability, nice to haveDecision Theory, nice to haveCausal Inference, nice to havePreference Aggregation, nice to haveGenerative AI Evaluation, nice to have

Qualifications

PhD in CS, ML, Statistics or Related, requiredMaster's in CS, ML, Statistics or Related or 3+ Years Applied Research, required3+ Years Applied Research Experience, required

Full job description

Weekly Hours: 40

Role Number: 200695298-3337

Summary

AI systems are only as trustworthy as the methods used to evaluate them. At Apple, where AI powers experiences for billions of people, getting evaluation right is not a support function, it is a foundational science. Our team, part of Apple Services Engineering, is building that scientific foundation: rigorous, scalable evaluation methodology for LLMs, agentic systems, and human-AI interaction. Our goal is evidence that teams can act on: what to measure, whether a change helped, why a system failed, and whether it is ready to ship.

What makes this team unusual is its interdisciplinary core. You will work alongside measurement scientists (psychometrics, validity theory), ML researchers, and platform engineers, bringing together ML research, statistical rigor, and production engineering.

We are looking for an Applied Scientist who works across the full arc of evaluation science: turning research into tools engineers can use on their own, and turning what those engineers run into new research. You care whether a method holds up on real data, at real scale, for a real decision, and you are curious enough to go after the open question underneath when it doesn't.

Description

Our team is building self-service evaluation: tools, SDKs, and AI-powered capabilities that let engineers across Apple evaluate their AI features on their own, quickly and correctly, so they can build and ship those features faster. You will develop the methods that make this possible: raising the quality of the evaluations engineers run, and helping them understand what their results mean and what to do next. Often the method a problem needs doesn't exist yet. When that happens, you will figure it out, whether that means extending a recent paper or developing something new, and share what you learn through publications and internal research reports. We are looking for individuals across a range of experience levels.

With research and measurement scientists, you will co-develop new methods, make sure the tools give engineers valid, trustworthy signals, and bring back the hard problems that should shape the team's research agenda. With platform and research engineers, you will define what each method needs to work reliably without an expert in the loop, including its assumptions, failure modes, and validation tests, so it can ship in the evaluation SDKs and APIs used across Apple.

The successful candidate combines scientific depth with practical judgment: you know which methods fit a problem and why, can design experiments that show whether they actually work, can diagnose why they break down on messy real-world data, and can explain clearly what an evaluation result does and does not support.

Minimum Qualifications

  • Ph.D. in Computer Science, Machine Learning, Statistics, or a closely related field, or a Master's degree in one of these fields with 3+ years of applied research experience.

  • Deep expertise in evaluating AI systems, gained through research on evaluation methods, substantial hands-on experience evaluating LLMs or agentic systems in real products, or both.

  • Strong proficiency in Python and the modern ML ecosystem, with demonstrated ability to implement complex methods from recent papers and run large-scale experiments.

  • Hands-on experience with large language models, whether fine-tuning, inference, prompting pipelines, or building LLM-powered applications.

  • Demonstrated research ability through publications at peer-reviewed venues, preprints, or equivalent research output.

  • Track record of applying research to practical problems: methods, prototypes, or tools adopted by other teams or used to inform real product decisions.

  • Solid software engineering fundamentals for research code: version control, testing, reproducibility, and code others can build on.

  • Excellent written and verbal communication skills, including the ability to explain complex results, and their limitations, to diverse audiences.

  • Comfort with ambiguity and new problem spaces; you thrive when building something that doesn't yet have a playbook.

Preferred Qualifications

  • First-author publications at top-tier venues (NeurIPS, ICML, ICLR, ACL, EMNLP), especially on evaluation methodology: papers about how to evaluate, not just papers that use evaluation to demonstrate model improvements.

  • Specific experience in one or more of: LLM-as-judge approaches (calibration, rubric design, bias mitigation); benchmark design and validation (IRT, contamination detection); human evaluation methodology (protocol design, quality control); experimentation and comparisons between system versions (A/B testing, uncertainty quantification); or agentic and multi-agent system evaluation, including failure diagnosis and automated red-teaming.

  • Familiarity with theoretical foundations of evaluation: measurement theory and validity frameworks, statistical learning theory (calibration, reliability, decision theory), causal inference, or preference aggregation.

  • Experience with the practical challenges of evaluating generative AI and agentic systems: non-deterministic outputs, token economics, multi-turn agent trajectories, and tool use.

  • Experience building methods or tools used directly by non-experts, such as developer-facing ML tooling or self-service analytics.

  • Familiarity with modern evaluation tools and frameworks (e.g., DeepEval, Ragas, TruLens, LangSmith, or agent tracing and observability tools).

  • Experience working with distributed computing or cloud-native tooling (e.g., Ray, Spark, Docker/Kubernetes) to run evaluations at scale.

  • Experience with cross-disciplinary work, such as collaboration with social scientists, psychometricians, or domain experts.

  • Experience in fast-moving, early-stage teams where you helped define technical direction from the ground up.

Apple is an equal opportunity employer that is committed to inclusion and diversity. We seek to promote equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics.

How this job compares

Set against the 896 open Applied Scientist jobs on Olive Jobs, counted November 18, 2026.

Pay
  • This postingNot listed
  • Washington median$143–226K/yr237 salary ranges
Openings
Olive Jobs lists 244 open Applied Scientist jobs in Washington and 896 across the US. 1% of them nationwide are listed as remote or flexible.
Hiring the most in Washington
Experience
This posting asks for at least 3 years of experience; Applied Scientist postings that state a minimum ask for 3 years at the median (710 postings).
Education
298 of the 896 open Applied Scientist jobs (33%) ask for a bachelor's degree or higher.

Counted across the open, public US postings on Olive Jobs on November 18, 2026. Figures based on too few postings are left out.

Prepare for this role

You've read the whole posting — now see how you match it.