
AI Applied Scientist, AI Evaluation Science
Seattle, WAFull-timeSeen todayIn the employer's feed · checked in the last 2 days
See where you stand before you apply. No sign-up to start.
You apply on the posting's application page, with no Olive account needed.
Watch jobs like this. New roles like this one near Seattle, WA, by email.
Don't just apply. Show up ready.
Olive works from this exact posting.
At a glance
Olive focuses on jobs in the United States, including remote roles you can work from there.
Job overview
Apple’s AI Evaluation Science team develops rigorous, scalable methods and tools for evaluating LLMs, agentic systems, and human-AI interaction. The Applied Scientist will turn research into self-service evaluation capabilities, collaborate with measurement scientists and engineers, and help teams interpret results and make product decisions. The role also involves developing new methods when existing approaches fall short and sharing findings through publications and internal research reports.
Skills & qualifications
Skills
Qualifications
Full job description
Weekly Hours: 40
Role Number: 200695298-3337
Summary
AI systems are only as trustworthy as the methods used to evaluate them. At Apple, where AI powers experiences for billions of people, getting evaluation right is not a support function, it is a foundational science. Our team, part of Apple Services Engineering, is building that scientific foundation: rigorous, scalable evaluation methodology for LLMs, agentic systems, and human-AI interaction. Our goal is evidence that teams can act on: what to measure, whether a change helped, why a system failed, and whether it is ready to ship.
What makes this team unusual is its interdisciplinary core. You will work alongside measurement scientists (psychometrics, validity theory), ML researchers, and platform engineers, bringing together ML research, statistical rigor, and production engineering.
We are looking for an Applied Scientist who works across the full arc of evaluation science: turning research into tools engineers can use on their own, and turning what those engineers run into new research. You care whether a method holds up on real data, at real scale, for a real decision, and you are curious enough to go after the open question underneath when it doesn't.
Description
Our team is building self-service evaluation: tools, SDKs, and AI-powered capabilities that let engineers across Apple evaluate their AI features on their own, quickly and correctly, so they can build and ship those features faster. You will develop the methods that make this possible: raising the quality of the evaluations engineers run, and helping them understand what their results mean and what to do next. Often the method a problem needs doesn't exist yet. When that happens, you will figure it out, whether that means extending a recent paper or developing something new, and share what you learn through publications and internal research reports. We are looking for individuals across a range of experience levels.
With research and measurement scientists, you will co-develop new methods, make sure the tools give engineers valid, trustworthy signals, and bring back the hard problems that should shape the team's research agenda. With platform and research engineers, you will define what each method needs to work reliably without an expert in the loop, including its assumptions, failure modes, and validation tests, so it can ship in the evaluation SDKs and APIs used across Apple.
The successful candidate combines scientific depth with practical judgment: you know which methods fit a problem and why, can design experiments that show whether they actually work, can diagnose why they break down on messy real-world data, and can explain clearly what an evaluation result does and does not support.
Minimum Qualifications
-
Ph.D. in Computer Science, Machine Learning, Statistics, or a closely related field, or a Master's degree in one of these fields with 3+ years of applied research experience.
-
Deep expertise in evaluating AI systems, gained through research on evaluation methods, substantial hands-on experience evaluating LLMs or agentic systems in real products, or both.
-
Strong proficiency in Python and the modern ML ecosystem, with demonstrated ability to implement complex methods from recent papers and run large-scale experiments.
-
Hands-on experience with large language models, whether fine-tuning, inference, prompting pipelines, or building LLM-powered applications.
-
Demonstrated research ability through publications at peer-reviewed venues, preprints, or equivalent research output.
-
Track record of applying research to practical problems: methods, prototypes, or tools adopted by other teams or used to inform real product decisions.
-
Solid software engineering fundamentals for research code: version control, testing, reproducibility, and code others can build on.
-
Excellent written and verbal communication skills, including the ability to explain complex results, and their limitations, to diverse audiences.
-
Comfort with ambiguity and new problem spaces; you thrive when building something that doesn't yet have a playbook.
Preferred Qualifications
-
First-author publications at top-tier venues (NeurIPS, ICML, ICLR, ACL, EMNLP), especially on evaluation methodology: papers about how to evaluate, not just papers that use evaluation to demonstrate model improvements.
-
Specific experience in one or more of: LLM-as-judge approaches (calibration, rubric design, bias mitigation); benchmark design and validation (IRT, contamination detection); human evaluation methodology (protocol design, quality control); experimentation and comparisons between system versions (A/B testing, uncertainty quantification); or agentic and multi-agent system evaluation, including failure diagnosis and automated red-teaming.
-
Familiarity with theoretical foundations of evaluation: measurement theory and validity frameworks, statistical learning theory (calibration, reliability, decision theory), causal inference, or preference aggregation.
-
Experience with the practical challenges of evaluating generative AI and agentic systems: non-deterministic outputs, token economics, multi-turn agent trajectories, and tool use.
-
Experience building methods or tools used directly by non-experts, such as developer-facing ML tooling or self-service analytics.
-
Familiarity with modern evaluation tools and frameworks (e.g., DeepEval, Ragas, TruLens, LangSmith, or agent tracing and observability tools).
-
Experience working with distributed computing or cloud-native tooling (e.g., Ray, Spark, Docker/Kubernetes) to run evaluations at scale.
-
Experience with cross-disciplinary work, such as collaboration with social scientists, psychometricians, or domain experts.
-
Experience in fast-moving, early-stage teams where you helped define technical direction from the ground up.
Apple is an equal opportunity employer that is committed to inclusion and diversity. We seek to promote equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics.
How this job compares
Set against the 896 open Applied Scientist jobs on Olive Jobs, counted November 18, 2026.
- Pay
- This postingNot listed
- Washington median$143–226K/yr237 salary ranges
- Openings
- Olive Jobs lists 244 open Applied Scientist jobs in Washington and 896 across the US. 1% of them nationwide are listed as remote or flexible.
- Hiring the most in Washington
- Amazon189 open
- Microsoft Corporation33 open
- Oracle5 open
- Experience
- This posting asks for at least 3 years of experience; Applied Scientist postings that state a minimum ask for 3 years at the median (710 postings).
- Education
- 298 of the 896 open Applied Scientist jobs (33%) ask for a bachelor's degree or higher.
Counted across the open, public US postings on Olive Jobs on November 18, 2026. Figures based on too few postings are left out.
Prepare for this role
You've read the whole posting — now see how you match it.

