AI Agent Evaluation Lead

Essex Labs, Inc. dba Basis seeks AI Agent Evaluation Lead in New York, NY: Job Duties: Establish and lead the company's evaluation function, defining the strategy, frameworks, and standards used company-wide for measuring AI agent quality - rubric design, golden datasets, LLM-as-judge calibration, and behavioral specifications governing agent conduct. Own and develop the company's evaluation platform: the end-to-end system housing 100% of its evals, operated self-serve by engineers and accountants alike. Own the company's judge library: calibrated LLM and code-based graders scoring outputs and trajectories. Design and develop the Basis Accounting Intelligence Benchmark, measuring agent capability across all four accounting practice areas and exposing coverage gaps. Operate the online evaluation pipeline scoring live production agent jobs and catching regressions, and the auditing platform verifying agent context health, behavior fidelity, and eval-data trustworthiness. Own the company's evaluation and training data end to end: synthetic generation with expert curation, a governed and versioned data library, automated quality validation, vendor sourcing and negotiation, and RLHF data processes for model fine-tuning. Plan evaluation coverage for every agent the company builds; train engineers, product staff, and accounting experts through the company's evaluation curriculum and tooling. Requirements: Bachelors degree in Data Science or Computer Science, or related field and one (1) year of experience in the job offered or a related occupation. Special Skill Requirements: 1. Evaluation and benchmark methodology, design, and operations2. Rubric frameworks for Al evaluation and training3. RLHF technique and data processes for generative Al models4. Synthetic data generation with expert curation5. Data annotation, labeling, and curation design and operations at scale6. LLM-as-judge methodology, design, and validation7. Cross-functional program management across research, ML engineering, product, and domain-expert teams8. Statistical and machine-learning methods for data and text quality9. Data platforms: infrastructure, workflows, and schema design10. Analytics and decision-support dashboards for data and Al operationsAny suitable combination of education, training and experience is acceptable. Salary: $200,000.00 - $225,000.00 per year. Submit resume with references to: Req.# 22728.6.4 at: ATTN: Heather Downing, [email protected].