Lead Quality Engineer, Test Automation & AI Products

Job Description

Daily Responsibilities: A large healthcare organization is looking for a Lead Quality Engineer to lead quality engineering and test automation across its healthcare and AI products, including generative and agentic AI applications. This person builds reusable automation frameworks for UI, API, integration, data and end-to-end testing, and wires those tests into Azure DevOps pipelines as release quality gates. They also define how to test AI behavior: groundedness, hallucinations, guardrails, agent tool calls and fallback scenarios. They provide hands-on leadership to QA engineers and offshore vendor teams, review vendor test strategy, code and results, and shift quality earlier in development. Because these are regulated healthcare products, they cover HIPAA, PHI/PII, access control and performance testing, and keep traceable test evidence for every release.

• Set test strategy: Define risk-based strategy and quality metrics such as coverage, defect leakage and release readiness.
• Build automation: Create reusable frameworks for web, API, integration and end-to-end testing, with mocks, fixtures and test data.
• Test APIs and data: Validate APIs, events, messaging, databases and FHIR/HL7 integrations, plus data accuracy and reconciliation.
• Test AI: Automate evaluations for groundedness, accuracy, hallucination, guardrails, agent tool calls and human-in-the-loop controls.
• Gate releases: Integrate automated tests and AI evaluations into Azure DevOps CI/CD with clear quality thresholds.
• Test performance: Run load, stress and resiliency tests with JMeter, k6 or Azure Load Testing.
• Lead the team: Mentor QA engineers, oversee offshore and vendor QA, and drive root cause on escaped defects.
• Cover compliance: Test HIPAA, PHI/PII, RBAC and API authorization, and keep traceable test evidence.

Skills and Requirements

Top Requirements:

• 8-10+ years of software quality engineering experience, including senior or lead QA responsibility

• Healthcare application testing, including PHI/PII and regulated data

• Hands-on coding in JavaScript/TypeScript, Java, Python or C#

• Test automation with Playwright, Selenium or Cypress; has built reusable frameworks (not record/playback)

• API and backend automation, plus testing of distributed apps, messaging and integrations

• Automated tests integrated into CI/CD, Azure DevOps preferred

• SQL and data validation

• Performance testing with JMeter, k6 or Azure Load Testing

• Test data management: creation, masking, synthetic data, cleanup

• Security testing basics: authentication, authorization, RBAC, API security

• Technical QA oversight of vendor and offshore teams Additional Skills:

• Home Health, Hospice, Pharmacy, EMR or Provider app testing

• FHIR, HL7 and clinical integration testing

• AI evaluation: groundedness, hallucination, guardrails, tool execution

• Playwright + TypeScript

• Azure, AKS, APIM, Service Bus

• OWASP, SAST/DAST, pen-test remediation validation

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to [email protected].