AI Quality Assurance Specialist

Job Description

We are looking for AI Quality Assurance Specialists to join an Electric Utilities AI Delivery team in New Orleans, Louisiana to support the testing and validation of AI-powered applications, ML models, and generative AI solutions. This role blends traditional software QA skills with advanced evaluation methodologies for modern AI models, including LLMs and ML-driven product features. As a member of the team, you will work closely with experienced engineers and machine learning experts to qualify and refine features powered by vision, language, and cross-modal models. You will be responsible for designing rigorous evaluation strategies for both objective and subjective ML behaviors, creating reliable automated testing pipelines, and developing LLM-driven evaluators that complement human judgement. . Responsibilities include:

• Create detailed, comprehensive, and well-structured test plans and test cases for traditional and ML-based solutions. Estimate, prioritize, plan, and coordinate testing activities.

• Design, develop, and execute automation scripts. Identify, record, document, and track bugs. Perform thorough regression testing when bugs are resolved.

• Develop and maintain robust testing frameworks for machine learning models

• Develop and apply testing processes for new and existing products to meet client needs.

• Monitor debugging process results. Investigate the causes of non-conforming software. Track quality assurance metrics, such as defect densities and open defect counts.

• Collaborate with internal teams (e.g. developers, product managers, and designers) to identify testing requirements.

• Identify and prioritize quality control issues from an end-to-end perspective.

• Participate in scrum ceremonies such as refinement, demo, and release planning

We are looking for a QA Specialist (0-5 years of experience), Sr QA Specialist (6-8 years of experience), and Sr Lead QA (10+ years) and pay ranges depending on experience.

Skills and Requirements

7+ years of experience testing AI/ML systems in AWS Experience integrating automated AI testing into CI/CD pipelines (Git/GitHub) Proficient in both white box and black box testing methods. Experience with monitoring and logging systems for post-deployment model validation. Advanced experience testing generative AI systems, including LLMs for accuracy, bias, safety, and hallucinations. Familiarity with RAG evaluation workflows and vector databases (e.g., FAISS, Pinecone, Weaviate). Experience with prompt engineering, adversarial prompting, and synthetic data generation.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to [email protected].