Senior AI Quality Assurance (QA) Engineer
Most applications go out cold — see where you stand first. No sign-up to start.
Don't just apply. Show up ready.
Olive works from this exact posting — no sign-up to start.
At a glance
Job overview
DATAMAXIS is hiring a Senior AI Quality Assurance (QA) Engineer. DATAMAXIS seeks a Senior AI Quality Assurance Engineer in Bengaluru to lead testing of LLM‑powered applications and AI workflows, develop benchmark datasets, and build automated test frameworks using Python and Pytest while ensuring robust API and SQL validation.
Key focus areas include Evaluate LLM‑powered applications and conversational AI systems, Create benchmark datasets for regression testing of AI workflows, and Develop automated tests using Python and Pytest.
Important skills include Evaluating LLM-Powered Applications, Conversational AI Systems, NL2SQL Solutions, AI Workflows, AI Benchmarking Methodologies, and Creating Benchmark Datasets.
Skills & qualifications
Skills
Qualifications
Full job description
Job Position: Senior AI Quality Assurance (QA) Engineer
Experience: 7+ Years
Location: Bengaluru
Required Skills
-
7+ years of experience in Software Quality Assurance, Test Automation, or AI Quality Engineering.
-
Hands-on experience evaluating LLM-powered applications, conversational AI systems, NL2SQL solutions, or similar AI workflows.
-
Strong understanding of AI benchmarking methodologies and experience creating benchmark (golden) datasets for regression testing.
-
Experience with AI evaluation and observability platforms such as Lang Smith, MLflow, Arize Phoenix, or equivalent.
-
Strong understanding of AI evaluation metrics, including Precision, Recall, Precision@K, Recall@K, F1 Score, Exact Match (EM), Mean Reciprocal Rank (MRR), Execution Accuracy, latency (P50/P95/P99), token usage, and cost analysis.
-
Strong proficiency in Python with hands-on experience building automation using Pytest, including unit testing, integration testing, API testing, exception handling, mocking, fixtures, and parameterized testing.
-
Experience testing REST APIs and distributed backend services.
-
Strong SQL skills with the ability to validate generated queries, execution results, schema alignment, and business logic.
-
Strong analytical and debugging skills with the ability to perform root cause analysis across complex AI systems.
You've read the whole posting — now see how you match it.