GenAI/LLM Prompt Compliance & Evaluation Consultant

Washington, D.C., DC

Contract

• The Responsible AI Scaled Testing Team within Trust & Safety performs pre-launch structured testing for AI applications against safety, fairness and neutrality policies and standards.

• It is a global team with Responsible AI domain expertise and diverse backgrounds in operations, strategy, ethics, risk management, product management, and program management.

Responsibilities:

• Lead the end-to-end technical assessment of GenAI products, focusing on pre-launch testing of safety, neutrality, and fairness.

• Develop and implement rigorous testing methodologies, including automated prompt generation and response analysis, to ensure compliance with defined standards.

• Leverage data-driven insights to identify potential risks and inform product development iterations, ensuring robust and reliable GenAI deployments.

• Automated Testing and Data Pipeline Management: Design and implement automated prompt generation strategies and data analysis solutions to efficiently collect and analyze GenAI model responses. Manage and optimize data pipelines for efficient processing and analysis of large datasets.

• Quantitative and Qualitative Analysis of Model Behavior: Conduct in-depth statistical analysis and qualitative evaluations of model outputs to identify deviations from defined standards. Develop and apply metrics for evaluating safety, neutrality, and fairness, and generate detailed reports with actionable insights.

• Technical Guideline Development and Execution: Translate abstract safety, neutrality, and fairness standards into precise technical guidelines and evaluation criteria. Develop and maintain clear documentation for vendor teams, including detailed instructions, edge case clarifications, and quality calibration protocols.

Experience:

• 4+ years of experience in data analysis, AI/ML testing, cybersecurity, or related technical domains.

• Proficiency in data analysis tools and languages (e.g., SQL, Python, R) for processing and analyzing large datasets.

• Experience developing and implementing automated testing frameworks.

• Strong analytical and problem-solving skills, with the ability to interpret complex data and identify patterns.

• Excellent technical communication skills, with the ability to clearly articulate complex technical concepts to both technical and non-technical audiences.

• This role may be exposed to graphic, controversial, and/or upsetting content.

Skills:

• 2+ years of experience in AI testing, adversarial testing, red teaming, or related areas (Nice to have).

• Experience with LLM-based prompt generation and evaluation tools (Nice to have).

• Familiarity with machine learning concepts and algorithms (Nice to have).

• Experience with bug tracking systems (Nice to have).

• Experience in developing and maintaining technical documentation (Nice to have).

• Experience in defining and implementing process improvements (Nice to have).

• Ability to think strategically about emerging AI threats and vulnerabilities (Nice to have).

Education:

• Bachelor's degree in Computer Science, Data Science, Statistics, or a related technical field, or equivalent practical experience.

About US Tech Solutions:

US Tech Solutions is a global staff augmentation firm providing a wide range of talent on-demand and total workforce solutions. To know more about US Tech Solutions, please visit www.ustechsolutions.com (http://www.ustechsolutionsinc.com) .

US Tech Solutions is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, colour, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.