AI Engineer II

Job Description

Insight Global is seeking an AI Engineer II (AI-Ready Data) for a top research and technology-driven client. This candidate will play a critical role in transforming complex research, survey, and operational datasets into reliable, AI-ready data that can power AI assistants, agents, RAG solutions, and analytical tools. They will build Python and Databricks-based data pipelines, create metadata and semantic layers, develop retrieval components using embeddings and vector indexes, and ensure data quality and governance standards are met. The ideal candidate combines strong Python and SQL development skills with Databricks, data science, or research data expertise and has hands-on exposure to modern LLM and RAG technologies. This is a great opportunity for a mid-level engineer who wants ownership over AI-focused data engineering initiatives while partnering closely with research teams and mentoring junior engineering talent. Provide your feedback on BizChat

Skills and Requirements

Master’s degree in Data Science, Statistics, Computer Science, quantitative social science, or related field OR bachelor’s degree with equivalent experience 3–5 years of experience in data engineering, data science, or AI/ML development Strong Python development skills with pandas, PySpark, or similar technologies Strong SQL proficiency Hands-on Databricks experience and/or strong data science/research data background working with survey, administrative, or statistical datasets Experience preparing, cleaning, structuring, and transforming data for analytics or AI use Experience with LLMs, embeddings, vector databases, and/or RAG workflows Understanding of data quality, metadata management, and data governance Strong communication skills with both technical and research stakeholders Combination of hands-on Databricks experience and research/data science experience Delta Lake, Unity Catalog, or MLflow Survey methodology, codebooks, weighting, or statistical data experience R, Stata, or SAS AWS or Azure cloud services Experience handling sensitive or restricted datasets, including PII, PHI, or federal project data Experience building semantic layers and metadata for AI consumption Experience with data lineage and quality-validation frameworks

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to [email protected].