Cloud Data Platform Architect

Job Description

A client of Insight Global is seeking an experienced Senior Cloud Data Engineer / Data Platform Engineer to design, implement, test, deploy, and support enterprise-scale data pipelines within a modern Azure and Snowflake environment. This individual will own solutions across the full data lifecycle, from raw data acquisition and ingestion through transformation, curation, and end-user consumption. The engineer will develop scalable, repeatable, maintainable, and highly automated solutions leveraging Azure Data Factory, Azure Functions, Databricks, Snowflake, SQL, and related Big Data technologies. This person will work with data originating from multiple sources and formats, including relational databases, NoSQL databases, files, Kafka, and Big Data platforms.This role will have a strong focus on building reusable platforms and frameworks that provide consistent, verifiable, and automated application and infrastructure management between non-production and production environments. The engineer will support a Cloud DevOps model centered around automation, repeatable engineering patterns, CI/CD, Git, Azure DevOps/GitHub, and Infrastructure as Code using Terraform. The engineer will partner closely with business stakeholders, Cloud administrators, system teams, product teams, data warehouse teams, Business Intelligence teams, and advanced analytics groups to gather requirements and design solutions that improve stability, scalability, security, availability, and performance. Responsibilities also include developing monitoring capabilities, supporting enterprise Cloud security standards, troubleshooting production issues, escalating issues to internal teams and vendors, and participating in a rotational on-call schedule. A major component of this position will involve evaluating and enabling AI capabilities within the Cloud Data Platform. The engineer will research emerging Snowflake and Azure AI capabilities, including Snowflake Cortex, Cortex AI Functions, Cortex Search, Cortex Analyst, Cortex Code Assistant, and Azure AI services. This person will help determine how AI can improve data engineering productivity, Snowflake development, pipeline optimization, documentation, testing, monitoring, data quality, metadata management, and platform automation. The engineer will establish best practices and governance patterns for secure, responsible, and cost-effective AI adoption. This includes understanding AI cost drivers such as token consumption, model selection, Snowflake warehouse consumption, data volume, prompt design, and repeated processing. The role will also reinforce strong foundational data engineering practices, including data cleansing, standardization, curation, semantic layers, reusable pipeline design, and metadata-driven frameworks. This individual will serve as a technical business partner across multiple teams, lead complex design and process initiatives, and provide technical direction across projects. They will help develop project plans involving financial considerations, resource management, and risk while partnering with leadership on the strategic direction of a modern enterprise data platform. Strong presentation and communication skills are essential, as this individual will communicate complex technical concepts to both engineering teams and organizational leadership.

Skills and Requirements

• 10+ years of professional experience in software design and architecture across database, Big Data, and cloud platforms
• Strong SQL expertise
• 5+ years of programming experience with Python, Java, and/or Scala
• 3+ years of Big Data/distributed systems experience, including Azure/AWS and Snowflake
• CI/CD using Azure DevOps and/or GitHub
• Infrastructure as Code experience with Terraform
• Data warehousing and Business Intelligence experience
• Experience designing scalable, automated, reusable data pipelines/frameworks
• Experience researching emerging technologies and translating findings into technology roadmaps
• Strong communication, presentation, technical leadership, and stakeholder management skills
• Snowflake Cortex, Cortex AI Functions, Cortex Search, Cortex Analyst, and Cortex Code Assistant
• AI-ready data products, semantic layers, metadata frameworks, and AI governance
• Understanding of AI cost optimization, including token usage, model selection, compute/warehouse consumption, and prompt design
• Spark, NiFi, Impala, Sqoop, Hive, Hadoop/Cloudera
• Parquet, AVRO, ORC, CSV, and JSON
• Healthcare industry experience, HL7, and Epic data model knowledge
• Performance tuning within Snowflake, ADF, and Big Data environments
• Automated testing frameworks for data pipelines

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to [email protected].