Data Engineer - Databricks

Job Description

About the Role

We’re seeking a Data Engineer with strong Databricks experience to join a high-impact Data Platform team building a modern, analytics-ready lakehouse environment. You’ll play a key role in developing scalable pipelines and delivering trusted datasets that power both internal analytics and customer-facing solutions.

This is a hands-on role across the full data lifecycle, from ingestion through transformation and consumption-ready data models.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to [email protected] learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.

Skills and Requirements

What You’ll Do

• Build and maintain data pipelines in Databricks across batch and streaming patterns

• Develop and optimize Delta Lake / lakehouse transformations (silver/gold layers, facts & dimensions)

• Implement incremental processing, SCD Type 1 & 2 logic, and data quality controls

• Translate business requirements into reusable, production-ready data pipelines

• Contribute to CI/CD and infrastructure-as-code for the data platform

• Monitor and optimize pipeline performance, cost, and reliability

• Collaborate with stakeholders to support data SLAs, contracts, and reporting needs

• Support onboarding of new data sources as the platform scales

What You Bring

• 3+ years of hands-on data engineering experience

• Strong experience with Databricks (required)

• Advanced SQL (window functions, MERGE/upsert patterns, incremental pipelines)

• Proficiency in Python for data ingestion and orchestration

• Experience with Apache Spark and Delta Lake

• Strong understanding of dimensional data modeling (facts, dimensions, SCDs)

• Experience building and supporting production data pipelines

Nice to Have

• Databricks Certified Data Engineer Associate (or similar certification)

• Experience with Delta Live Tables, Unity Catalog, or Databricks workflows

• Exposure to streaming/event-driven architectures (Kafka, etc.)

• Familiarity with Terraform / Infrastructure-as-Code

• Experience supporting BI/reporting tools (Power BI or similar)

Why Join?

• Work on a modern Databricks lakehouse platform with real business impact

• Collaborate with a high-performing, cross-functional team

• Opportunity to deepen expertise in cutting-edge data architecture

• Clear path toward greater technical ownership and growth