Sr. Data Engineer - Databricks

Celebal Technologies

Remote · USJobPosted 1 day agoStill listed today

Remote · USJobNo compensation foundPosted 1 day agoStill listed today

Evaluate my fit

Most applications go out cold — see where you stand first. No sign-up to start.

Watch jobs like this. New remote roles like this one, by email.

Don't just apply. Show up ready.

Olive works from this exact posting.

Evaluate your fitSee where you stand — strengths, gaps, and what to fix first.
Tailor your résuméRewritten against this posting, every reworded bullet shown before and after.
Practice the interviewA voice mock interview built from this job description.

Evaluate your fitSee where you stand — strengths, gaps, and what to fix first.Get started →Tailor your résuméRewritten against this posting, every reworded bullet shown before and after.Get started →Practice the interviewA voice mock interview built from this job description.Get started →

At a glance

Compensation

No compensation found

Location

Remote · US

Work Authorization

Not specified

Olive lists jobs from US employers, including remote roles you can work from the United States.

Job overview

Celebal Technologies is seeking a senior data engineer focused on Databricks, PySpark, Scala-Spark, and advanced SQL. The role involves building and optimizing data pipelines and ETL processes, implementing or migrating projects to Unity Catalog, and orchestrating workflows with tools such as Databricks Workflow, Azure Data Factory, Apache Airflow, or AWS Glue. The candidate will also support data security, quality, and governance.

Skills & qualifications

RequiredNice to have

Skills

Performance OptimizationSQLCross-Functional Team LeadershipProject DeliveryPerformance TuningDatabase Arch/DesignData GovernanceAWS GlueOrchestrationPySparkAirflowScalaADFInformation SecurityData EngineeringTeradataUnityAzure Data FactoryAmazon Web ServicesHadoopFormulationCloud ServicesTalendMicrosoft AzureAnalyticsBig DataETLGovernanceApache SparkDatabricksScala-SparkAdvanced SQLUnity CatalogData Pipeline DesignData FormulationData Delivery ValidationDatabricks WorkflowApache AirflowAWSData SecurityBig Data Technologies

Qualifications

4+ Years Project DeliveryDatabricks Migration ProjectThree Databricks Optimization ProjectsTwo Orchestration Tools

Full job description

Description: We are seeking a highly skilled and experienced Sr. Data Engineer to join our team. The ideal candidate will have a strong background in data engineering, with a focus on working with Databricks, PySpark, Scala-Spark, and advanced SQL. This role requires hands-on experience in implementing or migrating projects to Unity Catalog, optimizing performance on Databricks Spark, and orchestrating workflows using various tools.

Key Responsibilities:

Data engineering and analytics project delivery experience – Min. 4+ years
Minimum one project done in past of Databricks Migration (Ex. Hadoop to Databricks, Teradata to Databricks, Oracle to Databricks, Talend to Databricks etc)
Hands on with Advanced SQL and Pyspark and/or Scala Spark
Design, develop, and optimize data pipelines and ETL processes using Databricks and Apache Spark.
Implement and optimize performance on Databricks Spark, ensuring efficient data processing and management.
Develop and validate data formulation and data delivery for Big Data projects.
Collaborate with cross-functional teams to define, design, and implement data solutions that meet business requirements.
Conduct performance tuning and optimization of complex queries and data models.
Manage and orchestrate data workflows using tools such as Databricks Workflow, Azure Data Factory (ADF), Apache Airflow, and/or AWS Glue.
Maintain and ensure data security, quality, and governance throughout the data lifecycle

Technical Skills:

Extensive experience with PySpark and Scala-Spark.
Advanced SQL skills for complex data manipulation and querying.
Proven experience in performance optimization on Databricks Spark across at least three projects.
Hands-on experience with data formulation and data delivery validation in Big Data projects.
Experience in data orchestration using at least two of the following: Databricks Workflow, Azure Data Factory (ADF), Apache Airflow, AWS Glue.

Preferred Qualifications:

Experience with cloud platforms such as AWS.
Familiarity with data governance and data security best practices.
Experience with other Big Data technologies and frameworks is a plus

Similar jobs, posted recently

Open roles like this one, listed in the last 30 days.

You've read the whole posting — now see how you match it.

Evaluate my fit