Platform Engineer Consultant

Are you an experienced, passionate pioneer in technology who wants to work in a collaborative environment? As an experienced Platform Engineer Consultant you will have the ability to share new ideas and collaborate on projects as a consultant without the extensive demands of travel. If so, consider an opportunity with Deloitte under our Project Delivery Talent Model. Project Delivery Model (PDM) is a talent model that is tailored specifically for long-term, onsite client service delivery.

Work you'll do/Responsibilities

As a Platform Engineer Consultant, we are seeking a hands-on Senior Data & AI Engineer with deep expertise in the Databricks platform, Python/PySpark development, and emerging agentic AI frameworks. This role combines traditional data engineering/platform operations with modern AI orchestration, requiring someone equally comfortable tuning Spark jobs as debugging an LLM agent's tool-calling loop.

Responsibilities;

• Troubleshoot cluster, job, and workflow failures across Databricks (compute, networking, permissions, library conflicts)

• Perform cost optimization: cluster right-sizing, job cluster vs. all-purpose cluster strategy, auto-scaling policies, spot/on-demand mix, DBU usage analysis

• Conduct performance tuning: Spark query optimization, partitioning strategies, caching, Photon engine utilization, Delta Lake optimization (Z-ordering, compaction, vacuum)

• Administer and automate platform usage via the Databricks REST API and SDKs (workspace provisioning, job orchestration, cluster policies, Unity Catalog management)

Python & PySpark Development

• Build and maintain production-grade ETL/ELT pipelines

• Write performant, testable PySpark code for large-scale distributed data processing

• Apply software engineering best practices (version control, CI/CD, code review, unit/integration testing)

AI / Agentic AI Development

• Design, build, and deploy at least one AI agent end-to-end - from use case definition through production deployment

• Call and orchestrate LLMs (prompt design, context management, tool/function calling, multi-step reasoning chains)

• Implement agent workflows using frameworks such as LangChain and LangGraph (state machines, multi-agent orchestration, tool routing)

• Instrument and monitor agent behavior using LangFuse or similar observability tools (tracing, evaluation, prompt versioning, cost/latency monitoring)

• Collaborate with data scientists/ML engineers to productionize AI-driven features

Azure Cloud Infrastructure

• Design and manage Azure Storage Accounts (ADLS Gen2, blob storage, access tiers, lifecycle policies) integrated with Databricks

• Implement secure secrets management using Azure Key Vault (service principals, managed identities, Databricks secret scopes)

• Work across core Azure services supporting the data platform (e.g., Azure Data Factory, Azure Monitor/Log Analytics, Virtual Networks, Entra ID/RBAC, Azure Databricks workspace configuration)

Infrastructure as Code & Containerization

• Author and maintain Terraform modules for provisioning Databricks workspaces, Azure resources, and networking

• Manage version-controlled, modular IaC with proper state management practices

• Deploy and manage containerized workloads on Kubernetes (deployments, services, scaling, resource management); integrate with Databricks where applicable (e.g., model serving, custom containers)

The Team

Our Deloitte AI & Engineering team helps organizations to transform technology platforms, drive innovation, and help make a significant impact on our clients' success. You'll work alongside talented professionals reimagining and reengineering operations and processes that are critical to businesses. Your contributions can help clients improve financial performance, accelerate new digital ventures, and fuel growth through innovation.

Our AI & Data offering provides a full spectrum of solutions for designing, developing, and operating cutting-edge Data and AI platforms, products, insights, and services. Our offerings help clients innovate, enhance and operate their data, AI, and analytics capabilities, ensuring they can mature and scale effectively.

Qualifications

A successful candidate would possess these skills:

• Ability to work independently and collaborate as part of a team• Effective written and verbal communication skills• Meticulous attention to detail and quality of work product• Ability to build and sustain professional relationships• Ability to lead projects or workstreams• Ability to manage and prioritize multiple tasks in a fast-paced and dynamic environment• Strong interpersonal skills and professional demeanor

Required

• 4+ years of hands-on Databricks experience in production environments

• Strong proficiency in Python and PySpark

• Demonstrated experience building at least one end-to-end AI agent (personal, academic, or professional project acceptable)

• Practical knowledge of LLM orchestration concepts (prompt chaining, tool use, memory/context management)

• Experience with LangChain, LangGraph, LangFuse, or comparable tools (e.g., LlamaIndex, Semantic Kernel, Haystack, Weights & Biases)

• Solid Azure cloud experience, particularly Storage Accounts and Key Vault

• Working knowledge of Terraform for infrastructure provisioning

• Experience deploying/managing workloads on Kubernetes

• Strong troubleshooting and performance optimization mindset

• Limited immigration sponsorship may be available

• Ability to travel 10%, on average, based on the work you do and the clients and industries/sectors you serve. This may include overnight travel.

• Bachelor's degree, preferably in Computer Sciences, Information Technology, Computer Engineering, or related IT discipline; or equivalent experience

Preferred

• Databricks certifications (Data Engineer Associate/Professional, Machine Learning Associate)

• Azure certifications (AZ-104, AZ-204, or AZ-305)

• Experience with Unity Catalog, Delta Live Tables, or MLflow

• Familiarity with vector databases (e.g., Azure AI Search, Pinecone, Chroma) for RAG implementations

• Experience with CI/CD tools (Azure DevOps, GitHub Actions) for MLOps/LLMOps pipelines

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability or protected veteran status, or any other legally protected basis, in accordance with applicable law.