Senior Data Engineer — Microsoft Fabric & Databricks (India)

XCentium is looking for a Senior Data Engineer with deep hands-on expertise in Microsoft Fabric and/or Databricks to join our data practice.

 

The ideal candidate can own the end-to-end implementation of complex data pipelines and lakehouse components, translate architectural designs into production-grade solutions, and provide technical guidance to junior engineers. You will work closely with data architects and client stakeholders to build ingestion, transformation, data quality, and semantic modeling layers while leading performance optimization and engineering best practices across medallion architecture tiers.

 

Key Responsibilities

• Own the end-to-end implementation of complex data pipelines and lakehouse components.

• Translate architectural designs into scalable, production-grade data solutions.

• Work closely with data architects and client stakeholders to clarify requirements and technical design decisions.

• Build ingestion, transformation, data quality, and semantic modeling layers across enterprise data platforms.

• Implement raw ingestion at the Bronze layer, cleansing and validation at the Silver layer, and business-ready modeling at the Gold layer.

• Build star schemas in the Gold layer for BI and analytics consumption.

• Implement Change Data Capture (CDC) patterns within the medallion architecture framework.

• Build ingestion pipelines using Azure Data Factory, Dataflows Gen2, Databricks Workflows, or Autoloader.

• Develop solutions using PySpark and Spark Structured Streaming within Fabric and/or Databricks notebooks.

• Design and support real-time and batch ingestion paths using technologies such as Eventstream, Spark Structured Streaming, and scheduled pipelines.

• Optimize Spark and Delta workloads through performance tuning, partitioning, and other engineering best practices.

• Implement and design data quality frameworks and testing standards across medallion tiers.

• Support data catalog, lineage, and governance using Microsoft Purview or Unity Catalog.

• Support master data management integration patterns and related enterprise data requirements.

• Mentor and provide technical guidance to junior engineers.

• Troubleshoot complex pipeline, integration, and performance issues independently.

• Serve as a senior technical resource on client engagements, proof-of-concepts, accelerators, and enterprise-scale implementations.

 

Qualifications

• 6+ years of experience in data engineering or enterprise data platform roles.

• Strong hands-on experience with Microsoft Fabric, including OneLake, Lakehouse, Data Factory, Eventstream, and KQL Database, and/or Databricks, including Workflows, Unity Catalog, and Delta Live Tables.

• Strong proficiency with Delta Lake and its role across lakehouse architecture tiers.

• Strong proficiency with PySpark and Spark Structured Streaming.

• Experience supporting Power BI and DirectLake connectivity or other BI tools consuming Gold-layer data.

• Experience implementing medallion architecture patterns across Bronze, Silver, and Gold layers.

• Experience building star schemas for BI consumption.

• Experience implementing Change Data Capture (CDC) patterns.

• Experience building ingestion pipelines using Azure Data Factory, Dataflows Gen2, Databricks Workflows, or Autoloader.

• Familiarity with source systems such as Salesforce, Oracle Service Cloud, Profisee, or similar master data management platforms.

• Understanding of real-time and batch ingestion architecture.

• Experience with performance tuning, partitioning, and optimization of Spark and Delta workloads.

• Experience implementing data quality frameworks and testing standards.

• Familiarity with Microsoft Purview or Unity Catalog for data cataloging, lineage, and governance.

• Experience with master data management integration patterns.

• Ability to work directly with client stakeholders and architects to clarify requirements and design decisions.

• Ability to operate as the senior technical resource on client engagements.

• Ability to troubleshoot complex pipeline and performance issues independently.

 

Preferred Qualifications

• Microsoft Fabric, Azure Data Engineer, or Databricks certification, such as Databricks Certified Data Engineer Professional.

• Experience working across both Microsoft Fabric and Databricks platforms.

• Experience with CI/CD for data pipelines using Azure DevOps, GitHub Actions, or similar tools.

• Experience mentoring or providing technical guidance to junior engineers.

• Prior consulting experience across multiple client engagements.

• Exposure to Azure AI Foundry or AI/ML integration within the lakehouse.

 

Location

• India

 

How to Apply

Interested in joining XCentium? Please send your resume to Nidhi.Verma@xcentium.com with Senior Data Engineer — Microsoft Fabric & Databricks in the subject line.