Azure Datafactory

Azure Datafactory

Azure Synapse Analytics, Azure Data Lake Storage (ADLS), PySpark, Delta Lake, CI/CD for data pipelines, AZURE DATAFACTORY • Experience designing scalable orchestration patterns in ADF (parameterization, reusable components, dependency handling). • Strong understanding of data transformation best practices and implementing efficient processing in Databricks. • Exposure to building data quality checks and operational dashboards for pipeline health and SLA tracking. • Experience collaborating with stakeholders to translate business requirements into technical pipeline designs and delivery plans. • Familiarity with performance tuning approaches for cloud data pipelines and distributed processing workloads. Key Responsibilities: • Design, develop, and maintain end-to-end data pipelines using Azure Data Factory for batch and scheduled workloads. • Build and orchestrate data transformations and processing workflows using Databricks aligned to business requirements. • Implement robust pipeline monitoring, alerting, and failure handling to ensure reliable and repeatable executions. • Perform data validation and reconciliation checks to ensure accuracy, completeness, and consistency across sources and targets. • Optimize pipeline performance by tuning ADF activities, improving orchestration logic, and streamlining transformations in Databricks. • Collaborate with cross-functional teams to gather requirements, estimate effort, and deliver solutions within timelines. • Maintain technical documentation for pipelines, datasets, schedules, dependencies, and operational runbooks. • Support deployments and environment promotions by following structured release and change management practices. Minimum Qualifications: • Education: BTECH, MTECH, MCA, MSC. • Overall experience: 3–5 years in data engineering / data integration roles. • Hands-on experience with Azure Data Factory including pipeline development, scheduling, triggers, and integration patterns. • Hands-on experience with Databricks for data processing and transformation workflows. • Ability to troubleshoot pipeline failures, analyze logs, and implement corrective actions to improve stability.