Spark-Scala, Databricks

Spark-Scala, Databricks

Primary skills:Technology->Big Data - Data Processing->Spark Technology->Data Engineering->Databricks Technology->Functional Programming->Scala Minimum Qualifications: • Bachelor’s/Master’s degree (BE/BTech/MSc/MCA/MTech or equivalent). • 3–5 years of experience in data engineering or ETL development roles. • Strong hands-on experience with Spark using Scala and working on Databricks. • Solid understanding of ETL concepts, data transformations, and pipeline troubleshooting. • Ability to write clean, testable code and collaborate effectively with technical and non-technical stakeholders. Preferred Qualifications: • Experience building end-to-end pipelines on Databricks including notebooks, jobs/workflows, and cluster configuration basics. • Strong SQL skills and experience integrating Spark pipelines with structured data sources and curated datasets. • Familiarity with data quality frameworks, reconciliation strategies, and automated validation checks. • Exposure to CI/CD practices for data engineering (version control, automated testing, release management). • Proven ability to optimize distributed workloads and deliver measurable improvements in runtime and stability. Good to have skills: SQL, Delta Lake, Apache Airflow, Azure Data Lake Storage (ADLS), Git Key Responsibilities: Data Engineering & ETL Development • Design, develop, and maintain ETL pipelines using Spark-Scala on Databricks for batch and incremental processing. • Implement data transformations, joins, aggregations, and validations to ensure accurate and consistent outputs. • Build reusable Spark components and follow best practices for modular, maintainable code. Performance, Reliability & Operations • Tune Spark jobs (partitioning, caching, shuffle optimization) to improve performance and cost efficiency. • Monitor job execution, troubleshoot failures, and provide timely production support with root-cause analysis. • Implement logging, error handling, and data quality checks to improve pipeline reliability. Collaboration & Delivery • Work with cross-functional teams to gather requirements and translate them into technical solutions. • Participate in code reviews, documentation, and knowledge sharing to uplift team standards. • Support release cycles by validating outputs, ensuring backward compatibility, and maintaining deployment readiness.