Senior Consultant - Databricks

Role Overview:

Senior Data Engineer

Required Skills and Experience:

• 6+ years of hands-on data engineering experience, with 2+ years specifically working on Databricks in Azure or AWS.
• Proficiency in building and optimizing Spark pipelines (batch and streaming).
• Strong experience implementing bronze/silver/gold data models.
• Working knowledge of cloud storage systems (ADLS, S3) and compute services.
• Experience migrating data from RDBMS (Oracle, SQL Server) or Hadoop ecosystems.
• Familiarity with Airflow, Azure Data Factory, or AWS Glue for orchestration.
• Good scripting skills (Python, Scala, SQL) and version control (Git).

Preferred Qualifications:

• Databricks Certified Data Engineer Associate or Professional.
• Experience with Delta Live Tables (DLT) and Databricks SQL.
• Understanding of cloud security best practices (IAM roles, encryption, ACLs).

Key Responsibilities:

• Design, develop, and operationalize scalable data pipelines on Databricks following medallion architecture principles.
• Migrate and transform large data volumes from traditional on-prem systems (Oracle, Hadoop, Exadata) into cloud data platforms.
• Develop efficient Spark (PySpark/Scala) jobs for ingestion, transformation, aggregation, and publishing of data.
• Implement data quality checks, error handling, retries, and data validation frameworks.
• Build automation scripts and CI/CD pipelines for Databricks workflows and deployment.
• Tune Spark jobs and optimize cost and performance in cloud environments.
• Collaborate with data architects, product owners, and analytics teams.

Attributes for Success:

• Strong analytical and problem-solving skills.
• Attention to scalability, resilience, and cost efficiency.
• Collaborative attitude and passion for clean, maintainable code.