Data Engineer

Ro

• We are seeking a highly experienced Data Engineer with over 10+ years of expertise in Data Engineering, Databricks, Azure, SQL, PySpark, and Python.
• The ideal candidate will collaborate with Business and functional teams and create the data pipelines, apply CICD and DevOps practices, and optimize query performance and data transformations.
• This role requires a deep understanding of data governance, quality, and access control policies, as well as the ability to support production data pipelines and contribute to architectural decisions.

Key Responsibilities

• Apply CICD and DevOps practices to automate data workflows and deployments (e.g., with GitHub Actions, Jenkins, Terraform).
• Optimize query performance and data transformations using advanced SQL.
• Implement and uphold data governance, quality, and access control policies.
• Support production data pipelines and respond to issues and performance bottlenecks.
• Contribute to architectural decisions around data strategy and platform scalability.
• Develop and maintain Databricks notebooks and jobs for large-scale batch and streaming data processing.
• Write modular, production-grade PySpark and Python code, including reusable functions and libraries for data transformation.
• Implement streaming data ingestion and Structured Streaming in Databricks for near real-time data solutions.
• Apply performance tuning techniques in Spark, including job optimization, caching, and partitioning strategies.
• Utilize data quality frameworks and testing practices.
• Manage data governance, access controls, and lineage tracking using Unity Catalog.

• Requirements

  : Proven expertise in Databricks, Delta Lake, and Apache Spark (PySpark preferred).
• Deep understanding of Unity Catalog for fine-grained data governance and lineage tracking.
• Proficiency in SQL for large-scale data manipulation and analysis.
• Solid understanding of CICD, infrastructure automation, and DevOps principles.
• Familiarity with cloud platforms (Azure) and data services.
• Strong problem-solving, debugging, and system design skills.
• Excellent communication and collaboration abilities in cross-functional teams.
• Strong hands-on experience with Delta Lake, including table management, schema evolution, and implementing ACID-compliant pipelines.
• Knowledge of performance tuning techniques in Spark, including job optimization, caching, and partitioning strategies.

le & responsibilities

Preferred candidate profile