Senior Data Engineer

Senior Data Engineer

reusable data engineering patterns

PySpark, Spark SQL, Delta Lake, and Databricks

Key Responsibilities

• Design, build, and maintain

  ETL/ELT pipelines

  using Databricks following the

  Medallion Architecture (Bronze Silver Gold)

  .
• Implement reusable transformation patterns such as

  conformance, SCD (Slowly Changing Dimensions), deduplication, and data enrichment

  .
• Integrate data pipelines with orchestration tools such as

  Databricks Workflows, Apache Airflow, or AWS Glue

  .
• Capture operational metadata and improve

  observability, monitoring, and alerting

  for data pipelines.
• Collaborate closely with

  Data Quality teams

  to embed rule checks, validations, and data profiling.
• Optimize

  Spark jobs

  for performance, scalability, and cost efficiency.
• Participate in

  code reviews

  , enforce best practices, and mentor junior engineers.
• Contribute to defining

  standardized frameworks and patterns

  to improve platform stability and developer productivity.

Mandatory Skills & Qualifications

• 6+ years

  of experience in

  Data Engineering

  .
• Strong hands-on experience with

  AWS Cloud

  services.
• Advanced proficiency in

  Python and PySpark

  .
• Solid experience with

  Databricks

  (Delta Lake, Workflows, Repos, cluster configuration).
• Strong knowledge of

  Spark SQL

  and distributed data processing concepts.
• Deep understanding of

  Medallion Architecture

  and data modeling best practices.
• Experience implementing

  schema evolution

  and

  SCD logic

  .
• Comfortable working with

  YAML/JSON

  for metadata-driven pipeline design.
• Experience with

  Git, CI/CD pipelines

  , and standard software engineering practices.

  Role & responsibilities