Opportunity for Data Engineer

Education:

Experience:

• 8+ years of experience in data engineering or a related field.
• Strong hands-on experience with

  Azure Databricks

  ,

  Spark

  ,

  Python/Scala, CICD, Scripting

  for data processing.
• Experience working in multiple file formats like

  Parquet

  ,

  Delta

  , and

  Iceberg

  .
• Knowledge of Kafka or similar streaming technologies for real-time data ingestion.
• Experience with data governance and data security in Azure.
• Proven track record of building large-scale data ingestion and

  ETL pipelines

  in cloud environments, specifically Azure.
• Deep understanding of

  Azure Data Services

  (e.g., Azure Blob Storage, Azure Data Lake, Azure SQL Data Warehouse, Event Hubs, Functions etc.).
• Familiarity with

  data lakes

  ,

  data warehouses

  , and modern data architectures.
• Experience with

  CI/CD pipelines

  , version control (Git), Jenkins and agile methodologies.
• Understanding of cloud infrastructure and architecture principles (especially within

  Azure

  ).

Technical Skills:

• Expert-level proficiency in

  Spark, SPARK Streaming

  , including optimization, debugging, and troubleshooting Spark jobs.
• Solid knowledge of

  Azure Databricks

  for scalable, distributed data processing.
• Strong coding skills in

  Python

  and

  Scala

  for data processing.
• Experience working with

  SQL

  , especially for large datasets.
• Knowledge of data formats like

  Iceberg

  ,

  Parquet

  ,

  ORC

  , and

  Delta Lake

  .

Leadership Skills:

• Proven ability to lead and mentor a team of data engineers, ensuring adherence to best practices.
• Excellent communication skills, capable of interacting with both technical and non-technical stakeholders.
• Strong problem-solving, analytical, and troubleshooting abilities.