Data Engineer

Job Summary:

Data Engineer

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data.
  
• Work with

  Azure Databricks

  or

  AWS Glue/EMR

  to create and maintain ETL workflows using

  PySpark

  and

  Python

  .
• Write complex

  SQL

  queries to extract and manipulate data from relational and non-relational data sources.
• Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements.
  
• Optimize performance of large-scale data processing systems and ensure data quality, integrity, and governance.
  
• Deploy and monitor data pipelines in a

  cloud-native

  environment (Azure or AWS).
• Implement best practices for data modeling, metadata management, and data security.
  

Required Skills & Qualifications:

• 5+ years of experience as a Data Engineer or in a similar role.
  
• Proficiency in

  Python

  and

  SQL

  for data manipulation and scripting.
• Strong hands-on experience with

  Databricks

  and

  PySpark

  .
• Experience with

  Azure

  (e.g., Data Factory, Data Lake, Synapse) or

  AWS

  (e.g., S3, Glue, Redshift).
• Knowledge of data warehousing concepts and distributed systems.
  
• Familiarity with CI/CD for data workflows and version control systems like Git.
  
• Experience with data pipeline orchestration tools is a plus (e.g., Airflow, Azure Data Factory).