Data Analytics Engineer

Data Engineer

Key Responsibilities

• Design, develop, and maintain

  data pipelines

  (batch and real-time) using

  PySpark

  ,

  Scala

  , and

  Azure Databricks

  .
• Build and optimize

  data ingestion frameworks

  integrating streaming and static data sources.
• Implement

  real-time data streaming

  solutions using tools like

  Apache Kafka

  ,

  Azure Event Hubs

  , or

  Spark Structured Streaming

  .
• Develop and deploy

  ETL/ELT workflows

  leveraging

  Azure Data Factory

  ,

  Databricks

  , and related Azure data services.
• Work with

  SQL Server

  for relational data storage and

  NoSQL databases

  (Cassandra, Cosmos DB, MongoDB, etc.) for semi-structured/unstructured data.
• Design and maintain scalable

  data lake

  and

  data warehouse

  architectures in

  Azure Data Lake

  and

  Synapse Analytics

  .
• Ensure

  data quality

  ,

  security

  , and

  governance

  through proper validation, monitoring, and auditing processes.
• Collaborate with cross-functional teams including data scientists, analysts, and DevOps — to support analytics and AI workloads.
  
• Optimize Spark jobs and pipelines for

  performance, reliability, and cost efficiency

  .
• Implement

  data versioning

  ,

  change data capture (CDC)

  , and

  real-time event-driven architectures

  where applicable.

Required Skills and Qualifications

• Bachelor’s or Master’s degree

  in Computer Science, Information Systems, or related field.

• 3+ years

  of experience in

  data engineering

  or a related role.
• Proficiency in

  Python

  and

  PySpark

  for distributed data processing.
• Experience with

  Scala

  for Spark development.
• Hands-on expertise in

  Azure Databricks

  ,

  Azure Data Factory

  , and

  Azure Data Lake

  .
• Strong SQL skills with

  SQL Server

  (queries, stored procedures, indexing, tuning).
• Knowledge of

  NoSQL databases

  (Cassandra, Cosmos DB, MongoDB, etc.).
• Practical experience with

  data streaming frameworks

  such as:

• Apache Kafka

  ,

  Azure Event Hubs

  , or

  AWS Kinesis

  .

• Spark Structured Streaming

  or

  Flink

  .
• Understanding of

  data modeling

  ,

  data warehousing

  , and

  ETL/ELT principles

  .
• Familiarity with

  data pipeline orchestration

  ,

  CI/CD

  , and

  version control

  tools (Azure DevOps, Git, etc.).
• Strong analytical and problem-solving abilities, with excellent communication skills.