Data Engineer

Job Title :

Location:

Experience:

Role Overview

Data Engineer

Key Responsibilities:

• Design, develop, and maintain scalable big data pipelines and architectures to support business and analytical needs
• Build and optimize data processing systems using technologies such as Hadoop, Hive, Spark, and Python/Java
• Implement data governance principles, ensuring data quality, integrity, and security across all data layers.
• Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
• Develop and manage data workflows and orchestration using tools like Airflow, Azkaban, or Luigi
• Integrate and manage data from diverse sources including SQL, NoSQL databases, APIs, and streaming data.
• Design and maintain real-time data streaming and message queuing systems (e.g., Kafka, Spark Streaming, Storm).
• Support the deployment and monitoring of data systems on cloud platforms such as AWS or Azure.
• Conduct data profiling, cleansing, and validation to ensure accuracy and reliability.
• Continuously improve system performance, scalability, and efficiency through automation and optimization.

Qualification:

• Bachelor's or Graduate degree in Computer Science, Statistics, Informatics, Information Systems, or a related quantitative field.

• 4+ years

  of hands-on experience as a Data Engineer or in a related data engineering
• role.
• Proven expertise with big data technologies such as

  Hadoop, Hive, Spark, and Kafka.

• Strong programming skills in Python, Java, or Scala
• Proficiency with SQL and NoSQL databases (e.g., MySQL, PostgreSQL, Cassandra).
• Experience with data pipeline and workflow management tools such as Airflow, Azkaban, or Luigi.
• Familiarity with cloud data services on AWS or Azure (e.g., EC2, EMR, RDS, Redshift, HDInsights, DataLake, DataBricks).
• Working knowledge of stream-processing frameworks such as Spark Streaming or Storm.
• Strong analytical and problem-solving skills with the ability to design solutions for complex business use cases.
• Excellent project management and organizational skills with experience working in dynamic, cross-functional environments.