Data Engineer - Python and Databricks

We are seeking an experienced Data Engineer with expertise in Big Data technologies and a strong background in distributed computing. The ideal candidate will have a proven track record of designing, implementing, and optimizing scalable data solutions using tools like Apache Spark, Python, and various cloud-based platforms.

Key Responsibilities

1. Experience

   : 5-12 years of hands-on experience in Big Data and related technologies.

2. Distributed Computing Expertise

   : Deep understanding of distributed computing principles and their application in real-world data systems.

3. Apache Spark Mastery

   : Extensive experience in leveraging Apache Spark for building large-scale data processing systems.

4. Python Programming

   : Strong hands-on programming skills in Python, with a focus on data engineering and automation.

5. Big Data Ecosystem Knowledge

   : Proficiency with Hadoop v2, MapReduce, HDFS, and Sqoop for managing and processing large datasets.

6. Stream Processing Systems

   : Proven experience in building and optimizing stream-processing systems using technologies like Apache Storm or Spark Streaming.

7. Messaging Systems

   : Experience with messaging and event streaming technologies, such as

   Kafka

   or

   RabbitMQ

   , for handling real-time data.

8. Big Data Querying

   : Solid understanding of Big Data querying tools such as

   Hive

   and

   Impala

   for querying distributed data sets.

9. Data Integration

   : Experience in integrating data from diverse sources like

   RDBMS

   (e.g., SQL Server, Oracle),

   ERP systems

   , and

   flat files

   .

10. SQL Expertise

    : Strong knowledge of SQL, including advanced queries, joins, stored procedures, and relational schemas.

11. NoSQL Databases

    : Hands-on experience with

    NoSQL

    databases like

    HBase

    ,

    Cassandra

    , and

    MongoDB

    for handling unstructured data.

12. ETL Frameworks

    : Familiarity with various ETL techniques and frameworks for efficient data transformation and integration.

13. Performance Optimization

    : Expertise in performance tuning and optimization of Spark jobs to handle large-scale datasets effectively.

14. Cloud Data Services

    : Experience working with cloud-based data services such as

    AWS

    ,

    Azure

    ,

    Databricks

    , or

    GCP

    .

15. Team Leadership

    : Proven ability to lead and mentor teams effectively, ensuring collaboration, growth, and project success.

16. Big Data Solutions

    : Strong experience in designing and implementing comprehensive Big Data solutions that are scalable, efficient, and reliable.

17. Agile Methodology

    : Practical experience working within Agile frameworks to deliver high-quality data solutions in a fast-paced environment.

Please note: The role is having some F2F events so please do not apply from other locations, please be assured your resumes will be highly confidential, and will not be taken ahead without your consent.