Big Data Engineer (8+ Years )

Job-Big Data Engineer

Location - Chennai or Pune

Qualification:

• Minimum of 8+ years of progressive experience in Data Engineering, with a proven track record in pipeline creation and implementation.
• Strong awareness and hands-on experience with various data tools and platforms, including Hive, Hadoop, Starburst, and other Big Data technologies.
• Proficiency in programming languages such as SQL, Python, Spark, and Java.
• Good working knowledge and practical experience with ETL processes utilizing

  Java Spark, Python Spark, or Spring Batch.

• Advanced SQL skills and extensive experience with relational databases and database design principles.
• Strong proficiency in object-oriented programming languages, specifically Python, PySpark and Java, Java Spark (a must-have).
• Demonstrable experience working with data Lakehouse architectures, such as Iceberg/Starburst.
• Strong proficiency in scripting languages like Bash for automation and system administration tasks.
• Strong proficiency in designing, implementing, and managing data pipeline and workflow management tools (e.g., Apache Airflow, Luigi, etc.).
• Knowledge of cloud platforms such as Databricks,

  AWS

  (Amazon Web Services), or

  GCP

  (Google Cloud Platform) is a significant plus.

Responsibilities:

• Design, build, and maintain efficient and scalable batch or real-time data pipelines within our data platform.
• Maintain and optimize the underlying data infrastructure to facilitate accurate extraction, transformation, and loading of data from diverse sources.
• Develop, implement, and manage ETL (Extract, Transform, Load) processes to extract and manipulate data from multiple heterogeneous sources.
• Automate data workflows, including data ingestion, aggregation, and ETL processing, to improve efficiency and reliability.
• Prepare and transform raw data residing in Data Warehouses into clean, organized, and consumable datasets for both technical and non-technical stakeholders.
• Build, maintain, and deploy data products tailored for analytics and data science teams on our data platform.
• Ensure data accuracy, integrity, privacy, security, and compliance through the implementation of rigorous quality control procedures.
• Proactively monitor data system performance and implement effective optimization strategies to enhance efficiency and scalability.