Big Data Engineer

Job Summary:

Big Data Engineer

Key Responsibilities:

• Design, develop, and optimize

  Big Data pipelines

  using

  PySpark

  ,

  Hive

  ,

  Sqoop

  , and

  Spark SQL

  .
• Implement

  ETL processes

  for ingesting, transforming, and loading large-scale structured and unstructured data from diverse sources (JSON, Parquet, Avro, ORC).
• Manage and optimize

  data storage

  in

  HDFS

  ,

  AWS S3

  , and

  Azure Data Lake

  .
• Develop and maintain

  data workflows and orchestration

  using

  Apache Airflow

  or similar tools.
• Work with

  AWS Services

  (EC2, S3, Glue, Athena, Redshift, Lambda, DMS) and

  Azure Data Factory

  ,

  Synapse

  , and

  Data Lake

  for end-to-end data pipeline automation.
• Implement

  data integration

  between

  RDBMS

  and

  Big Data systems

  using

  Sqoop

  and other connectors.
• Utilize

  Databricks notebooks

  and

  DBFS

  for data engineering and analytics.
• Ensure

  data quality, validation, and consistency

  across multiple environments.
• Collaborate with cross-functional teams to support

  data migration

  ,

  transformation

  , and

  cloud adoption

  initiatives.
• Apply

  performance tuning

  and

  Spark memory optimization

  for cost-efficient processing.
• Contribute to

  streaming data solutions

  using

  Kafka

  and related technologies.
• Use

  Git

  for version control and collaborate effectively with other developers.

Required Skills:

• Programming:

  PySpark, Python, SQL

• Big Data Tools:

  Spark, Hive, Sqoop, HDFS

• Cloud Platforms:

  AWS (EC2, S3, Glue, Athena, Lambda, Redshift, DMS), Azure (ADF, Synapse, ADLS)

• Workflow Orchestration:

  Apache Airflow

• Data Formats:

  JSON, Parquet, Avro, ORC

• Version Control:

  GitHub

• Data Warehousing:

  Snowflake, Redshift

• ETL & Transformation Tools:

  DBT, Glue

• Streaming:

  Kafka (POC implementation experience)

• Optimization:

  Spark tuning and performance enhancement