Big Data Engineer

Big Data Engineer

Key Responsibilities

• Data Pipeline Development

  : Design, build, and maintain robust and scalable

  ETL (Extract, Transform, Load)

  pipelines to ingest, process, and transform large datasets from various sources into data warehouses or data lakes.

• Big Data Technologies

  : Work extensively with

  Big Data platforms

  including

  Hadoop, Hive, Pig, and Spark

  , leveraging their capabilities for data processing and analytics.

• Programming & Scripting

  : Utilize

  Python, PySpark, and SQL

  for data manipulation, scripting, and developing data engineering solutions.

• Database Management

  : Interact with and manage data in various database systems such as

  MySQL, SQL Server, and MongoDB

  .

• Data Warehousing

  : Contribute to the design, implementation, and optimization of

  Data Warehouses

  , ensuring data quality and accessibility for business intelligence and reporting.

• Orchestration & Automation (Advantage)

  : (If applicable) Use

  AirFlow

  for scheduling, monitoring, and orchestrating complex data workflows.

• Containerization (Advantage)

  : (If applicable) Leverage

  OpenShift

  for deploying and managing data applications in a containerized environment.

• Troubleshooting & Optimization

  : Identify and resolve data-related issues, and optimize data processes for performance and efficiency.

Skills

• Solid experience in

  ETL / Data Engineering / Data Warehouse

  .
• Strong proficiency in

  Python, PySpark, and SQL

  or other ETL tools.
• Hands-on experience with databases including

  MySQL, SQL Server, MongoDB

  .
• Expertise in

  Hadoop, Hive, Pig, and Spark

  .
• Experience in

  AirFlow

  (big advantage).
• Experience in

  OpenShift

  (big advantage).
• Strong analytical and problem-solving skills.
• Ability to work with large, complex datasets.

Qualifications

• Proven professional experience as a Big Data Engineer or in a similar data-focused role.
• Demonstrable experience in building and managing data pipelines.
• Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.