Big Data Hadoop ETL

As an experienced Big Data Engineer, your role will involve designing, developing, and optimizing scalable Big Data pipelines. You will work closely with cross-functional teams on data acquisition, transformation, and processing. Your key responsibilities will include performing ETL workflows, data ingestion, and data processing using the Hadoop ecosystem. It will be your responsibility to build and maintain data solutions ensuring performance, scalability, and reliability. Additionally, you will need to monitor, troubleshoot, and tune data pipelines to ensure optimal performance. Key Responsibilities: - Design, develop, and optimize scalable Big Data pipelines. - Work closely with cross-functional teams on data acquisition, transformation, and processing. - Perform ETL workflows, data ingestion, and data processing using Hadoop ecosystem. - Build and maintain data solutions ensuring performance, scalability, and reliability. - Monitor, troubleshoot, and tune data pipelines to ensure optimal performance. Qualifications Required: - Strong experience in Big Data / Hadoop Technologies: Hive, HQL, HDFS. - Proficiency in programming languages such as Python, PySpark, and SQL with strong query writing skills. - Familiarity with schedulers like Control-M or equivalent. - Database & ETL experience, particularly in SQL Server / Oracle or similar. - Experience in ETL pipeline development and performance tuning. In addition to the mandatory skills, the following qualifications are preferred: - Experience with GCP Services such as BigQuery, Composer, DataProc, and GCP Cloud architecture. - Knowledge of Agile delivery methodology. - Familiarity with Terraform coding and Infrastructure as Code (IaC) concepts.,