Hadoop Developer

You will be responsible for leading a migration project from Oracle Cloud to MySQL database (on-premises) with a focus on SQL script and Python migration. Your expertise in on-premises solutions, specifically Linux-based standalone database, and connecting with multiple data sources will be crucial for the success of the project. Your core competencies should include proficiency in Big Data Technologies such as Hadoop (HDFS, YARN), Hive, and PySpark. You should have experience in designing end-to-end data ingestion & ETL pipelines, orchestrating and monitoring them efficiently. Additionally, your skills in cluster management, including primary/secondary cluster configuration, load balancing, and HA setup, will be essential. You should be well-versed in on-premises infrastructure, MySQL databases, and storage & management tools like HDFS and Hive Metastore. Your expertise in programming and scripting languages such as Python, PySpark, and SQL will be required for performing tasks related to performance optimization, Spark tuning, Hive query optimization, and resource management. Experience in building and deploying automated data pipelines in an on-premises environment using CI/CD practices will be advantageous. Furthermore, your knowledge of integrating with logging and monitoring tools, along with experience in security frameworks, will contribute to effective monitoring and governance of the data environment.,