Job
Description
What you’ll doAs a Data Engineer – Data Platform Services, you will be responsible for: Data Migration & Modernization Leading the migration of ETL workflows from IBM DataStage to PySpark, ensuring performance optimization and cost efficiency. Designing and implementing data ingestion frameworks using Kafka and PySpark, replacing legacy ETL Pipeline using DataStage. Migrating the analytical platform from IBM Integrated Analytics System (IIAS) to Cloudera Data Lake on CDP. Data Engineering & Pipeline Development Developing and maintaining scalable, fault-tolerant, and optimized data pipelines on Cloudera Data Platform. Implementing data transformations, enrichment, and quality checks to ensure accuracy and reliability. Leveraging Denodo for data virtualization and enabling seamless access to distributed datasets. Performance Tuning & Optimization Optimizing PySpark jobs for efficiency, scalability, and reduced cost on Cloudera. Fine-tuning query performance on Iceberg tables and ensuring efficient data storage and retrieval. Collaborating with Cloudera ML engineers to integrate machine learning workloads into data pipelines. Security & Compliance Implementing Thales CipherTrust encryption and tokenization mechanisms for secure data processing. Ensuring compliance with Bank/regulatory body security guidelines, data governance policies, and best practices. Collaboration & Leadership Working closely with business stakeholders, architects, and data scientists to align solutions with business goals. Leading and mentoring junior data engineers, conducting code reviews, and promoting best practices. Collaborating with DevOps teams to streamline CI/CD pipelines, using GitLab and Nexus Repository for efficient deployments. Required education Bachelor's Degree Preferred education Master's Degree Required technical and professional expertise 12+ years of experience in Data Engineering, ETL, and Data Platform Modernization. Hands-on experience in IBM DataStage and PySpark, with a track record of migrating legacy ETL workloads. Expertise in Apache Iceberg, Cloudera Data Platform, and Big-data processing frameworks. Strong knowledge of Kafka, Airflow, and cloud-native data processing solutions. Experience with Denodo for data virtualization and Talend DQ for data quality. Proficiency in SQL, NoSQL, and Graph DBs (DGraph Enterprise). Strong understanding of data security, encryption, and compliance standards (Thales CipherTrust). Experience with DevOps, CI/CD pipelines, GitLab, and Sonatype Nexus Repository. Excellent problem-solving, analytical, and communication skills. Preferred technical and professional experience Experience with Cloudera migration projects in Banking or financial domains. Experience working with Banking Data model. Knowledge of Cloudera ML, Qlik Sense/Tableau reporting, and integration with data lakes. Hands-on experience with QuerySurge for automated data testing. Understanding of code quality and security best practices using CheckMarx. IBM, Cloudera, or AWS/GCP certifications in Data Engineering, Cloud, or Security. “Meghdoot” Cloud platform knowledge. Architectural designing and recommendations the best possible solutions.
