Job
Description
Job Description AWS Data engineer Hadoop Migration We are seeking an experienced AWS Principal Data Architect to lead the migration of Hadoop DWH workloads from on-premise to AWS EMR. As an AWS Data Architect, you will be a recognized expert in cloud data engineering, developing solutions designed for effective data processing and warehousing requirements of large enterprises. You will be responsible for designing, implementing, and optimizing the data architecture in AWS, ensuring highly scalable, flexible, secured and resilient cloud architectures solving business problems and helps accelerate the adoption of our clients data initiatives on the cloud. Key Responsibilities: Lead the migration of Hadoop workloads from on-premise to AWS-EMR stack. Design and implement data architectures on AWS, including data pipelines, storage, and security. Collaborate with cross-functional teams to ensure seamless migration and integration. Optimize data architectures for scalability, performance, and cost-effectiveness. Develop and maintain technical documentation and standards. Provide technical leadership and mentorship to junior team members. Work closely with stakeholders to understand business requirements, and ensure data architectures meet business needs. Work alongside customers to build enterprise data platforms using AWS data services like Elastic Map Reduce (EMR), Redshift, Kinesis, Data Exchange, Data Sync, RDS , Data Store, Amazon MSK, DMS, Glue, Appflow, AWA Zero-ETL, Glue Data Catalog, Athena, Lake Formation, S3, RMS, Data Zone, Amazon MWAA, APIs Kong Deep understanding of Hadoop components, conceptual processes and system functioning and relative components in AWS EMR and other AWS services. Good experience on Spark-EMR Experience in Snowflake/Redshift Good idea of AWS system engineering aspects of setting up CI-CD pipelines on AWS using Cloudwatch, Cloudtrail, KMS, IAM IDC, Secret Manager, etc Extract best-practice knowledge, reference architectures, and patterns from these engagements for sharing with the worldwide AWS solution architect community Basic Qualifications: 10+ years of IT experience with 5+ years of experience in Data Engineering and 5+ years of hands-on experience in AWS Data/EMR Services (e.g. S3, Glue, Glue Catalog, Lake Formation) Strong understanding of Hadoop architecture, including HDFS, YARN, MapReduce, Hive, HBase. Experience with data migration tools like Glue, Data Sync. Excellent knowledge of data modeling, data warehousing, ETL processes, and other Data management systems. Strong understanding of security and compliance requirements in cloud. Experience in Agile development methodologies and version control systems. Excellent communication an leadership skills. Ability to work effectively across internal and external organizations and virtual teams. Deep experience on AWS native data services including Glue, Glue Catalog, EMR, Spark-EMR, Data Sync, RDS, Data Exchange, Lake Formation, Athena, AWS Certified Data Analytics – Specialty. AWS Certified Solutions Architect – Professional. Experience on Containerization and serverless computing. Familiarity with DevOps practices and automation tools. Experience in Snowflake/Redshift implementation is additionally Azure Data Engineer job description As an Azure Data Engineer, the candidate is expected to be specializing in designing, implementing, and managing large-scale data solutions on the Microsoft Azure cloud platform. They possess expertise in various aspects of data engineering like data storage, data integrations, analytics etc using Azure data services. Ideal candidate will have a strong background in data engineering, Azure cloud services and data processing technologies. # Requirements: 1. 5+ years of experience in data engineering, preferably on Microsoft Azure. This will increase with senior positions as other DE roles we have. 2. Strong knowledge of Azure cloud services, including Azure Data Factory, Azure Databricks, Azure Synapse Analytics, and Azure Storage. 3. Experience with data processing technologies, such as Apache Spark, Apache Hadoop, and SQL. 4. Strong understanding of data modeling, data warehousing, and data governance. 5. Experience with data security, compliance, and regulatory requirements. 6. Strong programming skills in languages such as Python, Scala, or Java. 7. Experience with agile development methodologies and version control systems such as Git 8. Azure certifications, such as Azure Data Engineer Associate or Azure Solutions Architect expert. 9. Experience with containerization technologies such as Docker. 10. Knowledge of data visualization tools, such as Power BI, Tableau etc 11. Experience in machine learning and AI technologies will be added advantage. Soft Skills: Problem-solving: Solve complex data problems effectively. Communication: Communicate clearly with both technical and non-technical team members. Attention to detail: Pay close attention to detail to ensure accuracy. Adaptability and learning: Stay up to date with new technologies and trends. Teamwork: Collaborate well with others for project success. Leadership and mentorship: Take on leadership roles and mentor junior team members for growth.
