Primary skills : PySpark , AWS Data Lake, Lake Formation Various AWS Data related services GLUE, Athena, Lambda, Kinesis 1. Professional experience as a data engineer: Min 10+ years 2. Experience designing and deploying end to end data lake solution in AWS. 3. Experience with migrating data from db2 to cloud 4. Experience with AWS ETL tools 5. Experience with AWS Databases (Redshift, DynamoDB) 6. Experience with Python, Kafka, Spark, Advance SQL 7. Technical mentoring experience & communication skills 8. Experience working with Atlassian suite (Confluence and Jira) Detailed Skills ----------------- # Must be able to design, develop, deploy and maintain data lake solution in AWS from scratch. Migration of db2 to AWS experience is necessary. # Strong hands-on experience with AWS ETL/File Movement tools for example GLUE, Athena, Lambda, Kinesis. # Proficient in Python, Spark, Kafka and advanced SQL # Proficient with at least two to three AWS database technologies (Redshift, Aurora, RDS, S3 & other AWS Data Service) covering security, policies, and access management. # Experience with one or more relational databases on AWS RDS such as SQL Server, NoSQL database such as DynamoDB and cloud data warehouse databases such as AWS Redshift # Good understanding of Agile Development, Cloud formation Template, AWS Code Built, AWS Code Pipeline # Understanding of Virtual Private Cloud environment including VPC, IAM, EC2 instances, Security groups, Network Access control, Elastic IPs, public & private subnets etc. # Familiarity in building scalable production systems with high availability using Elastic load balancers & Auto Scaling on different regions and availability zones. # Familiarity working with DevOps tools, deployment, and orchestration technologies
