Job
Description
As a skilled Data Engineer, you will be responsible for designing, developing, and optimizing large-scale ETL pipelines using Python, PySpark, and the AWS Data Engineering stack. Your expertise in data engineering best practices, automation, and modern cloud-based architectures will be crucial for the successful execution of your tasks. Additionally, your passion for utilizing Generative AI tools to enhance productivity and code quality will set you apart in this role. Key Responsibilities: - Develop and optimize ETL pipelines using Python and PySpark - Implement data engineering best practices including data validation, transformation logic, and performance optimization - Work with large-scale data processing and distributed computing environments - Utilize Scala programming for Spark-based use cases - Explore AI-assisted development tools like GitHub Copilot for improved productivity and code quality - Hands-on experience with AWS Glue, Lambda Functions, EventBridge, SQS, SNS, DynamoDB, and Streams for building serverless and event-driven data pipelines - Proficiency in CloudWatch for creating dashboards, setting up alarms, and monitoring pipeline health - Basic understanding of AWS CDK and CloudFormation for infrastructure automation and deployment Qualifications Required: - Strong proficiency in Python and PySpark - In-depth understanding of data engineering best practices - Experience with large-scale data processing and distributed computing environments - Knowledge of Scala programming and AI-assisted development tools is a plus - Hands-on experience with AWS Data Engineering stack components In addition to the above, the company values skills in Python, PySpark, data validation, and transformation logic for this role. As a skilled Data Engineer, you will be responsible for designing, developing, and optimizing large-scale ETL pipelines using Python, PySpark, and the AWS Data Engineering stack. Your expertise in data engineering best practices, automation, and modern cloud-based architectures will be crucial for the successful execution of your tasks. Additionally, your passion for utilizing Generative AI tools to enhance productivity and code quality will set you apart in this role. Key Responsibilities: - Develop and optimize ETL pipelines using Python and PySpark - Implement data engineering best practices including data validation, transformation logic, and performance optimization - Work with large-scale data processing and distributed computing environments - Utilize Scala programming for Spark-based use cases - Explore AI-assisted development tools like GitHub Copilot for improved productivity and code quality - Hands-on experience with AWS Glue, Lambda Functions, EventBridge, SQS, SNS, DynamoDB, and Streams for building serverless and event-driven data pipelines - Proficiency in CloudWatch for creating dashboards, setting up alarms, and monitoring pipeline health - Basic understanding of AWS CDK and CloudFormation for infrastructure automation and deployment Qualifications Required: - Strong proficiency in Python and PySpark - In-depth understanding of data engineering best practices - Experience with large-scale data processing and distributed computing environments - Knowledge of Scala programming and AI-assisted development tools is a plus - Hands-on experience with AWS Data Engineering stack components In addition to the above, the company values skills in Python, PySpark, data validation, and transformation logic for this role.
