Sr.Consultant- Data Engineer(AWS)

Key Responsibilities:

• Design, develop, and manage scalable, secure, and efficient data pipelines on AWS to process and transform large datasets from multiple sources.
• Implement and optimize AWS data services such as Amazon Redshift, S3, RDS, Glue, Athena, EMR, and Lambda.
• Architect and maintain ETL/ELT processes to ensure efficient data ingestion, transformation, and storage.
• Architect and implement data models on AWS platforms to support efficient data storage and retrieval.
• Collaborate with data architects to design and implement data lakes, data warehouses, and data marts.
• Build and maintain data integration workflows using tools like AWS Glue, Apache Airflow, or Step Functions.
• Utilize AWS DevOps for source control, build automation, release management, and infrastructure as code (IaC) to streamline the development, deployment, and management of data solutions.
• Implement CI/CD pipelines within AWS DevOps to automate the build, test, and deployment of data pipelines and infrastructure changes, ensuring rapid and reliable delivery of data solutions.
• Utilize big data technologies and frameworks such as EMR, Apache Spark, etc. to handle large volumes of data and perform complex analytics tasks.
• Implement real-time data processing solutions using streaming technologies like AWS Kinesis to enable timely insights and actions.
• Implement data governance policies and security controls within AWS environments to protect sensitive data and ensure compliance with regulatory requirements such as GDPR, HIPAA, or PCI DSS.
• Optimize data pipelines and processing workflows for performance, scalability, and cost-effectiveness on AWS, leveraging cloud-native services and optimizations.
• Monitor, troubleshoot, and optimize the performance of data pipelines and cloud-based systems to ensure high availability, low latency, and scalability.
• Work closely with cross-functional teams to understand business requirements and translate them into technical data solutions.
• Ensure data security and compliance with industry best practices and regulatory requirements (e.g., encryption, IAM roles, VPC).
• Conduct regular code reviews, provide feedback, and ensure adherence to best practices and standards.
• Keep up to date with the latest AWS services and cloud technologies to drive innovation within the team.

Job Qualifications:

• Bachelor’s degree in computer science engineering, or a related field.
• 4-6 years of experience as a Cloud Data Engineer or similar role, with a strong focus on AWS cloud services.
• Proficiency in designing and building scalable data architectures using AWS data services such as S3, Redshift, Glue, RDS, Lambda, and EMR.
• Strong experience with ETL/ELT frameworks and tools like Glue, Apache Airflow, or Step Functions.
• Expertise in SQL, Python, or other programming languages used in data engineering.
• Hands-on experience with data lakes, data warehouses, and data pipeline orchestration on AWS.
• Familiarity with CI/CD pipelines and infrastructure-as-code (IaC) tools like CloudFormation or Terraform.
• Understanding of data governance, security, and compliance standards, including encryption and IAM.
• Experience in data modeling, data normalization, and performance optimization for cloud-based data solutions.
• AWS Certifications such as AWS Certified Data Analytics, AWS Certified Solutions Architect, or AWS Certified Developer.
• Experience with Apache Spark or Databricks on AWS.
• Knowledge of machine learning workflows and working with data science teams is good to have.
• Familiarity with DevOps practices and tools such as Docker and Kubernetes.
• Excellent communication and collaboration skills, with the ability to work effectively in a team environment.
• Strong problem-solving and troubleshooting skills, with a proactive approach to identifying and resolving issues.
• Ability to adapt to a fast-paced, agile environment and manage multiple priorities effectively.