Job
Description
Key Responsibilities • Design, develop, and optimize data pipelines using Python and AWS services such as Glue, Lambda, S3, EMR, Redshift, Athena, and Kinesis. • Implement ETL/ELT processes to extract, transform, and load data from various sources into centralized repositories (e.g., data lakes or data warehouses). • Collaborate with cross-functional teams to understand business requirements and translate them into scalable data solutions. • Monitor, troubleshoot, and enhance data workflows for performance and cost optimization. • Ensure data quality and consistency by implementing validation and governance practices. • Work on data security best practices in compliance with organizational policies and regulations. • Automate repetitive data engineering tasks using Python scripts and frameworks. • Leverage CI/CD pipelines for deployment of data workflows on AWS. Required Skills and Qualifications • Professional Experience: 5+ years of experience in data engineering or a related field. • Programming: Strong proficiency in Python, with experience in libraries like pandas, pySpark, or boto3. • AWS Expertise: Hands-on experience with core AWS services for data engineering, such as AWS Glue for ETL/ELT, S3 for storage. • Redshift or Athena for data warehousing and querying. • Lambda for serverless compute. • Kinesis or SNS/SQS for data streaming. • IAM Roles for security. • Databases: Proficiency in SQL and experience with relational (e.g., PostgreSQL, MySQL) and NoSQL (e.g., DynamoDB) databases. • Data Processing: Knowledge of big data frameworks (e.g., Hadoop, Spark) is a plus. • DevOps: Familiarity with CI/CD pipelines and tools like Jenkins, Git, and CodePipeline. • Version Control: Proficient with Git-based workflows. • Problem Solving: Excellent analytical and debugging skills. Optional Skills • Knowledge of data modeling and data warehouse design principles. • Experience with data visualization tools (e.g., Tableau, Power BI). • Familiarity with containerization (e.g., Docker) and orchestration (e.g., Kubernetes).
