ETL Developer - Pyspark + AWS

Roles and Responsibilities:

• Design, develop, and maintain scalable ETL pipelines using PySpark and Python for large-scale data processing.
  
• Work closely with data architects and analysts to understand business requirements and translate them into technical solutions.
  
• Develop and optimize SQL queries for data extraction, transformation, and loading from various data sources.
  
• Integrate data from multiple sources (structured and unstructured) into data lakes or data warehouses on AWS.
  
• Manage data ingestion from real-time and batch sources, including Kafka, S3, and relational databases.
  
• Develop and deploy AWS Lambda functions for event-driven processing and automation tasks.
  
• Ensure data quality, integrity, and consistency throughout the ETL lifecycle.
  
• Monitor ETL job performance, troubleshoot failures, and optimize pipeline execution time.
  
• Implement error handling, logging, and alerting mechanisms to ensure pipeline reliability.
  
• Collaborate with DevOps teams to automate deployments using CI/CD tools.
  
• Maintain documentation for ETL processes, data models, and system architecture.
  
• Ensure adherence to data security and compliance standards across all ETL operations