Data Unveil - Data Engineer - ETL/Python

Description

Key Responsibilities

• Develop and optimize ETL workflows and data pipelines using Python and PySpark.
• Write and optimize SQL queries for Oracle and MySQL databases.
• Perform distributed data processing and large-scale data transformations with PySpark.
• Implement data ingestion, transformation, validation, and integration procedures.
• Work with AWS services like S3, Lambda, EC2, RDS, Glue, and others.
• Ensure data integrity, security, and performance optimization.
• Collaborate with data teams to meet project requirements.
• Debug and optimize Spark jobs and database queries.
• Use version control (Git) and participate in code reviews.
  

Required Skills

• Strong Python and PySpark programming skills.
• Experience with Linux environment.
• Hands-on experience with AWS services (S3, Lambda, EC2, RDS, Glue).
• Strong SQL skills for Oracle and MySQL.
• Knowledge of ETL processes and data pipeline development.
• Familiarity with Git and CI/CD practices.
• Ability to optimize queries and troubleshoot data issues.
  

Education

• Bachelors degree in Computer Science, IT, or related field.
• Masters degree is a plus