Data Engineer

**Role Overview:** As a Data Engineer, your primary responsibility will involve designing, developing, and maintaining ETL processes to facilitate the smooth integration of raw data from diverse sources into data lakes or warehouses. You will be leveraging tools such as Python, PySpark, SQL, and various AWS services like Lambda, Glue, Redshift, and S3 to efficiently process, analyze, and store large-scale datasets. Additionally, you will be tasked with developing and optimizing data pipelines using AWS Glue, PySpark, and Python, while also utilizing your expertise in Apache Spark/Databricks for advanced ETL workflows. Writing and managing SQL queries for data retrieval, transformation, and storage in relational databases such as Redshift or PostgreSQL will be a crucial part of your responsibilities. Collaboration with cross-functional teams, troubleshooting performance issues, data quality problems, and errors in data pipelines, as well as documenting processes, code, and best practices for future reference and team training, will also be essential in this role. **Key Responsibilities:** - Design, develop, and maintain ETL processes for data integration. - Utilize Python, PySpark, SQL, and AWS services for dataset processing. - Develop and optimize data pipelines using AWS Glue, PySpark, and Python. - Write and manage SQL queries for data retrieval, transformation, and storage. - Collaborate with cross-functional teams for scalable solutions. - Troubleshoot and resolve performance issues, data quality problems, and pipeline errors. - Document processes, code, and best practices for team reference and training. **Qualifications Required:** - Strong understanding of data governance, security, and compliance principles preferred. - Ability to work independently and as part of a team in a fast-paced environment. - Excellent problem-solving skills with a focus on identifying inefficiencies and proposing solutions. - Experience with version control systems (e.g., Git) and scripting languages for automation tasks.,