Data Engineer (Python + Aws)

As an experienced data engineer, you will be responsible for handling large-scale data processing jobs, optimizing PySpark jobs, and designing efficient data pipelines. Your strong expertise in big data processing and ETL engineering, particularly in PySpark and Databricks, will be crucial for this role. Additionally, your familiarity with cloud platforms, especially AWS, will be highly beneficial. Knowledge of Snowflake warehousing technologies is considered a plus. Your key responsibilities will include: - Building and maintaining batch data pipelines across the advanced analytics platform. - Designing, developing, and orchestrating highly robust and scalable ETL/ELT pipelines. - Developing a highly optimal codebase and performing Spark optimizations for Big Data use cases. - Designing, developing, and deploying optimal monitoring and testing strategies for the data products. - Collaborating with stakeholders and advanced analytics business partners to understand business needs and translate requirements into scalable data engineering solutions. - Collaborating with data scientists to prepare data for model development and production. - Collaborating with data visualization and reporting application developers to ensure the sustainability of production applications and reports. Qualifications required for this role: - Computer science education from a reputable academic institution. Mandatory Skills: - Expertise in SQL - PySpark - Databricks - AWS - Proficiency in Python with OOPS concepts - Working knowledge of CSV, JSON, PARQUET file formats, and API access in Python Secondary Skills (Good to have): - Snowflake - DBT - Linux CLI - Python Web scraping - Datadog - Github You will play a critical role in optimizing data processing jobs, designing efficient data pipelines, and collaborating with various stakeholders to deliver scalable data engineering solutions. Your expertise in PySpark, Databricks, AWS, and other relevant skills will be essential for success in this role.,