Pyspark Data Engineer

You will be responsible for designing, developing, and maintaining scalable and efficient data processing pipelines using PySpark and Python. Your key responsibilities will include: - Building and implementing ETL (Extract, Transform, Load) processes to ingest data from various sources and load it into target destinations. - Optimizing PySpark applications for performance and troubleshooting existing code. - Ensuring data integrity and quality throughout the data lifecycle. - Collaborating with cross-functional teams, including data engineers and data scientists, to understand and fulfill data needs. - Providing technical leadership, conducting code reviews, and mentoring junior team members. - Translating business requirements into technical solutions and contributing to architectural discussions. - Staying current with the latest industry trends in big data and distributed computing. You must possess the following mandatory skills and experience: - Advanced proficiency in PySpark and Python with extensive experience in building data processing applications. - In-depth understanding of distributed computing principles, including performance tuning. - Experience with big data ecosystem technologies such as Hadoop, Hive, Sqoop, and Spark. - Hands-on experience with cloud platforms like AWS (e.g., Glue, Lambda, Kinesis). - Strong knowledge of SQL, experience with relational databases, and data warehousing. - Experience with software development best practices, including version control (Git), unit testing, and code reviews. The following desired or "nice-to-have" skills would be advantageous: - Familiarity with orchestration tools like Apache Airflow. - Experience with other data processing tools like Kafka or Pandas/Numpy. - Knowledge of API development and creating RESTful services. - Experience with data file formats like Parquet, ORC, and Avro. - Experience with Agile development methodologies.,