AWS Databricks + Pyspark

You will be responsible for developing and optimizing ETL pipelines using PySpark and Databricks. Your role will involve working with large-scale datasets and building distributed computing solutions. You will design and implement data ingestion, transformation, and processing workflows, as well as write efficient and scalable Python code for data processing. Collaboration with data engineers, data scientists, and business teams to deliver insights is a key aspect of this role. Additionally, you will be expected to optimize performance and cost efficiency for big data solutions and implement best practices for CI/CD, testing, and automation in a cloud environment. Monitoring job performance, troubleshooting failures, and tuning queries will also be part of your responsibilities.,