Databricks Pyspark SQL

Job Description: You will collaborate with cross-functional teams to understand data requirements and design efficient data processing solutions. Your responsibilities will include developing and maintaining ETL processes using Databricks and PySpark for large-scale data processing. You will optimize and tune existing data pipelines to ensure optimal performance and scalability. Additionally, you will create and implement data quality and validation processes to ensure the accuracy of data. Working with stakeholders to understand business needs and translating them into actionable data solutions will be a crucial part of your role. You will also collaborate with the data science team to support machine learning model deployment and integration into production systems, as well as troubleshoot and resolve data-related issues promptly. Key Responsibilities: - Collaborate with cross-functional teams to understand data requirements and design efficient data processing solutions. - Develop and maintain ETL processes using Databricks and PySpark for large-scale data processing. - Optimize and tune existing data pipelines to ensure optimal performance and scalability. - Create and implement data quality and validation processes to ensure the accuracy of data. - Work with stakeholders to understand business needs and translate them into actionable data solutions. - Collaborate with the data science team to support machine learning model deployment and integration into production systems. - Troubleshoot and resolve data-related issues promptly. Qualifications Required: - Bachelor's degree in Computer Science, Engineering, or a related field. - Proven experience working with Databricks, PySpark, and SQL in a professional setting. - Strong proficiency in designing and optimizing ETL processes for large-scale data sets. - Experience with data modeling, data warehousing, and database design principles. - Familiarity with cloud platforms such as AWS, Azure, or GCP.,