Senior Data Engineer

Job Description

About the Role: We are seeking a highly skilled and experienced Senior Data Engineer to join our growing data team. In this role, you will be responsible for designing, developing, and maintaining scalable and efficient data pipelines and solutions on the Azure Databricks platform. You will leverage your expertise in PySpark, Python, and SQL to transform raw data into actionable insights, supporting critical business decisions. Responsibilities: Design, develop, and maintain robust and scalable data pipelines using PySpark and Python within the Azure Databricks environment. Optimize data processing and storage for performance and efficiency. Develop and implement data models and schemas for efficient data storage and retrieval. Write complex SQL queries to extract, transform, and load data from various sources. Implement data quality checks and monitoring to ensure data accuracy and reliability. Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions. Troubleshoot and resolve data-related issues in a timely manner. Implement and maintain CI/CD pipelines for data engineering workflows. Stay up-to-date with the latest data engineering technologies and best practices. Contribute to the design and implementation of data governance and security policies. Work with Azure services like Azure Data Lake Storage, Azure SQL Database, and Azure Data Factory. Performance tuning of Spark jobs. Qualifications: Bachelor's or Master's degree in Computer Science, Engineering, or a related field. 5+ years of experience in data engineering, with a strong focus on PySpark and Python. Extensive experience with Azure Databricks and its associated services. Strong proficiency in SQL, including writing complex queries and optimizing performance. Experience with data warehousing and ETL/ELT processes. Solid understanding of distributed computing concepts and architectures. Experience with version control systems (e.g., Git). Familiarity with CI/CD pipelines and DevOps practices. Excellent problem-solving and analytical skills. Strong communication and collaboration skills. Experience with Azure Data Lake Storage Gen2. Experience with Azure Data Factory. Knowledge of data modeling techniques. Experience with data quality and data governance concepts.