Job
Description
Role Overview: You will be responsible for designing and implementing data solutions using technologies such as PySpark, Spark SQL, and Power BI. As a Lead Python Data Engineer, you will play a key role in building and optimizing scalable data pipelines while ensuring data governance, security, and compliance in cloud environments. Key Responsibilities: - Utilize PySpark to build and optimize scalable data pipelines, including Spark SQL, DataFrames, and Datasets. - Work with Azure Databricks (Unity Catalog) to manage Delta tables, workflows, and script in Python. - Design and model data for Data Warehousing and Visualization projects through Semantic Layers. - Write complex SQL queries and build views involving joins across multiple tables. - Proficient in tools like DevOps, version control, and CI/CD pipelines for project management. Qualifications Required: - 7+ years of hands-on experience in software development, with a strong emphasis on Python programming. - Deep expertise in PySpark for building and optimizing scalable data pipelines. - Solid experience with Azure Databricks (Unity Catalog) and working with Delta tables. - Understanding of data governance, data management, security, and compliance in cloud environments. - Familiarity with Azure cloud services, especially Azure Data Lake and Azure Data Factory. - Experience with Power BI, including DAX and Power Query. - Strong collaboration mindset with excellent problem-solving skills and communication abilities. - Proactive and responsive during business hours for any ad-hoc calls or meetings. - Ability to prioritize effectively, think creatively, and adapt quickly to new technologies and feedback. Role Overview: You will be responsible for designing and implementing data solutions using technologies such as PySpark, Spark SQL, and Power BI. As a Lead Python Data Engineer, you will play a key role in building and optimizing scalable data pipelines while ensuring data governance, security, and compliance in cloud environments. Key Responsibilities: - Utilize PySpark to build and optimize scalable data pipelines, including Spark SQL, DataFrames, and Datasets. - Work with Azure Databricks (Unity Catalog) to manage Delta tables, workflows, and script in Python. - Design and model data for Data Warehousing and Visualization projects through Semantic Layers. - Write complex SQL queries and build views involving joins across multiple tables. - Proficient in tools like DevOps, version control, and CI/CD pipelines for project management. Qualifications Required: - 7+ years of hands-on experience in software development, with a strong emphasis on Python programming. - Deep expertise in PySpark for building and optimizing scalable data pipelines. - Solid experience with Azure Databricks (Unity Catalog) and working with Delta tables. - Understanding of data governance, data management, security, and compliance in cloud environments. - Familiarity with Azure cloud services, especially Azure Data Lake and Azure Data Factory. - Experience with Power BI, including DAX and Power Query. - Strong collaboration mindset with excellent problem-solving skills and communication abilities. - Proactive and responsive during business hours for any ad-hoc calls or meetings. - Ability to prioritize effectively, think creatively, and adapt quickly to new technologies and feedback.
