Lead Data Architect

As a Lead Data Engineer at Weekday's client, you will play a crucial role in designing, building, and optimizing scalable data pipelines for large-scale data processing. Your expertise in Python, PySpark, SQL, and Azure technologies will be utilized to handle complex data engineering tasks efficiently. Here's what you can expect in this role: **Key Responsibilities:** - Design, build, and optimize scalable data pipelines using Python and Apache Spark. - Utilize Azure Databricks and PySpark for big data processing and analysis. - Implement data integration and orchestration using Azure Data Factory. - Utilize Azure Data Lake Storage for big data analytics and transformation. - Write efficient Python code with libraries like Pandas and NumPy for data manipulation. - Work with Azure data services such as Azure SQL Database, Azure Synapse Analytics, and Azure Blob Storage. - Design and maintain complex SQL queries for analyzing relational databases. - Ensure data management, integrity, and security throughout the data processing and analysis stages. - Collaborate with cross-functional teams to understand data requirements and build solutions that meet business needs. - Provide mentorship and guidance to junior team members. **Key Skills & Qualifications:** - Experience: 7+ years in data engineering with a focus on cloud technologies and big data. - Programming: Proficiency in Python, including knowledge of data structures and algorithms. - Cloud Technologies: In-depth knowledge of Azure services like Azure SQL Database, Azure Synapse Analytics, and Azure Data Factory. - Data Processing: Strong experience with Apache Spark using Azure Databricks and PySpark. - SQL: Strong skills in writing complex and optimized SQL queries for relational databases. - Communication: Excellent communication skills to collaborate effectively across teams and present technical solutions. - Problem-Solving: Strong analytical and problem-solving skills to troubleshoot and resolve complex data challenges. This role offers the opportunity to work on cutting-edge technologies and be a part of a growing team focused on data processing and analytics.,