Lead Azure Big Data Engineer

Role Overview: As a Senior Data Engineer, you will be responsible for designing, building, and optimizing scalable data pipelines for large-scale data processing. Your expertise in Python, Spark, Databricks, and Azure will be crucial in handling complex data engineering tasks. You will work with cloud platforms, specifically Azure, to process and analyze large datasets efficiently. Key Responsibilities: - Design, build, and optimize scalable data pipelines using Python and Apache Spark. - Utilize Azure Databricks and PySpark for big data processing and analysis. - Implement data integration and orchestration using Azure Data Factory. - Utilize Azure Data Lake Storage for big data analytics and transformation. - Write efficient Python code using libraries like Pandas and NumPy for data manipulation. - Work with Azure services such as Azure SQL Database, Azure Synapse Analytics, Azure Cosmos DB, and Azure Blob Storage. - Experience with Infrastructure as Code (IaC) tools like Terraform or Biceps for provisioning Azure infra resources. - Architectural experience is a plus. - Design and maintain complex SQL queries for relational databases. - Ensure high-quality data management, integrity, and security throughout data processing. - Collaborate with cross-functional teams to understand data requirements and build solutions. - Provide mentorship and guidance to junior team members. Key Skills & Qualifications: - Proficiency in Python, including knowledge of Pythonic principles, data structures, and algorithms. - In-depth knowledge of Azure services such as Azure SQL Database, Azure Synapse Analytics, Azure Cosmos DB, Azure Data Factory, Azure Blob Storage, and Azure Data Lake Storage. - Strong experience with Apache Spark using Azure Databricks and PySpark. - Strong SQL skills for querying and managing relational databases. - Excellent communication skills for effective collaboration and technical solution presentation. - Strong analytical and problem-solving skills for resolving complex data challenges. Qualification Required: - B.Tech in any branch or specialization.,