Azure Data Engineer

Role Overview: As a Senior Azure Data Engineer, you will utilize Azure technologies to facilitate data transformation, analytics, and machine learning. Your primary responsibility will involve designing scalable Databricks data pipelines using PySpark to convert raw data into actionable insights. Additionally, you will be tasked with constructing, deploying, and managing machine learning models using MLlib or TensorFlow, all while optimizing cloud data integration from various Azure sources such as Blob Storage, Data Lake, and SQL/NoSQL databases. Your role will also encompass executing large-scale data processing using Spark Pools and fine-tuning configurations for efficiency. Key Responsibilities: - Design and develop scalable data pipelines using Azure Databricks and PySpark. - Transform raw data into actionable insights through advanced data engineering techniques. - Build, deploy, and maintain machine learning models using MLlib, TensorFlow, and MLflow. - Optimize data integration workflows from Azure Blob Storage, Data Lake, and SQL/NoSQL sources. - Execute large-scale data processing using Spark Pools, fine-tuning configurations for performance and cost-efficiency. - Collaborate with data scientists, analysts, and business stakeholders to deliver robust data solutions. - Maintain and enhance Databricks notebooks and Delta Lake architectures. Qualifications Required: - Bachelor's or Master's degree in Computer Science, Data Science, or a related field. - 7+ years of experience in data engineering, with at least 3+ years specializing in Azure Databricks, PySpark, and Spark Pools. - Strong proficiency in Python, PySpark, Pandas, NumPy, SciPy, Spark SQL, DataFrames, RDDs, Delta Lake, Databricks Notebooks, and MLflow. - Hands-on experience with Azure Data Lake, Blob Storage, Synapse Analytics. - Excellent problem-solving and communication skills. - Ability to work independently and in a collaborative team environment.,