Job
Description
Overall, 4 to 8 years of experience inIT Industry. Min 4 years ofexperience working on Data Engineering using Azure Databricks, Synapse, ADF/Airflow. At least 3 Project experience in Building andmaintaining ETL / ELT pipelines for large data sets, complex data processing,transformations, business logics, cost monitoring & performanceoptimization, and feature engineering processes. Must Have skills: Extensive experience with Azure Databricks(ADB), Delta Lake, Azure Data Lake Storage (ADLS), Azure Data factory (ADF), AzureSQL Database (SQL DB), SQL, ELT / ETL Pipeline Development in Spark basedenvironment. Extensive Experience with SparkCore, PySpark, Python,SparkSQL, Scala, Azure Blob Storage. Experience in Real-Time Data Processing using Apache Kafka/EventHub/IoT, Structured Streaming and Stream analytics. Experience with Apache Airflow for ELTOrchestration. Experience withinfrastructure management, Infrastructure as code (e.g. Terraform) Experience with CI/CD, Version control tools likeGitHub, Azure DevOps. Experience with Version control tools, buildingCI/CD pipelines. Experience with Azure cloud platform. Good to have: Experience / Knowledge on Containerizations - Docker, Kubernetes Experience working in Agile Methodology Qualifications Qualifications - BE, MS, M.Tech or MCAAdditional Information Certifications: Azure Big Data, Databricks CertifiedAssociate
