Azure DataBricks Data Engineer ( Asst Manager/ Manager )

Responsibilities

• An Azure Data Engineer is responsible for designing, building, and maintaining the data infrastructure for an organization using Azure cloud services.
• This includes creating data pipelines, integrating data from various sources, and implementing data security and privacy measures.
• The Azure Data Engineer will also be responsible for monitoring and troubleshooting data flows and optimizing data storage and processing for performance and cost efficiency.

Mandatory Skills

• Bachelors or higher degree in Computer Science or a related discipline; or equivalent (minimum 7 years work experience).
• At least 4+ years of consulting or client service delivery experience on Azure
• At least 4+ years of experience in developing data ingestion, data processing and analytical pipelines for big data, relational databases such as SQL server and data warehouse solutions such as Azure Synapse
• Extensive experience providing practical direction with using Azure Native services.
• Extensive hands-on experience implementing data ingestion, ETL and data processing using Azure services: ADLS, Azure Data Factory, Azure Functions, Azure Logic App Synapse/DW, Azure SQL DB, Event Hub, IOT Hub, Azure Stream Analytics, Azure Analysis Service, Databricks, Cosmo DB etc.
• Minimum of 4+ years of hands-on experience in Azure and Big Data technologies such as Java, Python, SQL, ADLS/Blob, PySpark and SparkSQL , Databricks, HD Insight and live streaming technologies such as EventHub, Azure Stream Analytics etc.
• Well versed in DevSecOps and CI/CD deployments
• Cloud migration methodologies and processes including tools like Azure Data Factory, Data Migration Service, etc.
• Minimum of 4+ years of RDBMS experience
• Experience in using Big Data File Formats and compression techniques.
• Experience working with Developer tools such as Azure DevOps, Visual Studio Team Server, Git, Jenkins, etc.
• Experience with private and public cloud architectures, pros/cons, and migration considerations.