Azure Data Engineer

Jo b Description:

• Should be an experienced professional with Data Engineering background.
• Should be able to work without much guidance.
• Design, develop, and maintain scalable ETL pipelines using Azure Services to process, transform, and load large datasets into AWS Datalake or other data stores.
• Collaborate with cross-functional teams, including data architects, analysts, and business stakeholders, to gather data requirements and deliver efficient data solutions.
• Design, implement, and maintain data pipelines for data ingestion, processing, and transformation in Azure.
• Work together with data scientists and analysts to understand the needs for data and create effective data workflows.
• Create and maintain data storage solutions including Azure SQL Database, Azure Data Lake, and Azure Blob Storage.
• Utilizing Azure Data Factory or comparable technologies, create and maintain ETL (Extract, Transform, Load) operations.
• Implementing data validation and cleansing procedures will ensure the quality, integrity, and dependability of the data.
• Improve the scalability, efficiency, and cost-effectiveness of data pipelines.
• Monitoring and resolving data pipeline problems will guarantee consistency and availability of the data. Key Skill Sets Required
• Experience in designing and hands-on development in cloud-based analytics solutions.
• Expert level understanding on Azure Data Factory, Azure Synapse, Azure SQL, Azure Data Lake, and Azure App Service is required.
• Designing and building of data pipelines using API ingestion and Streaming ingestion methods.
• Knowledge of Dev-Ops processes (including CI/CD) and Infrastructure as code is desirable.
• Strong experience in common data warehouse modelling principles including Kimball, Inmon.
• Knowledge in Azure Databricks, Azure IoT, Azure HDInsight + Spark, Azure Stream Analytics, Power BI is desirable
• Working knowledge of Python is desirable • Experience developing security models.