Azure Data Engineer

Dear Candidates,

We’re Hiring! 🚀

Job Title:

Experience:

Location:

Employment Type:

Azure Data Engineer

Key Responsibilities:

1) Design and Development:

• Design, develop, and deploy scalable data pipelines using

  Databricks (PySpark, Spark SQL)

  ,

  Azure Data Factory

  , and other Azure data services.
• Implement ETL/ELT processes to ingest, transform, and load data from various sources into data lakes and data warehouses.
• Optimize and tune data pipelines for performance and scalability.

2) Data Processing:

• Write and optimize complex SQL queries for data extraction, transformation, and analysis.
• Use

  PySpark

  for large-scale data processing and analytics.
• Implement data partitioning, bucketing, and indexing strategies for efficient data retrieval.

3) Data Integration:

• Integrate data from multiple sources, including structured, semi-structured, and unstructured data.
• Work with APIs, streaming data, and batch processing to ensure seamless data integration.

4) Data Governance and Quality:

• Implement data governance practices to ensure data quality, consistency, and security.
• Monitor and troubleshoot data pipelines to ensure data accuracy and availability.

5) Collaboration:

• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
• Work closely with DevOps teams to deploy and monitor data pipelines in production environments.

6) Documentation:

• Document data pipelines, workflows, and processes for knowledge sharing and future reference.
• Maintain up-to-date documentation on data architecture and data models.

Must-Have Skills:

• Databricks:

  Hands-on experience with Databricks for data processing, analytics, and Serverless SQLWH, Unity Catalog, Lakehouse, Medallion Architecture

• Azure Data Services:

  Proficiency in Azure Data Factory, Azure Data Lake Storage, and Azure SQL Database, Key Vault, Azure Pricing Model

• SQL:

  Strong expertise in writing and optimizing complex SQL queries.

• PySpark:

  Experience in using PySpark for data processing and transformation.

• ETL/ELT:

  Strong understanding of ETL/ELT processes and tools.

• Data Modeling:

  Knowledge of data modeling techniques and best practices.

• Data Governance:

  Familiarity with data governance, data quality, and data security practices.

Good-to-Have Skills:

• Experience with

  Azure DevOps

  for CI/CD pipelines.
• Knowledge in

  DLT (Delta Live Table

  ) in databricks

Qualifications:

• Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related field.
• 2-8 years of experience in data engineering, with a focus on Azure and Databricks.
• Relevant certifications such as

  Microsoft Certified: Azure Data Engineer Associate

  or

  Databricks Certified Associate Developer

  are a plus.

Soft Skills:

• Strong problem-solving and analytical skills.
• Excellent communication and collaboration skills.
• Ability to work in a fast-paced, dynamic environment.
• Self-motivated with a strong sense of ownership and accountability

If you are interested in the job opportunity,

📩 Apply here - Azure Data Engineer - Banglore, PAN India – Fill out form