Hiring For Senior Azure Data Engineer

Job Summary:

Key Responsibilities:

Data Ingestion & Pipeline Development

• Design and develop scalable, secure, and efficient data pipelines using Azure Data Factory, PySpark, or Databricks notebooks.
• Ingest structured and unstructured data from multiple sources (SQL, APIs, flat files, on-premise systems) into Azure Data Lake (ADLS Gen2).
• Ensure robust partitioning, metadata management, and schema evolution in the ingestion process.

Data Transformation & Curation

• Perform advanced data wrangling and transformation in Azure Databricks (PySpark/SQL).
• Develop and maintain reusable code modules and transformation logic using Delta Lake architecture (Bronze/Silver/Gold layers).
• Optimize pipelines for performance, cost-efficiency, and maintainability.

Testing, Validation & Monitoring

• Implement unit testing, data validation, and pipeline monitoring to ensure data quality and reliability.
• Collaborate with QA and data governance teams to enforce data integrity, lineage, and compliance.

Deployment & Automation

• Manage CI/CD pipelines for data workflows using Azure DevOps or Git-based repos.
• Develop modular pipelines that support version control, rollback, and scheduled orchestration.

Collaboration & Stakeholder Support

• Work closely with data architects, analysts, and business stakeholders to translate data needs into technical requirements.
• Provide documentation and training for data consumers and ensure ongoing support for data products.

Required Qualifications:

• 10+ years of experience in data engineering, big data development, or ETL/ELT pipeline design.
• Strong hands-on experience with:
• Azure Data Lake (ADLS Gen2)
• Azure Databricks (PySpark, Delta Lake)
• Azure Data Factory / Synapse Pipelines
• Strong SQL and Python skills for data processing and transformation.
• Experience with CI/CD practices and tools like Azure DevOps.
• Deep understanding of data architecture patterns, including medallion architecture (Bronze/Silver/Gold).
• Familiarity with data governance, metadata management, and privacy standards (e.g., GDPR, HIPAA).

Preferred Qualifications:

• Azure Data Engineer Associate certification (DP-203)
• Databricks Certified Data Engineer Associate or Professional
• Experience with REST APIs and JSON ingestion
• Familiarity with Power BI or other reporting tools (for data validation and testing)
• Exposure to real-time data ingestion using Event Hubs, Kafka, or Stream Analytics

Soft Skills:

• Excellent communication and stakeholder management skills
• Ability to work independently and mentor junior developers
• Strong problem-solving and analytical thinking