Infocus Technologies - Azure Databricks Engineer - Python/Spark

Description

Position Summary :

Key Responsibilities

Data Engineering & Pipeline Development :

• Design, build, and maintain ETL/ELT pipelines in Azure Databricks using PySpark and Azure Data Factory, ensuring data reliability, accuracy, and performance.
  

Data Integration

• Integrate and transform structured, semi-structured, and unstructured data from multiple data sources into unified, analytics-ready datasets.
  

Performance Optimization

• Implement performance tuning, caching, and partitioning strategies to improve Databricks and Spark job efficiency.
  

Cloud Architecture & Deployment

• Leverage Azure components (Data Lake, Synapse Analytics, Key Vault, Event Hub, etc.) for data ingestion, transformation, and storage.
  

Automation & CI/CD

• Work with DevOps teams to automate deployment and monitoring processes using Azure DevOps pipelines, Git integration, and version control best practices.
  

Collaboration

• Collaborate with data scientists, BI developers, and business analysts to translate analytical requirements into technical solutions.
  

Data Governance & Security

• Ensure data quality, lineage, and compliance with data governance, privacy, and security standards within Azure.
  

Technical Skills & Expertise

• Strong experience in Azure Databricks, Azure Data Factory (ADF), and PySpark.
• Proficiency in Python for data engineering and automation tasks.
• Familiarity with Azure Data Lake Storage (ADLS), Azure Synapse Analytics, and SQL-based data modeling.
• Understanding of Spark architecture, data partitioning, job scheduling, and performance optimization.
• Experience with data orchestration, workflow automation, and error handling in ADF pipelines.
• Hands-on with CI/CD implementation, Git, and Azure DevOps workflows.
• Working knowledge of ETL best practices, data transformation logic, and data quality frameworks.
• Familiarity with Power BI or other visualization tools (optional but desirable).
  

Preferred Candidate Profile

• Experience : 3 - 5 years in Azure Cloud, Azure Data Factory, and PySpark.
• Educational Qualification : Bachelors degree in Computer Science, Information Technology, or a related discipline.
• Strong analytical, problem-solving, and debugging skills.
• Excellent communication and collaboration abilities within cross-functional teams.
• Ability to deliver quality results under tight deadlines and evolving priorities