Senior Databricks SQL Engineer

Senior SQL Engineer/Developer supporting EIM/OLAP Team

What Will I Be Doing?

The SQL Engineer will design and implement data solutions in Azure Databricks to consolidate, integrate, and ingest data from multiple sources into curated tables for reporting and analytics. This role involves transformation work, environment migration, QA, and performance optimization while adhering to best practices and security standards.

Key Responsibilities:

• Data Integration & Ingestion:

  Build and maintain pipelines to ingest data from diverse sources (transactional systems, APIs, flat files) into Databricks. Consolidate and normalize data for reporting and analytics use cases.

• Data Transformation & Modeling:

  Transform raw, multi-domain datasets into structured tables, resolving issues with unclear or missing keys.Design schemas and implement robust data models for scalability and accuracy.

• Environment Management:

  Migrate code and workflows through Dev, QA, and Prod environments using CI/CD best practices. Ensure proper version control and deployment standards.

• Quality Assurance & Review:

  Conduct thorough QA checks and peer reviews to validate data integrity and compliance.Implement automated testing where possible.

• Performance Optimization:

  Diagnose and improve query performance, optimize Spark jobs, and manage cluster resources efficiently.

Additional Duties:

• Apply Databricks and Azure best practices for development, governance, and security.
• Ensure compliance with PII handling and implement robust security measures.
• Collaborate with business stakeholders to gather requirements and present solutions clearly.
• Document processes and provide knowledge transfer to team members.

Core Skills:

• Advanced SQL for data ingestion, transformation, and consolidation.
• Strong experience with Databricks,PySpark, and Python for distributed data processing.
• Familiarity with Azure ecosystem and CI/CD pipelines.
• Ability to handle complex, multi-domain data sets without clear keys.
• Experience with transactional data and performance tuning in large-scale environments.

Preferred Experience:

• Hospitality, retail, membership, and inventory data domains.
• Working with PII and security compliance in Databricks.
• Communicating technical concepts to business stakeholders and presenting solutions effectively.