Company: Efforts Solutions IT
Job Location: Remote Initially from Chennai | Abu Dhabi, UAE (Potential Onsite Relocation after 2 months if required)
Experience: 7+ Years
Reporting To: AI & Data Manager
About the Opportunity
Efforts Solutions IT is looking for a highly skilled Senior Data Engineer (Azure Databricks) to lead the architecture, development, and implementation of an enterprise-grade AI-powered Agentic Lakehouse for a major enterprise initiative.
This is a hands-on technical leadership role focused on building a modern data platform using Azure Databricks, Delta Lake, Unity Catalog, PySpark, and Azure Data Factory, integrating enterprise data with intelligent AI-driven automation.
The successful candidate will work closely with the AI & Data Manager, internal engineering teams, and external delivery partners to transform complex enterprise data into secure, governed, and reusable data products.
Work Arrangement & Relocation
- Initial Engagement: Remote for the first 2 months.
- Performance Evaluation: Technical capabilities and delivery performance will be assessed during the initial engagement.
- Potential Relocation: Following successful evaluation, the role may require relocation to Abu Dhabi, UAE, for an onsite or hybrid engagement.
- Contract Extension: Subject to project requirements, performance, and mutual agreement.
Important: Candidates must be willing and able to relocate to Abu Dhabi after the initial remote engagement if required. Please apply only if you are comfortable with this arrangement.
Key Responsibilities
1. Data Lakehouse Architecture & Engineering
- Lead the design and implementation of an enterprise-scale Azure Databricks Lakehouse.
- Establish Medallion Architecture (Bronze, Silver, Gold) using Delta Lake.
- Define data modelling standards, data contracts, and reusable engineering frameworks.
- Design scalable ingestion and transformation pipelines.
2. Data Integration & Pipeline Development
- Develop ingestion pipelines using Azure Data Factory, Databricks Workflows, PySpark, and SQL.
- Integrate data from ERP databases, legacy systems, SharePoint, business applications, and REST APIs.
- Implement automated data cleansing, validation, transformation, and migration.
- Optimise pipelines for performance, scalability, reliability, and cost.
3. Data Governance & Security
- Implement Unity Catalog for data governance, access controls, lineage, and classification.
- Configure governed data access across enterprise data domains.
- Apply data-quality controls, exception handling, quarantine processes, and auditability.
- Support secure integration with Azure services and enterprise environments.
4. AI & Agentic Data Automation
- Design and prototype AI agents for automated data discovery, profiling, and onboarding.
- Implement AI-assisted schema inference, data-quality rule generation, and anomaly detection.
- Support entity resolution, metadata enrichment, and intelligent data cleansing.
- Enable governed data access for AI agents using SQL endpoints, vector indexes, and MCP-style services.
- Integrate LLM capabilities using Azure OpenAI and related Azure AI services.
5. Technical Leadership & Delivery
- Lead technical design decisions and establish data engineering best practices.
- Review code and guide internal engineers and external delivery partners.
- Implement Git-based version control and CI/CD for data pipelines.
- Apply Databricks cost optimisation practices, including cluster policies and compute selection.
- Collaborate with stakeholders to deliver project milestones.
The selected candidate will be expected to:
- Assess existing enterprise data sources, pipelines, and data-quality challenges.
- Define the target Lakehouse architecture and implementation blueprint.
- Establish Databricks Medallion layers, Unity Catalog, and ingestion foundations.
- Build initial ingestion and transformation pipelines for priority business domains.
- Implement data-quality validation and governance controls.
- Prototype AI-assisted data profiling, quality-rule generation, and onboarding workflows.
- Demonstrate working deliverables and document the technical implementation.
Required Skills & Qualifications
- 7+ years of professional experience in Data Engineering, including senior-level technical ownership.
- Proven experience designing and implementing enterprise Lakehouse or modern data platforms.
- Strong hands-on expertise in Azure Databricks, Delta Lake, Unity Catalog, and Medallion Architecture.
- Advanced proficiency in Python, PySpark, and SQL.
- Experience with Databricks Workflows, Delta Live Tables / Lakeflow, and production data pipelines.
- Strong knowledge of Azure Data Factory, ADLS Gen2, Azure Key Vault, and Azure networking.
- Experience with data governance, access controls, lineage, and data-quality frameworks.
- Hands-on experience integrating and cleansing ERP or legacy enterprise data.
- Practical exposure to LLM-powered data workflows, Azure OpenAI, or AI-driven automation.
- Experience implementing CI/CD using Azure DevOps or Git-based pipelines.
- Strong problem-solving, architecture design, communication, and technical leadership skills.
Preferred Qualifications
- Databricks Certified Data Engineer Professional.
- Experience with knowledge graphs, semantic layers, or enterprise metadata management.
- Exposure to Azure AI Search, vector databases, and retrieval-augmented generation (RAG).
- Experience developing AI agents, MCP integrations, or intelligent data services.
- Previous experience in enterprise digital transformation or complex data migration projects.
Technology Stack
Azure Databricks | Delta Lake | Unity Catalog | PySpark | Python | SQL | Azure Data Factory | ADLS Gen2 | Azure OpenAI | Azure AI Search | Azure DevOps | Git | CI/CD | Lakeflow | Databricks Workflows
How to Apply
Interested candidates are requested to submit their updated CV along with the following information:
- Total years of Data Engineering experience.
- Years of hands-on Azure Databricks experience.
- Experience with Delta Lake, Unity Catalog, and PySpark.
- Experience in Lakehouse architecture and AI/LLM-based data workflows.
- Current location and availability to join.
- Expected monthly remuneration for the initial remote engagement and for onsite role.
- Confirmation of willingness to relocate to Abu Dhabi after 2 months, if required.
Please apply only if you have strong hands-on Azure Databricks experience and are open to potential relocation to Abu Dhabi, UAE.
About Efforts Solutions IT
Efforts Solutions IT is an Abu Dhabi-based technology company established in 2009, delivering Enterprise AI, Intelligent Automation, Data Engineering, Digital Transformation, and IT Outsourcing services to enterprise and government organisations across the UAE, Saudi Arabia, and international markets.
Website: www.effortz.com
Pay: From Rs.1,50,000 per month
Work Location: Remote Initially from Chennai | Abu Dhabi, UAE (Potential Onsite Relocation)
Work Location: In person
