Job Title:
Senior Data Engineer Financial Services (Data Lakehouse & Real-Time Systems)
About the Role
We are seeking seasoned Data Engineering and architecture leaders to architect and deliver a next-generation data lake/house platform for a leading financial services client. This engagement will involve thorough data architecture and engineering components, analysis of complex data and its ecosystem and integrating real-time trade, crypto, securities and fixed income data, analytics, and risk systems for various use-cases. The ideal candidate combines deep domain experience in retail brokerage with hands-on expertise in AWS, MS- Azure Databricks, Spark, and modern data lake/house architectures, and can balance delivery under pressure with architectural foresight and team mentorship. This role requires a deep skill-set in real-time architectural systems and pipelines, adept with financial services specifically, investment domain and a hands-on engineer with a knack of innovative insight on integration of various data architecture/s within the cloud ecosystem.
Key Responsibilities
Architect and lead the design, implementation, and optimization of large-scale data lake/house systems on Azure Databricks using Spark, Delta Lake, and Python.
(advanced)
Build and optimize streaming pipelines for real-time ingestion, processing, and analytics of trade and market data across asset classes (FX, equities, futures, options, bonds, and crypto).
(intermediate)
Collaborate closely with the CTO and data architects to evolve the end-to-end application architecture for scalability, resilience, and low-latency performance.
(advanced)
Mentor and guide junior and mid-level engineers, establishing best practices in data modelling, orchestration, and operational monitoring.
Ensure data integrity, lineage, and governance within regulatory and compliance frameworks.
(advanced)
Partner cross-functionally with business and technology teams to align data infrastructure with business KPIs and capital markets product goals.
Drive automation and CI/CD for data pipelines using Terraform, GitHub Actions, and cloud-native DevOps practices.
(advanced)
Deliver under tight timelines, managing sprint-level execution and ensuring production readiness within a short time-line for GO-LIVE.
Required Skills and Qualifications:
15+ years of professional experience, with at least 10 years in financial services and 8+ years in data engineering leadership roles.
Proven track record building real-time, high-throughput data platforms at enterprise scale.
Strong proficiency in Azure Databricks, Apache Spark, and Python (PySpark, Panda).
Solid experience with streaming technologies (Kafka, Kinesis, Event Hubs, or Flink).
Deep understanding of data lake/house design principles (Cloud Engineering and architecture, AWS, Delta Lake, Iceberg, or Hudi) and data modelling for analytics and ML pipelines.
Strong background in retail brokerage and trading systems, including exposure to FX, equities, futures, options, bonds, and crypto.
Experience building or integrating risk management systems, trade surveillance, or crypto exchanges.
Practical knowledge of data governance, lineage, and metadata management tools (e.g., Azure Purview).
Proven ability to design scalable, cost-efficient pipelines on Azure using Data Factory, Synapse, and related services.
Excellent communication, collaboration, and documentation skills, with the ability to influence senior stakeholders.
Preferred Qualifications
Experience working with microservices-based or event-driven architectures supporting capital markets workloads.
Familiarity with machine learning pipelines for predictive risk and trading analytics.
Exposure to Shopify-like multi-tenant or marketplace data architectures.
Cloud certifications: Microsoft Certified: Azure Data Engineer Associate or equivalent.
Previous experience in start-up or fast-paced delivery environments with adaptability and velocity.
Core Attributes
Strong sense of ownership, reliability, and confidentiality in managing critical financial data.
Ability to balance delivery urgency with architectural integrity.
A collaborative mentor who uplifts engineering standards across the team.
Strategic mindset with hands-on execution capability.
Engagement Details
Number of Positions: 3
Location: Mumbai
Engagement Type: Full-Time / Contract-to-Hire
Timeline: Immediate start; production launch targeted for January
Reports To: Engagement Lead in conjunction with Client CTO.
