Posted:1 day ago|
Platform:
On-site
Full Time
Senior Data Engineer
Job Summary
We are seeking a Senior Data Engineer to design and implement the data foundation for a competitive intelligence MVP. The role involves building automated ingestion pipelines, orchestrating PDF extraction workflows, and creating normalized product datasets in Bronze/Silver/Gold layers using Azure Databricks. You will work in an Azure-first environment (ADF, Databricks, Azure Functions, Azure SQL) and collaborate with AI/ML engineers and BI teams to ensure high-quality, production-ready data pipelines within timeline.
Key Responsibilities
• Data Pipeline Development o Build ingestion pipelines to fetch PDFs from file stores and load into Bronze layer. o Implement orchestration using Azure Functions and integrate with Document Intelligence or Azure OpenAI for text extraction. o Develop transformation and cleaning jobs in Databricks (PySpark) for Silver layer normalization.
• Data Modeling & Quality o Design schemas for Bronze/Silver/Gold layers with Delta Lake. o Implement data quality checks (completeness, deduplication, normalization). o Standardize units, sizes, and packaging for product data.
• Integration & Deployment o Configure Azure Data Factory or Databricks Jobs for scheduling and monitoring. o Deploy Gold layer into Azure SQL for analytics and Power BI consumption. o Ensure secure data access and compliance with organizational standards.
• Collaboration & Documentation o Work closely with AI/ML engineers for feature-ready datasets. o Provide data dictionaries, lineage documentation, and runbooks for operations.
Requirements
Must-Have Skills
• 8+ years in data engineering with Azure cloud experience.
• Strong in Databricks (PySpark), Delta Lake, and Azure Data Factory.
• Experience with Azure Functions, event-driven orchestration, and API integration.
• Proficient in SQL, data modeling, and performance tuning.
• Knowledge of Infrastructure as Code (IaC) for provisioning data services.
• Familiarity with data quality frameworks and monitoring.
Nice-to-Have
• Exposure to Azure Document Intelligence or OCR/NLP pipelines.
• Experience with retail/product data normalization.
• Understanding of MLOps and collaboration with AI/ML teams.
• Familiarity with Power BI data shaping and optimization.
ValueLabs
Upload Resume
Drag or click to upload
Your data is secure with us, protected by advanced encryption.
Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.
We have sent an OTP to your contact. Please enter it below to verify.
Instantly access job listings, apply easily, and track applications.
bengaluru
0.5 - 3.0 Lacs P.A.
20.0 - 30.0 Lacs P.A.
bengaluru
17.0 - 27.5 Lacs P.A.
bengaluru, karnataka, india
21.0 - 35.0 Lacs P.A.
chennai, tamil nadu, india
Salary: Not disclosed
17.0 - 25.0 Lacs P.A.
pune, gurugram, bengaluru
14.0 - 24.0 Lacs P.A.
12.0 - 14.0 Lacs P.A.
hyderabad, bengaluru
18.0 - 30.0 Lacs P.A.
hyderabad
8.4 - 9.0 Lacs P.A.
