Key Responsibilities
- Design, develop, and maintain scalable data processing solutions using Databricks on AWS.
- Build and maintain ETL/ELT pipelines for batch and near real-time data ingestion.
- Develop data transformation and processing workflows using Apache Spark, PySpark, Spark SQL, Python, and Scala.
- Implement Delta Lake solutions, including ACID transactions, schema evolution, and performance optimization.
- Design and implement analytical and dimensional data models, including Star and Snowflake schemas.
- Work with AWS cloud storage services to manage data ingestion, storage, and processing.
- Optimize data pipelines for performance, scalability, reliability, and maintainability.
- Follow coding standards, documentation practices, and data engineering best practices.
- Collaborate with solution architects, BI teams, and data science teams to deliver end-to-end data solutions.
- Support data quality, troubleshooting, and pipeline performance improvements.
Required Skills and Qualifications
- 8+ years of relevant experience in data engineering and development.
- Hands-on development experience with AWS Databricks.
- Strong experience developing and maintaining ETL/ELT pipelines.
- Proficiency in Apache Spark, PySpark, Spark SQL, and Python.
- Working knowledge of Scala and Power BI.
- Hands-on experience with Delta Lake, including ACID transactions, schema evolution, and performance tuning.
- Experience building batch and near real-time data ingestion pipelines.
- Knowledge of analytical data modelling and dimensional modelling, including Star and Snowflake schemas.
- Experience with AWS cloud-based storage systems.
- Strong understanding of data engineering best practices, coding standards, and technical documentation.
- Ability to collaborate effectively with cross-functional technical teams.
Secondary Skills and Preferred Experience
- Exposure to AI/ML workloads on Databricks and feature engineering.
- Familiarity with MLflow or similar model tracking tools.
- Experience with Databricks CI/CD practices and Git-based version control.
- Understanding of data governance, security, and compliance.
- Familiarity with Agile/Scrum delivery methodologies.
- Experience in the insurance domain.
Pay: ₹1,400,000.00 - ₹1,800,000.00 per year
Application Question(s):
- By applying, you agree that your resume and personal details may be stored securely for up to 2 years for recruitment purposes.
Work Location: Hybrid remote in Bangalore City, Bengaluru, Karnataka
