Data Engineer

About The Role

Location: BangaloreOur growth plans:We process close to 4% of the country&aposs freight on our software platform. Our goal is to get to 20% of the country&aposs freight by 2028. This gives us a birds eye view of the market. Were already the largest road freight technology platform in the country and we plan to build on this base to drive growth in software, freight marketplace and supply chain financing to get to a 100M USD revenue by 2028.About the Role:We are seeking a highly skilled Senior Data Engineer with 5-6 years of experience to join our dynamic team. The ideal candidate will have a strong background in data engineering, with expertise in data warehouse architecture, data modeling, ETL processes, and building both batch and streaming pipelines. The candidate should also possess advanced proficiency in Spark, Databricks, Kafka, Python, SQL, and Change Data Capture (CDC) methodologies.Key responsibilities:

• Design, develop, and maintain robust data warehouse solutions to support the organization&aposs analytical and reporting needs.
• Implement efficient data modeling techniques to optimize performance and scalability of data systems.
• Build and manage data lakehouse infrastructure, ensuring reliability, availability, and security of data assets.
• Develop and maintain ETL pipelines to ingest, transform, and load data from various sources into the data warehouse and data lakehouse.
• Utilize Spark and Databricks to process large-scale datasets efficiently and in real-time.
• Implement Kafka for building real-time streaming pipelines and ensure data consistency and reliability.
• Design and develop batch pipelines for scheduled data processing tasks.
• Collaborate with cross-functional teams to gather requirements, understand data needs, and deliver effective data solutions.
• Perform data analysis and troubleshooting to identify and resolve data quality issues and performance bottlenecks.
• Stay updated with the latest technologies and industry trends in data engineering and contribute to continuous improvement initiatives.
  

Minimum Qualifications

• Bachelors degree in Computer Science, Information Technology, or related field.
• 12 years of hands-on professional experience working with data engineering or in a related role.
• Strong proficiency in SQL for data extraction, transformation, and optimization.
• Strong hands-on experience with Python for data processing, automation, and scripting.
• Familiarity with PySpark or other big data frameworks.
• Familiarity of Spark architecture (distributed processing) and distributed systems.
• Understanding of data structures, algorithms, and database concepts.
• Knowledge of coding best practices and code quality standards.
• Strong analytical and problem-solving skills.
• Eagerness to learn new technologies and work in a fast-paced, collaborative environment.
  

Preferred Qualifications

• Solid working knowledge of PySpark for handling large-scale distributed datasets.
• In depth understanding of Spark architecture (executors, partitions, shuffles, caching, performance tuning).
• Exposure to data modeling, ETL pipelines, and data quality best practices.
• Familiarity with cloud data platforms (e.g., Databricks and AWS).
• Strong problem-solving and debugging skills with an eye for performance optimization.
• Ability to work collaboratively in agile teams and communicate technical concepts effectively.
• Educational background: B.Tech/BE in Computer Science, Information Technology, or related field from IITs, NITs, or IIITs.
  

Certifications (preferred): Databricks Certified Data Engineer Associate/Professional, AWS Big Data Specialty, or equivalent.