DE&A - Senior Data Engineer (CDAI)

Bengaluru

Posted:11 hours ago

Job Description

Core Skills & Experience

  • Strong hands-on experience designing, building, and optimizing enterprise-scale data pipelines, data architecture, and data products for performance, scalability, and efficient AI agent consumption.

  • Deep expertise in Starburst/Trino and distributed SQL, including query optimization, materialized views (MVs), data modeling, storage design, and tuning data products for low-latency, high-concurrency AI agent access.

  • Strong experience optimizing data storage, partitioning, indexing, file formats, and materialization strategies to improve query performance, data freshness, and cost efficiency.

  • Hands-on development experience with Apache Spark using Scala and/or Python/PySpark, including migrating complex, SQL-intensive transformations to distributed processing frameworks to improve data curation speed and efficiency.

  • Strong ability to evaluate and redesign data processing workflows, selecting the appropriate technologies and execution patterns across SQL, Spark, and other processing frameworks to optimize performance.

  • Experience with RDBMS technologies and modern lakehouse architecture, preferably Apache Iceberg, including table design, partitioning, compaction, and data lifecycle optimization.

  • Understanding of integration patterns across NiFi, Kafka, Spark, RDBMS, and lakehouse platforms for efficient batch and real-time data processing.

  • Strong software engineering practices, including CI/CD, automated testing, source control, infrastructure automation, observability, and production support.

  • Experience with data governance, metadata management, lineage, data quality, and secure data access in enterprise environments.

Skills Required

Data Architecture SQL Query Optimization Data Modeling Apache Spark Python Pyspark Data Processing Spark Apache Kafka Software Engineering

Learn the skills this job asks for

See your skill gap for Data Engineer →

Free courses, interview questions and practice for each skill.

Mock Interview

Practice Video Interview with JobPe AI

Start Python Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

coding practice

Enhance Your Python Skills

Practice Python coding challenges to boost your skills

Start Practicing Python Now

RecommendedJobs for You