Senior Data Engineer - Big Data

As a Sr. Data Engineer at Lifesight in Bangalore, you will be responsible for building highly scalable, fault-tolerant distributed data processing systems that handle massive amounts of data ingested daily. You will work on processing petabyte-sized data warehouses and Elasticsearch clusters, optimizing data pipelines for quality and resilience, and refining diverse datasets into simplified models to encourage self-service. Your role will involve owning data mapping, business logic, transformations, and ensuring data quality through low-level systems debugging and performance optimization on large production clusters. Additionally, you will participate in architecture discussions, influence product roadmaps, and take ownership of new projects while maintaining and supporting existing platforms and transitioning to newer technology stacks. To excel in this role, you should have proficiency in Python and PySpark, a deep understanding of Apache Spark including tuning and data frame building, and the ability to create Java/Scala Spark jobs for data transformation and aggregation. Your experience with big data technologies such as HDFS, YARN, Map-Reduce, Hive, Kafka, Spark, Airflow, and Presto, as well as distributed environments using tools like Kafka, Spark, Hive, and Hadoop, will be invaluable. Familiarity with distributed database systems, various file formats like Parquet and Avro, and NoSQL databases is essential, along with experience in cloud platforms like AWS and GCP. Ideally, you should have at least 5 years of professional experience as a data or software engineer. Joining Lifesight means being part of a fast-growing Marketing Measurement Platform with a global impact, where you can influence key decisions on tech stack, product development, and scalable solutions. You will work in small, agile teams within a non-bureaucratic, fast-paced environment that values innovation, collaboration, and personal well-being. Competitive compensation, benefits, and a culture of empowerment that prioritizes work-life balance and team camaraderie await you at Lifesight.,