Apache Spark Senior Technical Lead

Company Description

Job Title: Technical Lead/Senior Technical Lead

Location: Chennai

Role Overview:

We are looking for a skilled Data Engineer to build and maintain scalable data pipelines, manage large datasets, and support analytics initiatives. This role involves working with Big Data technologies like Spark, Scala, SQL/PLSQL, and cloud-ready environments.

Experience:

• 8+ years of relevant experience in data engineering / big data development
• Proven track record of handling large-scale data solutions

Key Responsibilities:

• Maintain, develop, and design data ingestion and transformation pipelines using Spark and Scala
• Process and optimize SQL and PL/SQL queries for large datasets
• Integrate and process data from multiple sources for analytics and reporting
• Collaborate with BI and Analytics teams to support reporting and dashboard needs
• Ensure data quality, performance, and security
• Provide innovative solutions to complex business/technical problems
• Identify and resolve technical, integration and development issues
• Implement best practices, standards and processes to ensure quality of the final product
• Responsible for development of features independently
• Continuous build, automated testing, and release management
• Participate in status meetings and provide regular updates

Required Skills:

Technical:

• Expertise in Spark and Scala for large-scale data processing
• Proficiency in Python and SQL, including hands-on experience with PL/SQL
• Experience in data integration/ETL workflows;
• Strong knowledge of relational databases such as Oracle and PostgreSQL

Professional Competencies:

• Excellent verbal and written communication skills
• Result-oriented with strong analytical and problem-solving skills
• Ability to work in a fast-paced, agile, and collaborative environment

Desired Skills:

• Exposure to cloud data services (AWS preferred)
• Proficiency in Linux/Unix shell scripting for automation and data operation
• Experience working in Agile environments – Scrum and/or SAFe frameworks
• Exposure to DevOps culture and practices, including CI/CD and automation
• Understanding of software engineering lifecycle – Development, Testing, and Production Support
• Experience in production issue handling and transformation support management

Good to Have Skills:

• Exposure to Dataiku preferred
• Awareness of cloud and big data architecture principles

Total Experience Expected: 08-10 years

Qualifications

Additional Information

At our organization, we are committed to fighting against all forms of discrimination. We foster a work environment that is inclusive and respectful of all differences.

All of our positions are open to people with disabilities.