Data Processing Principal Engineer DataPelago

10.0 - 15.0 years

0 Lacs

maharashtra

On-site

As the Lead Developer for the next-generation data processing engine at Datapelagos, you will play a pivotal role in overseeing the architecture, design, and implementation of our cutting-edge technology that leverages accelerated computing capabilities. Your primary responsibilities will include collaborating with cross-functional teams to enhance the core architecture of Datapelagos" parallel and distributed execution engine, which is based on accelerated computing. You will lead the execution engine team in designing, implementing, and deploying an enterprise-grade acceleration engine for data processing, ensuring high reliability and performance. In this role, you will be responsible for designing, implementing, testing, and maintaining key components of the execution engine. Your expertise will be crucial in analyzing and identifying areas for innovation and enhancement within the execution engine. Additionally, you will work closely with other teams and team members to drive code reviews, design reviews, performance evaluations, and continuous improvement initiatives. To be successful in this role, you should possess a B.S. in EE/CS or equivalent with over 15 years of experience, or an MS with over 10 years of experience. You should have a proven track record of developing core components for enterprise-grade databases or analytics execution engines catering to large-scale data processing workloads. Experience with platforms like Apache Spark, Gluten, Velox, and DataFusion will be advantageous. Your background should include expertise in developing high-performance parallel implementations of data processing operators and functions such as joins, aggregations, and sorts. You should also have experience leading teams in the design, development, and deployment of high-performance data processing engines for large-scale production environments. Proficiency in programming languages such as C, C++, and Rust is essential for this role, along with a strong development background on Linux platforms. Your dedication to innovation, collaboration, and continuous improvement will be key to driving the success of our next-generation data processing engine at Datapelagos.,

Posted 1 week ago

Apply

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.