Lead Software Engineer - Python

Are you passionate about building robust Python applications and leveraging big data technologies to solve complex business challenges? Do you thrive in environments where your technical expertise drives meaningful impact? We are seeking an experienced Python Lead Developer to join our dynamic LSH Data Platform Team.

You will play a pivotal role in designing, developing, and maintaining Python-based data processing applications and services that support scalable big data solutions. If you're excited about combining deep Python skills with cutting-edge technologies like Apache Spark, PySpark, GraphDB, SQL, and relational databases, we want to hear from you.

About You experience, education, skills, and accomplishments

• 5+ years of hands-on Python software development experience.

• Advanced proficiency in SQL, database design, and relational database systems.

• Experience developing and deploying solutions on cloud platforms, particularly AWS (S3, ECS, EMR, Glue, RDS).

• Expertise in Apache Spark and PySpark for building scalable data solutions.

• Familiarity with orchestration tools like Apache Airflow.

• Knowledge of graph databases like Neo4j or AWS Neptune.

• Solid grasp of CI/CD principles and experience implementing automated pipelines.

It would be great if you also had

• Practical experience with Databricks and Snowflake.

• Exposure to Cassandra, Hadoop, Apache Hive.

What will you be doing in this role?

• Provide Technical Leadership: Guide the development team in implementing Python-based solutions, ensuring alignment with business requirements, performance standards, and scalability best practices.

• Application Development: Design, build, and maintain scalable Python applications for data processing, transformation, and analytics.

• Database Expertise:Utilize strong proficiency in SQL and relational database concepts to design efficient data models, optimize queries, and manage data quality.

• Best Practices Enforcement: Implement and uphold best practices in Python software development, testing, and CI/CD pipelines.

• Big Data Integration: Leverage Apache Spark, PySpark, and related big data technologies to develop distributed data processing pipelines.

• Continuous Improvement: Stay current with evolving Python, and big data ecosystem technologies, recommending improvements and new approaches to enhance our data platform.

• Mentorship & Collaboration: Mentor junior developers, facilitate knowledge sharing sessions, and foster a culture of continuous learning within the team.

Product you will be developing

You will be building core Python services for the C3 Data Platform, designed to power data delivery and deliver intelligent insights across life sciences and healthcare. These services, built with FastAPI, will expose data from Databricks and graph databases, enabling seamless integration and consumption by downstream applications.

Your work will also include designing and optimizing Spark/Databricks ETL pipelines written in PySpark to ensure high performance and scalability. You will contribute to building a knowledge graph that interconnects data across multiple healthcare and life science domains, forming the backbone for advanced data exploration and relationship discovery.

Additionally, you will have the opportunity to work on the backend of an AI assistant that leverages LLMs and machine learning algorithms to expose platform data to customers, extract valuable insights, and connect information across different domains.

About the Team

Youll be joining the C3 Data Platform Team, a passionate and collaborative group that thrives in a startup-like environment within a larger organization. We move fast, embrace new ideas, and value creativity and initiative. Our mission is to build a powerful, scalable data platform that transforms how life science and healthcare data is processed, connected, and delivered.

The team works with modern technologies including Spark, Databricks, Elasticsearch, Python, and graph databases, combining them with machine learning and AI to solve real-world challenges. We believe in open communication, continuous learning, and giving engineers the autonomy to shape solutions that have a direct impact on our products and customers.

Hours of Work

This is a full-time, hybrid role, requiring presence in the office 2-3 days per week, with flexibility to accommodate collaboration across global teams and time zones.