Senior Data Platform Engineer

In this vital role you will work as a member of a Data Platform Engineering team that uses Cloud and Big Data technologies to craft, develop, implement and maintain solutions to support various functions like Manufacturing, Commercial, Research and Development.

• Collaborate with Lead Architect, Business SMEs, and Data Scientists to design data solutions
• Serve as a Lead Engineer for technical implementation of projects including planning, architecture, design, development, testing, and deployment following agile methodologies
• Design and development of API services for managing Databricks resources, services & features and to support data governance applications to manage security of data assets following the standards
• Design and development of enterprise-level re-usable components, frameworks and services to enable data engineers
• Proactively work on challenging data integration problems by implementing efficient ETL patterns, frameworks for structured and unstructured data
• Automate and optimize data pipeline and framework for easier and efficient development process
• Overall management of the Enterprise Data Fabric/Lake on AWS environment to ensure that the service delivery is efficient and business SLAs around uptime, performance and capacity are met
• Help define guidelines, standards, strategies, security policies and change management policies to support the Enterprise Data Fabric/Lake
• Advice and support project teams (project managers, architects, business analysts, and developers) on cloud platforms (AWS, Databricks preferred), tools, technology, and methodology related to the design, build scalable, efficient and maintain Data Lake and other Big Data solutions
• Experience developing in an Agile development environment and ceremonies
• Familiarity with code versioning using GITLAB, and code deployment tools
• Mentor junior engineers and team members

What we expect of you

• Doctorate degree / Master's degree / Bachelor's degree and 8 to 13 years in Computer Science or Engineering

• Proficiency in Python-based microservices development and deployment.
• Strong development experience with Databricks, Apache Spark, PySpark, and Apache Airflow.
• Proven hands-on experience with cloud platformsAWS (preferred), Azure, or GCP.
• Experience with CI/CD pipelines, containerization (Docker, Kubernetes/EKS), and infrastructure-as-code tools.
• Demonstrated ability to build enterprise-grade, performance-optimized data pipelines in Databricks using Python and PySpark, following best practices and standards.
• Experience working with SQL and solid understanding of relational/dimensional data modelling techniques."
• Strong analytical and problem-solving skills to address complex data engineering challenges.
• Familiarity with software engineering standard methodologies, including version control, automated testing, and continuous integration.
• Hands-on experience with key AWS services: EKS, EC2, S3, EMR, RDS, Redshift/Spectrum, Lambda, and Glue.
• Exposure to Agile tools such as Jira or Jira Align.

• Experience building APIs and services for provisioning and managing AWS Databricks environments.
• Knowledge of Databricks SDK and REST APIs for managing workspaces, clusters, jobs, users, and permissions.
• Familiarity with building AI/ML solutions using Databricks-native features.
• Experience working with SQL/NoSQL databases and vector databases for large language model (LLM) applications.
• Exposure to model fine-tuning and timely engineering practices.
• Experience developing self-service portals using front-end frameworks like React.js.
• Ability to thrive in startup-like environments with minimal direction.
• Good communication skills to effectively present technical information to leadership and respond to collaborator inquiries.
• Certifications (preferred but not required):
• AWS Certified Data Engineer
• Databricks Certification
• SAFe Agile Certification

Soft Skills:

• Strong analytical and problem-solving attitude with the ability to troubleshoot sophisticated data and platform issues.
• Exceptional communication skillsable to translate technical concepts into clear, business-relevant language for diverse audiences.
• Collaborative and globally minded, with experience working effectively in distributed, multi-functional teams.
• Self-motivated and proactive, demonstrating a high degree of ownership and initiative in driving tasks to completion.
• Skilled at managing multiple priorities in fast-paced environments while maintaining attention to detail and quality.
• Team-oriented with a growth mindset, contributing to shared goals and fostering a culture of continuous improvement.
• Effective time and task management, with the ability to estimate, plan, and deliver work across multiple projects while ensuring consistency and quality