Principal Data Engineer

Roles & Responsibilities:

• Architect and maintain robust, scalable data pipelines using Databricks, Spark, and Delta Lake for both batch and real-time data processing.
• Lead technology evaluation and adoption initiatives to improve productivity, scalability, and data delivery.
• Optimize data processing performance through Spark tuning, job scheduling, and efficient resource utilization.
• Develop innovative solutions to enhance data ingestion, transformation, lineage tracking, and observability.
• Build metadata-driven frameworks to promote pipeline consistency and reuse.
• Promote a culture of engineering excellence, continuous improvement, and experimentation.
• Collaborate with architecture, platform, governance, and analytics teams to support the enterprise data strategy.
• Define and monitor SLOs, KPIs, and data quality metrics for production systems.
• Translate business requirements into scalable, governed data products in partnership with stakeholders.
• Mentor and guide engineers to adopt modern engineering tools and practices.
• Work closely with DevOps, architects, and analysts to ensure alignment of engineering strategies with business objectives.
• Stay current on data technology trends and best practices to continually enhance the data platform architecture.

Must-Have Skills:

• Strong hands-on experience with Databricks, PySpark, SparkSQL, Apache Spark, AWS, Python, and SQL.
• Deep understanding of workflow orchestration, job performance tuning, and big data processing.
• Proficient with AWS services relevant to data engineering.
• Knowledge of enterprise-wide data architecture patterns such as Data Fabric or Data Mesh.
• Demonstrated ability to learn and apply new technologies quickly.
• Strong problem-solving, analytical, and teamwork skills.
• Experience with Scaled Agile Framework (SAFe), Agile delivery, and DevOps practices.

Good-to-Have Skills:

• Industry expertise in biotech or pharmaceutical sectors.
• Experience writing APIs to enable data access for consumers.
• Familiarity with SQL/NoSQL databases and vector databases for LLM use cases.
• Experience with OLAP and OLTP data modeling and performance tuning.
• Exposure to software engineering best practices, including Git, CI/CD pipelines (e.g., Jenkins, Maven), and DevOps automation.

Education & Certifications:

• 12 to 17 years of experience in Computer Science, Information Technology, or related field.
• AWS Certified Data Engineer (preferred)
• Databricks Certification (preferred)
• SAFe Certification (preferred)

Soft Skills:

• Excellent analytical and troubleshooting capabilities.
• Strong written and verbal communication skills.
• Able to work effectively in global, distributed teams.
• Highly self-motivated and proactive.
• Capable of managing multiple priorities simultaneously.
• Strong team player with a focus on collaboration and shared success.
• Quick learner with strong organizational and presentation skills.