Data Engineer – Cloud Integration & Automation

Company Overview

Boston Insights is an innovative startup creating competitive advantage for pharmaceutical companies by unlocking their clinical supply chain data and enabling end-to-end visibility. We augment risk resiliency and agility to ensure uninterrupted supply of investigational drugs to patients on-time. Our mission is to transform how pharmaceutical companies manage their clinical supply chains through cutting-edge data solutions.

Position Overview

Key Responsibilities

Data Integration & Pipeline Development

• Design, build, and maintain scalable data pipelines using Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake
• Lead the integration strategy for ingesting and harmonizing data from external sources, including AWS-based data lakes/warehouses (such as S3, Redshift) and SAP systems.
• Automate ETL processes for data extraction, transformation, and loading across hybrid and multi-cloud environments.
• Build and maintain real-time and batch data integration workflows between Azure, AWS, and on-premises sources.

Data Architecture & Infrastructure

• Design and implement data lake and data warehouse solutions on Azure platform
• Establish data governance frameworks and ensure data quality across all pipelines
• Implement security best practices for handling sensitive pharmaceutical data
• Create and maintain data documentation and lineage tracking

Data Governance & Quality

• Define and enforce data governance frameworks: data cataloging, lineage, quality, privacy, and compliance
• Implement robust data validation, cleansing, and monitoring systems to ensure accuracy and reliability
• Support security standards through effective data management practices.

Automation & Tooling

• Develop data automation tools and reusable components using Python, PySpark (and other relevant frameworks/languages)
• Enable end-to-end process automation for data ingestion, processing, and reporting.
• Implement CI/CD processes for data solutions, including testing, monitoring, and alerting.

Analytics & Reporting Support

• Collaborate with data scientists and analysts to support advanced analytics
• Build data models that enable risk assessment and supply chain optimization
• Develop APIs and data services to support front-end applications
• Create monitoring and alerting systems for data pipeline health

Collaboration & Support

• Partner with supply chain, analytics, and business stakeholders to understand business requirements and translate them into scalable technical solutions.
• Collaborate with SAP functional and technical teams to optimize data extraction and synchronization.

Required Technical Qualifications

• 5+ years of professional data engineering experience.

• Data Integration:

  Proven track record integrating data from AWS services (S3, Redshift, Glue, etc.) into Azure or other cloud environments

• Azure Data Services

  : Expert-level knowledge of Azure Data Factory, Azure Synapse Analytics, Data Bricks, Azure Data Lake Storage, and Azure SQL Database, Apache Spark

• Database Technologies

  : Strong knowledge of both relational (SQL Server, PostgreSQL) and NoSQL (Cosmos DB) databases

• Programming Languages

  : Proficiency in Python, SQL, PySpark and PowerShell for data automation and wrangling

• Version Control

  : Proficiency with Git and Azure DevOps
• Hands-on experience with SAP data models and integrating SAP data with Azure data lake

Preferred Technical Skills

• Experience with API-based data integration for cloud and enterprise applications.
• Experience with Infrastructure as Code (ARM templates, Terraform)
• Familiarity with data quality tools, metadata management, and automated data lineage tracking.
• Knowledge of containerization (Docker, Kubernetes) for data automation workflows.
• Knowledge of machine learning pipelines and MLOps practices
• Experience with data visualization tool, Power BI

Professional Skills

• Strong problem-solving and analytical thinking abilities
• Excellent communication skills with ability to explain technical concepts to non-technical stakeholders
• Experience with Agile development methodologies
• Attention to detail and commitment to data quality

The opportunity we offer

• Competitive salary commensurate with experience.
• Professional development training and certifications.

Work Environment

• Remote work arrangement with flexible hours
• State-of-the-art technology and tools
• Collaborative, innovation-driven culture
• Access to cutting-edge pharmaceutical industry data and challenges
• Opportunity to shape an innovative pharma analytics platform.

Application Instructions

Please submit your resume and a cover letter highlighting:

• Experience in Azure, AWS data integration, and SAP data extraction.
• Examples of data automation tools or frameworks you have developed.

Join us to unlock new possibilities in pharmaceutical supply chain data through advanced engineering and multi-cloud innovation!

Boston Insights is transforming pharmaceutical supply chains through innovative data solutions. Join us in ensuring that life-saving investigational drugs reach patients on-time, every time.