Senior Software Engineer, Data Engineering

Role Overview: As a Data Engineer at ValGenesis, you will play a crucial role in designing, developing, and maintaining data ingestion, transformation, and orchestration pipelines. You will be responsible for building and optimizing data Lakehouse architectures using frameworks like Azure Synapse and Delta Lake. Collaborating with Data Scientists, you will prepare and serve ML-ready datasets for model training and inference. Ensuring scalability, performance, and observability of data workflows through effective monitoring and automation will be key to your role. Key Responsibilities: - Design, develop, and maintain data ingestion, transformation, and orchestration pipelines (batch and real-time) - Build and optimize data Lakehouse architectures using Azure Synapse, Delta Lake, or similar frameworks - Integrate and manage structured and unstructured data sources (SQL/NoSQL, files, documents, IoT streams) - Develop and operationalize ETL/ELT pipelines using Azure Data Factory, Databricks, or Apache Spark - Collaborate with Data Scientists to prepare and serve ML-ready datasets for model training and inference - Implement data quality, lineage, and governance frameworks across pipelines and storage layers - Work with BI tools (Power BI, Superset, Tableau) to enable self-service analytics for business teams - Deploy and maintain data APIs and ML models in production using Azure ML, Kubernetes, and CI/CD pipelines - Ensure scalability, performance, and observability of data workflows through effective monitoring and automation - Collaborate cross-functionally with engineering, product, and business teams to translate insights into action Qualifications Required: - Experience: 4 to 8 years in Data Engineering or equivalent roles - Programming: Strong in Python, SQL, and at least one compiled language (C#, Java, or Scala) - Databases: Experience with relational (SQL Server, PostgreSQL, MySQL) and NoSQL (MongoDB, Cosmos DB) systems - Data Platforms: Hands-on experience with Azure Data Lake, Databricks - ETL/ELT Tools: Azure Data Factory, Apache Airflow, or dbt - Messaging & Streaming: Kafka, Event Hubs, or Service Bus for real-time data processing - AI/ML Exposure: Familiarity with ML frameworks (TensorFlow, PyTorch) and MLOps concepts - Visualization & Analytics: Power BI, Apache Superset, or Tableau - Cloud & DevOps: Azure, Docker, Kubernetes, GitHub Actions/Azure DevOps - Best Practices: Solid understanding of data modeling, version control, and CI/CD for data systems About ValGenesis: ValGenesis is a leading digital validation platform provider for life sciences companies. Their suite of products is used by 30 of the top 50 global pharmaceutical and biotech companies to achieve digital transformation, total compliance, and manufacturing excellence/intelligence across their product lifecycle. ValGenesis continues to push the boundaries of innovation, enhancing and expanding their portfolio beyond validation with an end-to-end digital transformation platform. Note: No additional details of the company are present in the job description.,