Data Engineer

Role & responsibilities:-

Job Summary:-

We are seeking a highly skilled and motivated Data Engineer to join our Data, Reporting & Analytics team. The ideal candidate will have a strong background in data engineering, database design, SQL, excellent problem-solving skills, and a passion for working with large datasets to drive business insights and improve decision-making processes. The candidate should have extensive knowledge of data pipeline components in Azure or AWS, and have exposure to data warehousing.

Skills & Qualifications:

• Bachelor's degree in Computer Science, Engineering, or a related field.
• 5+ years of experience in data engineering.
• Proven Experience as a Data Engineering of building Data Pipelines, Architecting Data bases and Data Governance
• Expertise in data pipeline components(services) and data engineering tools on AWS (e.g., S3, Redshift, Kinesis, Lambda, Sagemaker, Kafka, etc.) and Azure (e.g., Azure Data Factory, Azure Databricks, Airflow, etc.).
• Proficiency in SQL and experience with relational databases (e.g., MySQL, PostgreSQL).
• Experience with big data technologies (e.g., Hadoop, Spark) and data warehousing solutions (e.g., Redshift, Azure SQL Data Warehouse).
• Strong programming skills in Python, with a focus on data processing and automation.
• Experience with cloud platforms (AWS or Azure) and data engineering tools (e.g., Airflow, Kafka).
• Excellent problem-solving skills and attention to detail.
• Strong communication and collaboration skills.

Key Responsibilities:

Data Management and Architecture:

• Implement the data architecture for the enterprise and its business units
• Implementation of data management systems including data warehouses, data lakes and data integration processes with other transaction systems
• Collaborate with IT teams to ensure the availability, scalability and performance of data infrastructure & systems
• Implement data lifecycle management processes to optimize data storage, retention and archival
• Design, develop, and maintain scalable data pipelines and ETL processes to support data integration from various sources.
• Implement data models and schemas to support analytics and reporting needs
• Handle large volumes of structured and unstructured data by means of data base queries, APIs, adapters or other data retrieval methods
• Optimize data workflows and improve data processing performance
• Develop and maintain documentation for data pipelines, processes, systems and databases.
• Monitor and troubleshoot data issues to ensure data accuracy and reliability.
• Work with cloud-based hydrid data platforms, including AWS (e.g., S3, Redshift, Kinesis, EBS, Lambda, Sagemaker) and Azure (e.g., Azure Data Factory, Azure Databricks, Azure SQL Data Warehouse) to manage and scale data infrastructure.
• Ensure compliance with data governance and security policies.

Stakeholder Collaboration:

• Collaborate with data scientists, power BI developers, external vendors and other stakeholders to understand data requirements and ensure data quality and integrity.
• Collaborate with IT teams to ensure the availability, scalability and performance of data infrastructure & systems.

Preferred Skills:

• Experience in building end-to-end data ingestion pipelines
• Experience in developing Data Lake(s) or Data Warehouse(s)
• Proficiency is Database schema design
• Hands-on in full-stack development
• Experience in the pharmaceutical or healthcare industry
• Knowledge of machine learning and AI concepts.
• Understanding of data privacy regulations (e.g., GDPR, HIPAA).