Data Engineer

Role & responsibilities

• Designing and Building Data Pipelines:

   Creating robust, scalable, and efficient ETL (Extract, Transform, Load) or ELT (Extract, Load, Transform) pipelines to move data from various sources into data warehouses, data lakes, or other storage systems. Ingest data which is structured, unstructured.

• Data Storage and Management:

   Selecting and managing appropriate data storage solutions (e.g., relational databases, S3, ADLS, data warehouses like SQL, Databricks.

• Data Architecture:

  Understand target data models, schemas, and database structures that support business requirements and data analysis needs.

• Data Integration:

   Connecting disparate data sources, ensuring data consistency and quality across different systems.

• Performance Optimization:

   Optimizing data processing systems for speed, efficiency, and scalability, often dealing with large source systems datasets.

• Data Governance and Security:

   Implementing measures for data quality, security, privacy, and compliance with regulations.

• Collaboration:

   Working closely with Data Scientists, Data Analysts, Business Intelligence Developers, and other stakeholders to understand their data needs and provide them with clean, reliable data.

• Automation:

   Automating data processes and workflows to reduce manual effort and improve reliability.

Preferred candidate profile

• 4 - 6 years of experience as an Data Engineer .
• ETL/ELT Tools: Experience with data integration tools and platforms like SSIS, Azure Data Factory
• SSIS Package Development
• Control Flow: Designing and managing the workflow of ETL processes, including tasks, containers, and precedence constraints.
• Data Flow: Building pipelines for extracting data from sources, transforming it using various built-in components
• SQL Server Management Studio (SSMS): For database administration, querying, and managing SSIS packages.
• SQL Server Data Tools (SSDT) / Visual Studio: The primary IDE for developing SSIS packages.
• Scripting (C# or VB.NET): For advanced transformations, custom components, or complex logic that cannot be achieved with built-in SSIS components.
• Programming Languages:  Advantage if experience on either of Python / Java Scala basics
• Cloud Platforms: Proficiency with cloud data services from providers like SSIS, Microsoft Azure (Azure Data Lake, Azure Data Factory) etc
• Data Warehousing: Understanding of data warehousing concepts, dimensional modelling, and schema design.
• Version Control: Familiarity with Git and collaborative development workflows.