Data Engineer

The data engineer is responsible for Designing, Architecting and implementing robust, scalable and maintainable data pipelines. candidate will work directly with upstream stockholders (applications owners, data providers) and downstream stakeholders (Data consumers, Data analysts, Data Scientists) to define the data pipeline requirements, implement solutions that serve downstream stakeholders needs through APIs, materialized views.

On day to day, Candidate works in conjunction with the Data Analyst, for the aggregation and preparation of data. interacts with security, continuity and IT architecture to validate the IT assets designs and develops. Furthermore, works with BNP Paribas international team

Responsibilities

Work on the stages from data ingestion to analytics, encompass integration, transformation, warehousing and maintenance

The Data Engineer designs architecture, orchestrate, deploys and monitors reliable data processing systems.

Implement Batch and streaming data pipelines to ingest data into the data warehouse

Perform undercurrent activities (Data architecture, Data management, DataOps, Security)

Perform data transformation and modeling, to convert data from OLTP to OLAP to speed up data querying, and best align with business needs

Serve downstream stakeholders across the organization, whose improved access to standardized data will make them more effective at delivering use cases, building dashboards and guiding decisions

Technical Competencies

Master Data engineering fundamentals concepts (Data warehouse, Data Lake, Data Lakehouse)

Master Golang, Bash, SQL, Python

Master of HTTP and REST API Best practices

Master batch and streaming datapipeline using Kafka

Master code versioning with Git and best practices for continuous integration & delivery (CI/CD)

Master writing clean and tested code following software engineering best practices (Readable, Modular, Reusable, Extensible)

Master data modeling (3NF, Kimball, Vault)

Knowledge ofdata orchestration using Airflow or Dagster

Knowledge to self-host and manage tools like Metabase, DBT,

Knowledge of cloud principals and infrastructure management (IAM, Logging, Terraform, Ansible)

Knowledge of data abstraction layers (Object Storage, Relational, NoSQL, Document, Trino, and Graph databases)

Knowledge with Containerization and workload orchestration with (Docker, Kubernetes, Artifactory)

Background in working in an agile environment (knowledge of the methods and their limits)