Data Engineer

Position Purpose
The data engineer is responsible for Designing, Architecting and implementing robust, scalable and maintainable data pipelines. The data engineer will work directly with upstream stockholders (applications owners, data providers) and downstream stakeholders (Data consumers, Data analysts, Data Scientists) to define the data pipeline requirements, implement solutions that serve downstream stakeholders needs through APIs, materialized views.On day to day, the data engineer works in conjunction with the Data Analyst, for the aggregation and preparation of data. The data engineer interacts with security, continuity and IT architecture to validate the IT assets the person designs and develops. Furthermore, the person works with BNP Paribas international teamResponsibilitiesDirect Responsibilities

• Work on the stages from data ingestion to analytics, encompass integration, transformation, warehousing and maintenance
• The Data Engineer designs architecture, orchestrate, deploys and monitors reliable data processing systems.
• Implement Batch and streaming data pipelines to ingest data into the data warehousePerform undercurrent activities (Data architecture, Data management, DataOps, Security)
• Perform data transformation and modeling, to convert data from OLTP to OLAP to speed up data querying, and best align with business needs
• Serve downstream stakeholders across the organization, whose improved access to standardized data will make them more effective at delivering use cases, building
• dashboards and guiding decisionsTechnical & Behavioral Competencies
• Master Data engineering fundamentals concepts (Data warehouse, Data Lake, Data Lakehouse)
• Master Golang, Bash, SQL, Python
• Master of HTTP and REST API Best practices
• Master batch and streaming datapipeline using Kafka
• Master code versioning with Git and best practices for continuous integration & delivery (CI/CD)
• Master writing clean and tested code following software engineering best practices (Readable, Modular, Reusable, Extensible)
• Master data modeling (3NF, Kimball, Vault)
• Knowledge of data orchestration using Airflow or Dagster
• Knowledge to self-host and manage tools like Metabase, DBT
• Knowledge of cloud principals and infrastructure management (IAM, Logging, Terraform, Ansible)
• Knowledge of data abstraction layers (Object Storage, Relational, NoSQL, Document, Trino, and Graph databases)
• Knowledge with Containerization and workload orchestration with (Docker, Kubernetes, Artifactory)
• Background in working in an agile environment (knowledge of the methods and their limits)