Data Engineer

Position Purpose

DPR team member guarantees application production for both technical and functional support for a given Business Line or Business Line application chain, on the IT asset management for the infrastructure (IAAS , PAAS, CAAS and IT cloud ).

He/she ensures that the service is maintained in operational condition. He/she ensures the smooth running of the processes (batch process[1]ing, transfer, etc.) and the proper functioning in the application sense of the term (relevant business data, functional result, correct execution of application workflows, etc.). In a context of accelerating change, he/she manages the various releases within his/her scope. He/she intervenes not only in the event of incidents to restore the service, but also in preventive terms thanks to data analysis and automation, thus improving service quality and the user experience.

He/she acts as an expert with the IT Ops particularly on production and infrastructure requirements. She/he acts as the guarantor of Production throughout the project cycle. He/she is a key player in taking production requirements into account: contribution to the definition of the technical solution, the stability of the technical environments (Production / non-Production), the monitoring and

supervision of the technical solutions. He/she participates in project team meetings and is involved in activities and collective decisions (assessment of complexity, risks, etc.) throughout the project.

Responsibilities

Direct Responsibilities

• Design and develop real-time and batch data pipelines using tools like Apache NiFi, Apache Kafka, and Apache Airflow.
• Implement and maintain ETL/ELT processes, ensuring data quality and integrity across various data sources.
• Integrate and manage data flows across diverse systems using NiFi and Kafka topics.
• Monitor and optimize the performance of data pipelines and troubleshoot issues proactively.
• Work with the ELK stack (Elasticsearch, Logstash, Kibana) for logging, monitoring, and real-time analytics.
• Write efficient and optimized SQL queries for data extraction, transformation, and reporting.
• Collaborate with data analysts and business stakeholders to develop meaningful Power BI dashboards and reports.
• Maintain documentation for data workflows, processes, and architecture.

Required Skills:

• Strong experience with Apache NiFi for data flow management.
• Expertise in Kafka (streaming data, Kafka Connect, schema management).
• Proficiency in Apache Airflow for orchestrating workflows.
• Solid understanding of SQL (writing complex queries, tuning, indexing).
• Experience working with the ELK stack Elasticsearch for search and analytics, Logstash for ingestion, Kibana for visualization.
• Experience with Power BI data modeling, DAX, dashboard development.
• Knowledge of data warehousing concepts, data modeling, and data lake architecture.