Lead Software Engineer - Cloudera Data Platform

DATA PLATFORM TECHNOLOGY / JOB DESCRIPTION

Senior Cloudera Data Platform, NiFi & Airflow Administrator

Senior / Expert Level | Enterprise Data Platform Operations

Short Overview
The Senior Cloudera, NiFi & Airflow Administrator is responsible for the administration, design, operation, reliability, security, and continuous improvement of enterprise data platform services. The role provides end-to-end administration of Cloudera Data Platform environments, Apache NiFi data-flow services, and Apache Airflow orchestration services across production and non-production environments. The administrator is expert for distributed data processing, data engines, data ingestion, workflow orchestration, platform automation, observability, troubleshooting, capacity planning, lifecycle management, and operational governance in a mission-critical enterprise environment. The role also requires strong Linux administration skills, including system configuration, troubleshooting, performance tuning, networking, storage management, and security.

Overall Experience: 5+ years, senior / expert level

Key Responsibilities

Operate and maintain enterprise data platforms

Ensure secure, reliable, and stable operation, availability, performance, and resilience of Cloudera, Apache NiFi, and Apache Airflow services across production and non-production environments, including platform security, access control, authentication, authorization, encryption, and compliance with enterprise security standards.

Design and evolve platform architecture

Define and review high-availability concepts, disaster recovery arrangements, environment topology, service placement, integration patterns, configuration standards, naming conventions, and operational guidelines.

Provide senior-level troubleshooting and incident support

Analyze and resolve complex issues across distributed services, workloads, data flows, workflows, metadata, security, networking, operating systems, storage, databases, JVMs, and dependent infrastructure.

Support application, analytics, data engineering and data science teams

Consult teams on solution onboarding, workload design, resource utilization, scheduling, data movement, dependency management, resilience and error handling, operational readiness, and production support requirements.

Manage platform configuration, security, and access control

Maintain authentication, authorization, roles, policies, service accounts, certificates, TLS, secrets, network connectivity, encryption settings, and integrations with enterprise identity and security services.

Perform capacity planning and performance optimization

Monitor workload concurrency, queues, memory, CPU, storage, network throughput, JVM behavior, NiFi back pressure, repository usage, Airflow scheduler performance, task throughput, metadata database health, and service saturation; recommend scaling and tuning actions.

Support and maintain non-production and production environments

Provide maintenance, refresh, patching, configuration alignment, access management, test support, troubleshooting, and lifecycle activities for DEV, TEST, SIT, UAT, PROD, and similar environments.

Develop automation and operational tooling

Create and improve scripts, configuration templates, deployment automation, health checks, dashboards, alerts, reporting, and self-service procedures using infrastructure-as-code and CI/CD practices.

Manage platform lifecycle and vulnerability remediation

Coordinate version currency, upgrades, patches, vulnerability assessment, remediation plans, exception handling, risk acceptance, end-of-support tracking, and compliance evidence for platform components.

Lead changes, upgrades, migrations, and recovery tests

Plan and execute platform upgrades, migrations, configuration changes, certificate renewals, component replacements, backup and restore validation, failover exercises, and disaster recovery tests with controlled business impact.

Ensure observability and operational reporting

Define monitoring standards, alert thresholds, service dashboards, availability views, incident trends, capacity reports, and operational KPIs for the supported services.

Contribute to service governance and compliance

Maintain architecture records, technical documentation, runbooks, standards, configuration inventories, risk records, audit evidence, operational controls, and support procedures.

Act as platform subject matter expert

Provide expert guidance on distributed data platforms, data ingestion, orchestration, resilience, security, platform selection, and operational best practices.

Participate in on-call, major incident, and problem management

Support critical incidents, root cause analysis, post-incident reviews, permanent corrective actions, and knowledge sharing across operations, engineering, architecture, and application teams.

Cloudera-Specific Responsibilities

• Administer Cloudera Data Platform services and supporting components, including Cloudera Manager, HDFS, YARN, Hive, Impala, Spark, Ranger, Atlas, Knox, ZooKeeper, Kerberos, Ozone , OTF Iceberg and related platform services where deployed.

• Manage service configuration, role placement, parcels and runtime versions, host maintenance, commissioning and decommissioning, rolling restarts, health checks, configuration validation, and environment consistency.

• Monitor and troubleshoot cluster health, HDFS and Ozone capacity, replication, under-replicated data, YARN queues, workload contention, Hive Metastore, Impala admission control, Spark execution, service logs, JVM resources, and host-level dependencies.

• Administer data-platform security, including Kerberos integration, Ranger policies, Knox access, TLS certificates, keytabs, service identities, encryption-related configuration, and audit logging.

• Support backup, restore, replication, disaster recovery, data retention, metadata protection, and recovery testing for platform services and configuration.

• Plan cluster expansion, operating system and Java compatibility changes, platform upgrades, service migrations, storage evolution, and decommissioning of obsolete components.

Apache NiFi-Specific Responsibilities

• Administer clustered Apache NiFi environments, including nodes, process groups, processors, controller services, parameter contexts, reporting tasks, connection pools, repositories, and registry integration.

• Operate and troubleshoot data flows, queues, back pressure, provenance, bulletins, failed transactions, state management, priorities, concurrent tasks, scheduling, record processors, and interactions with source and target systems.

• Define standards for reusable flow design, version control, promotion between environments, parameterization, secret handling, naming, documentation, error routing, retry logic, dead-letter patterns, and operational ownership.

• Manage NiFi security, including TLS, certificates, authentication, authorization policies, service accounts, proxy configuration, sensitive properties, and integration with enterprise identity services.

• Plan and execute NiFi upgrades, node replacement, cluster expansion, repository maintenance, certificate renewal, flow migration, backup and recovery activities, and resilience testing.

• Monitor flow throughput, queue growth, back pressure, repository utilization, JVM resources, garbage collection, thread usage, connection status, and end-to-end delivery behavior.

Apache Airflow-Specific Responsibilities

• Administer Apache Airflow environments and core components, including schedulers, webservers, triggers, workers, executors, metadata databases, DAG processing, connections, variables, pools, queues, plugins, and providers.

• Operate and troubleshoot DAG parsing, task scheduling, retries, dependencies, sensors, deferrable operators, worker capacity, queue behavior, executor performance, metadata database contention, failed tasks, and scheduler availability.

• Define standards for DAG structure, deployment, version control, environment promotion, secrets, connections, variables, pools, concurrency, retries, alerting, ownership, documentation, and operational readiness.

• Manage Airflow security, including authentication, role-based access control, secrets backends, service identities, TLS, API access, auditability, and integration with enterprise identity and security services.

• Plan and execute Airflow upgrades, provider updates, executor or worker changes, metadata database maintenance, DAG migration, backup and restore validation, and high-availability improvements.

• Monitor scheduler heartbeat, DAG processing time, task latency, queue depth, worker utilization, executor state, triggerer health, metadata database performance, and availability of dependent services.

Key Required Knowledge and Skills

• Deep knowledge of distributed data platform operations, data ingestion, batch and streaming integration, workflow orchestration, scheduling, dependency management, metadata, and data lifecycle concepts.

• Hands-on production administration experience with Cloudera Data Platform, Apache NiFi, and Apache Airflow.

• Strong troubleshooting capability across application, platform, JVM, database, operating system, network, storage, and security layers.

• Ability to analyze logs, metrics, traces, service states, resource queues, flow files, back pressure, provenance, DAG runs, task instances, scheduler behavior, and infrastructure dependencies.

• Ability to lead root cause analysis and define permanent corrective actions for recurring platform, integration, workload, or orchestration issues.

• Experience with high availability, clustering, failover, disaster recovery, backup and restore, business continuity, and recovery testing.

• Strong understanding of security controls: Kerberos, TLS, certificates, LDAP/Active Directory integration, role-based access control, Apache Ranger, secrets management, encryption, and audit logging.

• Good scripting and automation skills using Bash, Python, Ansible, Git, CI/CD tooling, or equivalent technologies.

• Strong Linux administration skills and practical knowledge of JVM tuning, systemd, networking, DNS, load balancing, firewalls, storage, databases, and container platforms.

• Knowledge of IT service management processes, including incident, problem, change, release, configuration, vulnerability, lifecycle, and capacity management.

• Ability to maintain clear technical documentation, runbooks, support models, architecture records, operational procedures, and audit evidence.

• Strong communication and stakeholder management skills, including the ability to coordinate technical activities during incidents, changes, upgrades, migrations, releases, and service improvements.

Key Required Technologies

Cloudera Data Platform

Cloudera Manager, CDP Private Cloud Base, HDFS, Ozone, OTF Iceberg, YARN, Hive, Impala, Spark, Ranger, Atlas, Knox, ZooKeeper, Kerberos, Hive Metastore, resource management, workload governance, backup and disaster recovery.

Apache NiFi

NiFi clustering, processors, controller services, process groups, queues, back pressure, FlowFile and content repositories, provenance, state management, parameter contexts, Registry, record-based processing, Site-to-Site, APIs, security, monitoring, and performance tuning.

Apache Airflow

Schedulers, webservers, triggerers, workers, DAG processors, executors, metadata database, DAGs, operators, sensors, task instances, pools, queues, connections, variables, providers, plugins, secrets backends, REST API, RBAC, monitoring, and performance tuning.

Automation and platform engineering

Python, Bash, Ansible, Git, CI/CD pipelines, infrastructure as code, REST APIs, configuration management, container images, Kubernetes, Helm, and artifact repositories.

Databases and integration

SQL, JDBC/ODBC, relational databases, object storage, Kafka or comparable messaging services, SFTP, REST services, and enterprise integration patterns.

Observability

Prometheus, Grafana, ELK/OpenSearch, Cloudera Manager monitoring, Airflow metrics, NiFi reporting tasks, or comparable enterprise tools.

Infrastructure and security

Linux, Java/JVM, Python runtimes, PostgreSQL or comparable metadata databases, networking, DNS, TLS, certificates, load balancing, nginx, firewalls, proxies, storage, identity management, and secrets management.

ITSM and collaboration

SMAX, Jira, Confluence, Azure DevOps, Teams, or comparable tools used for incident, change, problem, release, and documentation management.

Service Availability

• Business days (Monday–Friday), 8:30 AM–5:00 PM Central European Time.

• Participation in planned maintenance windows, major incident support, release support, lifecycle activities, vulnerability remediation, security patching, disaster recovery tests, and on-call or standby coverage may be required for production and critical non-production services.

“We are committed to creating a diverse environment and are proud to be an equal opportunity employer. All qualified applicants receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, or veteran status”.

At Société Générale, we are convinced that people are drivers of change, and that the world of tomorrow will be shaped by all their initiatives, from the smallest to the most ambitious. Whether you’re joining us for a period of months, years or your entire career, together we can have a positive impact on the future. Creating, daring, innovating, and taking action are part of our DNA. If you too want to be directly involved, grow in a stimulating and caring environment, feel useful on a daily basis and develop or strengthen your expertise, you will feel right at home with us!

Still hesitating?

You should know that our employees can dedicate several days per year to solidarity actions during their working hours, including sponsoring people struggling with their orientation or professional integration, participating in the financial education of young apprentices, and sharing their skills with charities. There are many ways to get involved.

We are committed to support accelerating our Group’s ESG strategy by implementing ESG principles in all our activities and policies. They are translated in our business activity (ESG assessment, reporting, project management or IT activities), our work environment and in our responsible practices for environment protection.