Senior Data Engineer — Backend Enablement Team

Introduction

• Opportunity to shape the next generation of observability data infrastructure within a global IBM product.
• Collaborative science-driven, focused on quality, automation, and long-term maintainability.
• Work alongside experienced architects and engineers on projects spanning schema evolution, encryption, and CI/CD optimization.
• Continuous learning and professional certification support.
  

Your Role And Responsibilities

• To build scalable, schema-driven, and compliant data pipelines powering real-time analytics, search, and long-term storage across Kafka, ClickHouse, PostgreSQL, and OpenSearch ecosystems.
• ClickHouse Optimization & Scaling — Research, plan, and implement ClickHouse cluster optimizations for multi-tenant workloads. Design rollup and retention strategies to balance performance, cost, and compliance.
• Big Data Tooling & Integrations — Introduce and evaluate Big Data components such as S3/object storage, serverless databases, Kafka, Kafka Connect, and Kafka Schema Registry, adopting modern data-warehouse practices. Design data-in-motion and data-at-rest pipelines with strong encryption, efficient compression and schema-driven governance.
• Elasticsearch → OpenSearch Migration — Lead research, planning, and phased migration of existing Elasticsearch clusters and APIs to OpenSearch. Ensure backward compatibility, observability, and search performance parity.
• Datastore End-of-Life (EOL) Management — Research, plan, and implement decommissioning strategies for Cassandra and other legacy datastores. Define migration paths, data export/import automation, and service continuity procedures.
• Schema & Data Format Engineering — Define and implement schema-driven data formats such as Parquet, Avro, and Cryo, or columnar compression for analytical workloads. Support versioned schema evolution across ClickHouse, PostgreSQL, and OpenSearch.
• Compliance, Security & Data Governance — Embed compliance requirements (SOC2, FedRAMP, GDPR) into data retention, protection, and encryption layers. Contribute to audit-ready documentation, validation tooling, and policy enforcement automation.
  

Required Technical And Professional Expertise

• Proven expertise with analytical databases and data pipeline design (ClickHouse, PostgreSQL, BigQuery, Redshift, etc.).
• Strong programming background in Python, SQL, and Bash (plus optional Java or JavaScript experience).
• Experience with Kafka, Kafka Connect, and Schema Registry, plus search systems (Elasticsearch/OpenSearch) and NoSQL (Cassandra, DynamoDB).
• Solid understanding of schema-driven design, data compression, and high-throughput ETL/ELT pipelines.
• Familiarity with data protection, compliance, and security frameworks (SOC2, FedRAMP, GDPR, FIPS).
• Practical experience in public cloud ecosystems (AWS, GCP, IBM Cloud) and certifications are a strong plus.
  

Preferred Technical And Professional Experience

• Experience with containerized test environments (Docker, Testcontainers) and CI/CD systems (CodeBuild, Jenkins).
• Contributions to data platform migrations or modernization projects.
• Knowledge of infrastructure-as-code, serverless services, and observability tooling.
• Certification or formal training in Data Engineering, Cloud Architecture, or Information Security.