Job
Description
We are seeking an experienced and highly skilled IBM StreamSets Developer to design, develop, and optimize high-performance data pipelines. The ideal candidate will be an independent contributor with deep expertise in StreamSets Data Collector (SDC) and Transformer, capable of analyzing pipeline performance bottlenecks and implementing optimizations for scalability, reliability, and efficiency. Key Responsibilities: Design, develop, and deploy robust StreamSets data pipelines for batch and real-time data ingestion, transformation, and delivery. Analyze and troubleshoot pipeline performance bottlenecks (CPU, memory, I/O, latency) and implement optimizations. Fine-tune JVM settings, parallelism, partitioning, and batch sizes for optimal throughput. Implement best practices for error handling, data validation, and recovery mechanisms in pipelines. Optimize slow-running stages, memory-heavy transformations, and network latency issues. Work with Kafka, JDBC, REST APIs and other data sources/destinations. Monitor pipeline health using StreamSets Control Hub and set up alerts for failures. Collaborate with Cross functional teams, architects, and DevOps to ensure high-performance data flows. Document pipeline architecture, optimizations, and performance benchmarks. Required Skills & Experience: 3+ years of hands-on experience with IBM StreamSets (Data Collector & Transformer). Strong understanding of pipeline performance tuning (e.g., stage optimization, buffer tuning, cluster resource allocation). Proficiency in Java/Python/Groovy for custom scripting in StreamSets. Knowledge of SQL, NoSQL databases, and CDC (Change Data Capture) techniques. Ability to diagnose and resolve memory leaks, thread contention, and network bottlenecks. Familiarity with CI/CD for StreamSets pipelines (Git, Jenkins, Docker). Strong analytical skills to profile and benchmark pipeline performance. Nice to Have: StreamSets certification (e.g., StreamSets Engineer). Experience with Kubernetes for containerized StreamSets deployments. Knowledge of data observability tools (Datadog).
