Big Data Developer

• Design and develop scalable data pipelines to migrate user knowledge objects from Splunk to ClickHouse and Grafana.
• Implement data ingestion, transformation, and validation processes to ensure data integrity and performance.
• Collaborate with cross-functional teams to automate and optimize data migration workflows.
• Monitor and troubleshoot data pipeline performance and resolve issues proactively.
• Work closely with observability engineers and analysts to understand data requirements and deliver solutions.
• Contribute to the continuous improvement of the observability stack and migration automation tools.

Required Skills and Qualifications

• Proven experience as a Big Data Developer or Engineer working with large-scale data platforms.
• Strong expertise with ClickHouse or other columnar databases, including query optimization and schema design.
• Hands-on experience with Splunk data structures, dashboards, and reports.
• Proficiency in data pipeline development using technologies such as Apache Spark, Kafka, or similar frameworks.
• Strong programming skills in Python, Java, or Scala.
• Experience with data migration automation and scripting.
• Familiarity with Grafana for data visualization and monitoring.
• Understanding of observability concepts and monitoring systems.

Would be a plus

• Experience with Bosun or other alerting platforms.
• Knowledge of cloud-based big data services and infrastructure as code.
• Familiarity with containerization and orchestration tools (Docker, Kubernetes).
• Experience working in agile POD-based teams.