Quality Engineer

1. Project description

Migration of User Knowledge Objects from Splunk to Clickhouse / Grafana / Bosun:

PRE Observability team is undertaking a large-scale observability transformation initiative aimed at migrating from Splunk to a more cost-effective and scalable open-source observability stack based on Clickhouse and Grafana. As part of this effort, we are seeking POD-based teams to support the migration of user knowledge objects such as dashboards, reports, alerts, macros and lookups.

2. Client description

3. Requirements

• Design, develop, and execute data quality test plans for big data systems.
• Monitor data pipelines to ensure data accuracy, completeness, and consistency.
• Implement automated data quality checks and alerting mechanisms using Splunk, ClickHouse, and Grafana.
• Participate in data migration projects, ensuring seamless and accurate data transfer between systems.
• Collaborate with data engineers and architects to identify data quality issues and recommend solutions.
• Create and maintain data validation scripts and test cases for ETL/ELT pipelines.
• Analyze data discrepancies and anomalies, providing root cause analysis and corrective actions.
• Develop dashboards and reports in Grafana to monitor data quality metrics and trends.
• Document data quality processes, standards, and best practices.

3. Details on tech stack

• Bigdata architecture understanding
• Exposure to
• Splunk
• Clickhouse
• Grafana

4. Nice to have requirements to the candidate

5. Current team size, their locations and whether team is going to grow

6. Selling points for this demand (what is good about project / client / team / etc)

1. Project description

Migration of User Knowledge Objects from Splunk to Clickhouse / Grafana / Bosun:

PRE Observability team is undertaking a large-scale observability transformation initiative aimed at migrating from Splunk to a more cost-effective and scalable open-source observability stack based on Clickhouse and Grafana. As part of this effort, we are seeking POD-based teams to support the migration of user knowledge objects such as dashboards, reports, alerts, macros and lookups.

2. Client description

3. Requirements

• Design, develop, and execute data quality test plans for big data systems.
• Monitor data pipelines to ensure data accuracy, completeness, and consistency.
• Implement automated data quality checks and alerting mechanisms using Splunk, ClickHouse, and Grafana.
• Participate in data migration projects, ensuring seamless and accurate data transfer between systems.
• Collaborate with data engineers and architects to identify data quality issues and recommend solutions.
• Create and maintain data validation scripts and test cases for ETL/ELT pipelines.
• Analyze data discrepancies and anomalies, providing root cause analysis and corrective actions.
• Develop dashboards and reports in Grafana to monitor data quality metrics and trends.
• Document data quality processes, standards, and best practices.

3. Details on tech stack

• Bigdata architecture understanding
• Exposure to
• Splunk
• Clickhouse
• Grafana