Job Description

Designing implementing Data Pipelines Frameworks to provide a better developer experience for our dev teams. Helping other PODs in IDfy define their data landscape and onboarding them onto our platform. Keep abreast of the latest trends and technologies in Data Engineering, GenAI, and Natural Language Query. Set up logging, monitoring, and alerting mechanisms for better visibility into data pipelines and platform health. Automate repetitive data tasks to improve efficiency and free up engineering bandwidth. Maintain technical documentation to ensure knowledge sharing and onboarding efficiency. Troubleshoot and resolve bottlenecks in data processing, ingestion, and transformation pipelines. We Are the Perfect Match If You Have experience creating and managing large-scale data ingestion pipelines using the ELT (Extract, Load, Transform) model. In your current role, take ownership of defining data models, transformation logic, and data flow. Are proficient in Logstash, Apache BEAM Dataflow, Apache Airflow, ClickHouse, Grafana, InfluxDB/VictoriaMetrics, and BigQuery. Strong understanding and hands-on experience with data warehouses, with at least 3 years of experience in any data warehousing stack. Have a keen eye for data and can derive meaningful insights from it. Understand product development methodologies; we follow Agile. Have experience with Time Series Databases (we use InfluxDB VictoriaMetrics) and alerting/anomaly detection frameworks (preferred but not mandatory). Are familiar with visualization tools such as Metabase, Power BI, or Tableau. Have experience developing software in the cloud (GCP/AWS is preferred, but hands-on experience is not mandatory). Are passionate about exploring new technologies and enjoy sharing your knowledge through technical blogs.