Data Engineer

Role : Data Engineer

Experience : 4 - 6 years

Location : Navi Mumbai

• Set up, configure, and maintain HDFS, Spark, and Druid clusters.
• Monitor cluster health, performance, and resource utilization.
• Manage disk allocations, node scaling, and failover strategies.
• Design and implement scalable ETL workflows for batch and streaming data.
• Optimize data ingestion and transformation processes.
• Implement monitoring solutions for data pipelines and clusters.
• Troubleshoot performance bottlenecks and ensure high availability.
• Use tools like Prometheus, Grafana, or custom dashboards for observability.
• Develop and maintain Kafka producers/consumers for real-time data streaming.
• Ensure reliable message delivery and topic management.
• Monitor Kafka clusters and optimize throughput and latency.
• Work closely with data scientists, analysts, and DevOps teams.
• Document architecture, processes, and best practices.