Senior Data Engineer

Experience: Min 6+ Years Job Title: Data Engineer – Real-Time Streaming & Integration (Apache Kafka) Location: Bhopal, Madhya Pradesh On-site role with opportunities to work on enterprise-scale data platforms Note: Resource working on site will be provided with accommodation, lunch, and dinner by the client for the complete project duration. The working week is 6 days (Monday – Saturday). Role Overview: We are seeking a highly skilled and experienced Data Engineer with 6+ years of experience in designing and implementing real-time data processing pipelines and streaming integrations. This role is ideal for professionals with deep expertise in Apache Kafka, Kafka Connect, and modern ETL/ELT processes. As a Data Engineer, you will play a critical role in building and optimizing data integration frameworks to support large-scale, low-latency, and high-throughput data platforms across enterprise systems. Your contributions will directly impact data accessibility, business intelligence, and operational efficiency. Key Responsibilities:  Design, develop, and maintain real-time streaming data pipelines using Apache Kafka and Kafka Connect.  Implement and optimize ETL/ELT processes for structured and semi-structured data from various sources.  Build and maintain scalable data ingestion, transformation, and enrichment frameworks across multiple environments.  Collaborate with data architects, analysts, and application teams to deliver integrated data solutions that meet business requirements.  Ensure high availability, fault tolerance, and performance tuning for streaming data infrastructure.  Monitor, troubleshoot, and enhance Kafka clusters, connectors, and consumer applications.  Enforce data governance, quality, and security standards throughout the pipeline lifecycle.  Automate workflows using orchestration tools and CI/CD pipelines for deployment and version control. Required Skills & Qualifications:  Strong hands-on experience with Apache Kafka, Kafka Connect, and Kafka Streams.  Expertise in designing real-time data pipelines and stream processing architectures.  Solid experience with ETL/ELT frameworks using tools like Apache NiFi, Talend, or custom Python/Scala-based solutions.  Proficiency in at least one programming language: Python, Java, or Scala.  Deep understanding of message serialization formats (e.g., Avro, Protobuf, JSON).  Strong SQL skills and experience working with data lakes, warehouses, or relational databases.  Familiarity with schema registry, data partitioning, and offset management in Kafka.  Experience with Linux environments, containerization, and CI/CD best practices. Preferred Qualifications:  Experience with cloud-native data platforms (e.g., AWS MSK, Azure Event Hubs, GCP Pub/Sub).  Exposure to stream processing engines like Apache Flink or Spark Structured Streaming.  Familiarity with data lake architectures, data mesh concepts, or real-time analytics platforms.  Knowledge of DevOps tools like Docker, Kubernetes, Git, and Jenkins. Work Experience:  6+ years of experience in data engineering with a focus on streaming data and real-time integrations.  Proven track record of implementing data pipelines in production-grade enterprise environments. Education Requirements:  Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related field.  Certifications in data engineering, Kafka, Show more Show less