Senior Data Engineer

Job Title: Senior Data Engineer - Apache Flink

Job Specs : 7+ Years

Location: Hyderabad, TG / Chennai, TN

Work Mode: Hybrid

Shifts: 2 PM - 11 PM

Key Responsibilities:

Role Summary:

• This role will be instrumental in building and maintaining robust, scalable, and reliable data pipelines using Confluent Kafka, ksqlDB, Kafka Connect, and Apache Flink.
• The ideal candidate will have a strong understanding of data streaming concepts, experience with real-time data processing, and a passion for building high-performance data solutions.
• This role requires excellent analytical skills, attention to detail, and the ability to work collaboratively in a fast-paced environment.

Essential Responsibilities

• Design & develop data pipelines for real-time and batch data ingestion and processing using Confluent Kafka, ksqlDB, Kafka Connect, and Apache Flink.
• Build and configure Kafka Connectors to ingest data from various sources (databases, APIs, message queues, etc.) into Kafka.
• Develop Flink applications for complex event processing, stream enrichment, and real-time analytics.
• Develop and optimize ksqlDB queries for real-time data transformations, aggregations, and filtering.
• Implement data quality checks and monitoring to ensure data accuracy and reliability throughout the pipeline.
• Monitor and troubleshoot data pipeline performance, identify bottlenecks, and implement optimizations.
• Automate data pipeline deployment, monitoring, and maintenance tasks.
• Stay up-to-date with the latest advancements in data streaming technologies and best practices.
• Contribute to the development of data engineering standards and best practices within the organization.
• Participate in code reviews and contribute to a collaborative and supportive team environment.
• Work closely with other architects and tech leads in India & US and create POCs and MVPs
• Provide regular updates on the tasks, status and risks to project manager
• The experience we are looking to add to our team

Required

• Bachelor’s degree or higher from a reputed university
• 7 to 10 years total experience with majority of that experience related to ETL/ELT, big data, Kafka etc.
• Proficiency in developing Flink applications for stream processing and real-time analytics.
• Strong understanding of data streaming concepts and architectures.
• Extensive experience with Confluent Kafka, including Kafka Brokers, Producers, Consumers, and Schema Registry.
• Hands-on experience with ksqlDB for real-time data transformations and stream processing.
• Experience with Kafka Connect and building custom connectors.
• Extensive experience in implementing large scale data ingestion and curation solutions
• Good hands on experience in big data technology stack with any cloud platform -
• Excellent problem-solving, analytical, and communication skills.
• Ability to work independently and as part of a team

Good to have

• Experience in Google Cloud
• Healthcare industry experience
• Experience in Agile