Data engineer + Kafka + Flink

Job Description

Role & responsibilities Design develop data pipelines for realtime and batch data ingestion and processing using Confluent Kafka ksqlDB Kafka Connect and Apache Flink Build and configure Kafka Connectors to ingest data from various sources databases APIs message queues etc into Kafka Develop Flink applications for complex event processing stream enrichment and realtime analytics Develop and optimize ksqlDB queries for realtime data transformations aggregations and filtering Implement data quality checks and monitoring to ensure data accuracy and reliability throughout the pipeline Monitor and troubleshoot data pipeline performance identify bottlenecks and implement optimizations Automate data pipeline deployment monitoring and maintenance tasks Stay uptodate with the latest advancements in data streaming technologies and best practices Contribute to the development of data engineering standards and best practices within the organization Participate in code reviews and contribute to a collaborative and supportive team environment Work closely with other architects and tech leads in India US and create POCs and MVPs Provide regular updates on the tasks status and risks to project manager Preferred candidate profile Bachelors degree or higher from a reputed university 8 to 10 years total experience with majority of that experience related to ETLELT big data Kafka etc Proficiency in developing Flink applications for stream processing and realtime analytics Strong understanding of data streaming concepts and architectures Extensive experience with Confluent Kafka including Kafka Brokers Producers Consumers and Schema Registry Handson experience with ksqlDB for realtime data transformations and stream processing Experience with Kafka Connect and building custom connectors Extensive experience in implementing large scale data ingestion and curation solutions Good hands on experience in big data technology stack with any cloud platform Excellent problemsolving analytical and communication skills Ability to work independently and as part of a team Good to have Experience in Google Cloud Healthcare industry experience Experience in Agile Skills Mandatory Skills : AWS Kinesis,Java,Kafka,Python,AWS Glue,AWS Lambda,AWS S3,Scala,Apache SparkStreaming,ANSI-SQL"