Data Engineer

We are looking for resource with Kafka + Kafka Streams with Scala Experience.

Responsibilities:

• Design, develop, and maintain real-time ETL pipelines using Apache Kafka Streams with Scala.
• Preprocess structured and unstructured data from various sources for analytics.
• Enhance data ingestion and collection procedures to ensure data quality and completeness.
• Process, cleanse, and validate data integrity for analytics and reporting.
• Analyze large-scale streaming data to identify patterns and actionable insights.
• Develop and implement machine learning models and algorithms within streaming data pipelines.
• Present analytical results and findings clearly to stakeholders.
• Propose solutions and strategies to address business challenges using streaming data.
• Perform data mining and extraction from diverse, high-volume data sources.
• Utilize machine learning tools to select features, build, and optimize real-time classifiers.

Qualifications:

• Strong programming skills in Scala; experience with Python or Java is a plus.
• Hands-on experience with Apache Kafka, Kafka Streams, and real-time ETL processing in Scala.
• Familiarity with Open Telemetry stack, including VMs, middleware (IIS/Apache), hypervisors, and databases; understanding of OTLP format and data structuring.
• Experience correlating streaming OLTP data from multiple components and applying machine learning for actionable outcomes.
• Solid understanding of distributed systems, data modeling, and data warehousing concepts.
• Experience with cloud platforms and containerization (Docker, Kubernetes) is desirable.
• Excellent problem-solving, communication, and collaboration skills.