Must have minimum of 12 15 years of experience implementing batch and streaming applications using Spark and Scala
Experience with Confluent or Apache Kafka and Connectors
Experience with Java and building microservices, web applications or other applications
Develop data pipelines responsible for ingesting large amounts of data from various sources
Help evolve data architecture and work on Next Generation real time pipeline algorithms and architecture in addition to supporting and maintaining current pipelines and legacy systems
Write code and develop worker node for business logic, ETL and orchestration processes
Develop algorithms for better attribution rules and category classifiers
Work with stakeholders throughout the organization to identify opportunities for leveraging company data to drive search, discovery, and recommendations.
Work closely with architects, engineers, data analysts, data scientists, contractors/consultants and project managers in assessing project requirements, design, develop and support data ingestions and API services
Work with Data Scientists in integrating machine learning models during content enrichment process
Able to influence on priorities working with various partners including engineers, project management office and leadership
Mentor junior team members, define architecture, code review, hands-on development and deliver the work in sprint cycle
Participate in design discussions with Architects and other team members for the design of new systems and re-engineering of components of existing systems
Wear Architect hat when required to bring new ideas to the table based on past experience, thought leadership and forward thinking
Take holistic approach to building solutions by thinking big picture and overall solution
Work on moving away from legacy systems into next generation architecture
Take complete ownership from requirements, solution design, development, production launch and post launch production support. Participate in code reviews and regular on-call rotations.
Desire to learn best practices and data engineering tools and technologies.

Experience and Education:

BS or MS in Computer Science (or related field) with 12 15 years of hands-on software development experience working in large-scale data processing pipelines
Must have skills are Apache Spark 3.x and Scala with 4-5 years of experience building production grade batch pipelines that handle large volumes of data.
8+ years of experience in Java and API / Microservices
8+ years of experience in understanding and writing complex SQL and stored procedures for processing raw data, ETL, data validation, using databases such as SQL Server, Redis and other NoSQL DBs
Knowledge of Big data technologies, Hadoop, HDFS
Expertise with building spark streaming pipelines with Kafka
Expertise with Amazon AWS stack such as EMR, EC2, S3
Experience working with APIs to collect and ingest data as well build the APIs for business logic
Experience working with setting up, maintaining, and debugging production systems and infrastructure
Experience in building fault-tolerant and resilient system
Experience in building worker nodes, knowledge of REST principles and data engineering design patterns
In-depth knowledge of Spark, Scala, Java, Orchestration tools, ESB, SQL, Stored procedures, Docker, RESTful web services, Kubernetes, CI/CD, Observability techniques, Kafka, Release processes, caching strategies, versioning, B&D, Bitbucket / Git and AWS Cloud Ecosystem
Strong software development, architecture diagramming, problem-solving and debugging skills
Phenomenal communication and influencing skills.

Nice to have:

Knowledge of Elastic APM, ELK stack and search technologies such as Elasticsearch/Solr
Exposure to Machine Learning (ML) concepts
Nice to have some experience in workflow orchestration tools such as Air Flow or Apache NiFi.

More Jobs at Divit Technologies

Sr Frontend Engineer

chennai, bengaluru

13.0 - 18.0 yrs

INR 45 - 50 Lacs

Java Full Stack + Test Automation Engineer

chennai, bengaluru

6.0 - 8.0 yrs

INR 8 - 10 Lacs

Oracle and Axiom Developer

chennai, bengaluru

8.0 - 10.0 yrs

INR 25 - 30 Lacs

DevOps Engineer

chennai, bengaluru

7.0 - 8.0 yrs

INR 5 - 9 Lacs

Universe Integration Lead

chennai, bengaluru

10.0 - 15.0 yrs

INR 15 - 19 Lacs

Mock Interview

Practice Video Interview with JobPe AI

Start Java Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.