Senior Data Engineer

Role:

• Ability to design, build and unit test applications on Spark framework on Scala and Python.
• Build Spark based applications for both batch and streaming requirements, which will require in-depth knowledge on majority of Hadoop and NoSQL databases as well.
• Develop and execute data pipeline testing processes and validate business rules and policies
• Optimize performance of the built Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame, and Pair RDD's.
• Optimize performance for data access requirements by choosing the appropriate native Hadoop file formats (Avro, Parquet, ORC etc) and compression codec respectively.
• Ability to design & build real-time applications using Apache Kafka & Spark Streaming
• Build integrated solutions leveraging Unix shell scripting, RDBMS, Hive, HDFS File System, HDFS File Types, HDFS compression codec.
• Build data tokenization libraries and integrate with Hive & Spark for column-level obfuscation
• Experience in processing large amounts of structured and unstructured data, including integrating data from multiple sources.
• Create and maintain integration and regression testing framework on Jenkins integrated with Bitbucket and/or GIT repositories
• Participate in the agile development process, and document and communicate issues and bugs relative to data standards in scrum meetings
• Work collaboratively with onsite and offshore team.
• Develop & review technical documentation for artifacts delivered.
• Ability to solve complex data-driven scenarios and triage towards defects and production issues
• Ability to learn-unlearn-relearn concepts with an open and analytical mindset
• Participate in code release and production deployment.
• Challenge and inspire team members to achieve business results in a fast paced and quickly changing environment.

Responsibilities:

• BE/B.Tech/ B.Sc. in Computer Science/ Statistics, Econometrics from an accredited college or university.
• Minimum 3 years of extensive experience in design, build and deployment of Spark-based applications.
• Expertise in handling complex large-scale Big Data environments preferably (20Tb+).
• Minimum 3 years of experience in the following: HIVE, YARN, HDFS preferably on Cloudera Data Platform.
• Good implementation experience of OOPS concepts.
• Hands-on experience writing complex SQL queries, exporting, and importing large amounts of data using utilities.
• Ability to build abstract, modularized reusable code components.
• Hands-on experience in generating/parsing XML, JSON documents, and REST API request/responses
• Able to quickly adapt and learn.
• Able to jump into an ambiguous situation and take the lead on resolution.
• Able to communicate and coordinate across various teams.
• Is comfortable tackling new challenges and new ways of working
• Are ready to move from traditional methods and adapt into agile ones
• Comfortable challenging your peers and leadership team.
• Can prove yourself quickly and decisively.
• Excellent communication skills and Good Customer Centricity.
• Strong Target & High Solution Orientation.

About EXL