Big Data Engineer

Key Responsibilities:

• Develop high quality, secure and scalable data pipelines using spark, Scala/Python/Java on Hadoop or object storage like MinIO.

• Leverage technologies and solutions to innovate with increasingly large data sets.

• Drive automation and efficiency in Data ingestion, data movement and data access workflows by innovation and collaboration.
• Understand, implement and enforce Software development standards and engineering principles in the Big Data space.

• Contribute ideas to help ensure that required standards and processes are in place and actively look for opportunities to enhance standards and improve process efficiency.

• Perform assigned tasks and production incident independently.

Must Have:

• 5-9 years of experience in Data Warehouse/Data Lake/Lake House related projects in product or service-based organization

• Expertise in Data Engineering and implementing multiple end-to-end DW projects in Big Data environment handling petabyte scale data.

• Solid Experience of building complex data pipelines through Spark with Scala/Python/Java on Hadoop or Object storage •

Experience of working with Databases like Oracle, Netezza and have strong SQL knowledge.

• Proficient in working within an Agile/Scrum framework, including creating user stories with well-defined acceptance criteria, participating in sprint planning and reviews
• Write and maintain Unix shell scripts, Oracle SQL, PL/SQL, and perform SQL tuning.
• Optimize and troubleshoot Spark applications for performance, scalability, and fault tolerance

• Use Git-based version control systems and CI/CD pipelines (e.g., Jenkins).

• Implement and manage HIVE external tables, partitions, and various file formats.

• Work across on-premises and cloud environments (AWS, Azure, Databricks).

• Strong experience with Hadoop ecosystem and Cloudera Data Platform (CDP).

Experience of building Nifi pipelines .

Preferred

• Strong analytical skills required for debugging production issues, providing root cause and implementing mitigation plan

• Strong communication skills - both verbal and written.

• Ability to multi-task across multiple projects, interface with external / internal resources.

• Proactive, detail-oriented and able to function under pressure in an independent environment along with a high degree of initiative and self-motivation to drive results

• Willingness to quickly learn and implement new technologies, participate POC to explore best solution for the problem statement

• Experience working diverse and geographically distributed project teams.

Skills & Qualifications: Bachelors degree in information systems, Information Technology, Computer Science or Engineering or equivalent work experience.