Job
Description
A career within Data and Analytics services will provide you with the opportunity to help organisations uncover enterprise insights and drive business results using smarter data analytics. We focus on a collection of organisational technology capabilities, including business intelligence, data management, and data assurance that help our clients drive innovation, growth, and change within their organisations in order to keep up with the changing nature of customers and technology. We make impactful decisions by mixing mind and machine to leverage data, understand and navigate risk, and help our clients gain a competitive edge. Manage the development of endtoend data ingestion, transformation, and ETL workflows using Hadoop, PySpark, and related big data technologies. Design and implementation of distributed computing frameworks and realtime data processing solutions using Spark and PySpark. Ensure the effective management and operation of Hadoop clusters (HDFS, YARN, Hive, HBase) and related big data tools, ensuring high availability, security, and performance. Develop and maintain data pipelines to process large volumes of structured and unstructured data, ensuring data consistency, integrity, and efficiency. Provide technical guidance, mentorship, and career development for junior engineers and team members in the Hadoop and PySpark ecosystem. Design and implementation of data models and data storage strategies, including integration with cloud platforms such as AWS, Google Cloud, or Azure. Optimize performance of data pipelines and queries, troubleshooting any issues related to data processing and performance bottlenecks. Ensure adherence to best practices in code quality, data governance, and security standards. Communicate complex technical challenges and solutions effectively to both technical and nontechnical stakeholders. Drive innovation by staying uptodate with the latest big data technologies and incorporating relevant advancements into the platform. 3 years of experience in Big Data Engineering, with a strong background in Hadoop and PySpark (at least 23 years in a leadership or managerial role). Proven experience in designing and implementing data pipelines using PySpark for largescale data processing. Strong understanding and handson experience with the Hadoop ecosystem, including HDFS, MapReduce, Hive, HBase, and YARN. Proficiency in Python and PySpark, including working with distributed computing frameworks and data processing workflows. Expertise in working with ETL processes, data lakes, data warehousing, and cloud platforms (AWS, GCP, or Azure). Mandatory skill sets Pyspark, Python, Hadoop, Sql Education qualification B.Tech / M.Tech / MBA / MCA Education Degrees/Field of Study required Master of Business Administration, Master of Engineering, Bachelor of Engineering Degrees/Field of Study preferred Required Skills Python (Programming Language) Optional Skills Accepting Feedback, Accepting Feedback, Active Listening, Agile Scalability, Amazon Web Services (AWS), Analytical Thinking, Apache Hadoop, Azure Data Factory, Communication, Creativity, Data Anonymization, Database Administration, Database Management System (DBMS), Database Optimization, Database Security Best Practices, Data Engineering, Data Engineering Platforms, Data Infrastructure, Data Integration, Data Lake, Data Modeling, Data Pipeline, Data Quality, Data Transformation, Data Validation
