Big Data Architect

Job Description

Company Description Airtel is a leading global connectivity provider dedicated to unlocking endless opportunities for its consumers. With a focus on innovation and creativity, Airtel delivers impactful solutions and cutting-edge technologies like 5G, IoT, IQ, and Airtel Black. Committed to a 'Digital India' vision, Airtel serves millions of individuals with a subscriber base exceeding 574 million as of 2023. Role Description This is a full-time on-site Big Data Architect role located in Gurugram at Airtel. The Big Data Architect will be responsible for designing, implementing, and maintaining the company's big data infrastructure. This role involves analyzing data requirements, creating data models, and ensuring data security and integrity. Responsibilities · Must be capable of handling existing or new Apache HDFS cluster having name node, data node & edge node commissioning & decommissioning. · Work closely with data architects and analysts to design technical solutions. · Integrate and ingest data from multiple source systems into big data environments. · Develop end-to-end data transformations and workflows, ensuring logging and recovery mechanisms. · Must able to troubleshoot spark job failures. · Design and implement batch, real-time, and near-real-time data pipelines. · Optimize Big Data transformations using Apache Spark, Hive, and Tez · Work with Data Science teams to enhance actionable insights. Ensure seamless data integration and transformation across multiple systems. Qualifications · 4+ year’s of mandatory experience with Big data · 4+ year’s mandatory experience in Apache Spark. · Proficiency in Apache Spark, Hive on Tez, and Hadoop ecosystem components. · Strong coding skills in Python & Pyspark. · Experience building reusable components or frameworks using Spark · Expertise in data ingestion from multiple sources using APIs, HDFS, and NiFi. · Solid experience working with structured, unstructured, and semi-structured data formats (Text, JSON, Avro, Parquet, ORC, etc.). · Experience with UNIX Bash scripting and databases like Postgres, MySQL and Oracle. · Ability to design, develop, and evolve fault-tolerant distributed systems. · Strong SQL skills, with expertise in Hive, Impala, Mongo and NoSQL databases. · Hands-on with Git and CI/CD tools · Experience with streaming data technologies (Kafka, Spark Streaming, Apache Flink, etc.). · Proficient with HDFS, or similar data lake technologies · Excellent problem-solving skills — you will be evaluated through coding rounds Show more Show less