Job
Description
Hi , Greetings from Nitor infotech! Please find below the Company Job Description with this mail 6+ years candidates only apply Immediate joiners Preferred Python ,Hadoop,hive,spark and SQL are mandatory Job Description: Primary Skills: •6+ years of Data Engineering experience. •4+ years of experience in Big Data technologies (e.g. Spark, Hive, Hadoop, etc.). •Strong Experience designing and implementing data pipelines. •Excellent knowledge of Data engineering concepts and best practices. •Proven ability to lead, mentor, inspire and support more junior team members. •Able to lead technical deliverables autonomously and lead more junior data engineers. •Strong attention to detail and working according to best practices. •Experience in designing solution using batch data processing methods, real-time streams, ETL processes and Business Intelligence tools. •Experience designing Logical Data Model and Physical Data Models including data warehouse and data mart designs. •Strong SQL knowledge & experience (T-SQL, working with SQL Server, SSMS) •Apache Spark: Advanced proficiency with Spark, including PySpark and SparkSQL, for distributed data processing. •Working knowledge of Apache Hive •Proficiency in Python, Pandas, PySpark (Scala knowledge is a plus). •Knowledge of Delta Lake concepts and common data formats, Lakehouse architecture. •Source control with Git. •Expertise in designing and implementing scalable data pipelines and ETL processes using the GCP data stack including: BigQuery, Dataflow, Pub/Sub, Cloud Storage, Cloud Composer, Cloud Functions, Dataproc (spark) •Apache Airflow - Expertise in building and managing ETL workflows using Airflow, including DAG creation, scheduling, and error handling. •Knowledge of CI/CD concepts, experience designing CI/CD for data pipelines. Secondary Skills: •Experience with Streaming services such as Kafka is a plus. •R & Sparklyr experience is a plus •Knowledge of MLOps concepts, AI/ML life-cycle management, Mlflow •Expertise in writing complex, highly optimized queries across large data sets to write data pipelines and data processing layers. •Jenkins Candidate Profile: Design, build, test and deploy innovative Big Data solutions at scale. Extract, Clean, transform, and analyse vast amounts of raw data from various Data Sources. Build data pipelines and API integrations with various internal systems. Work Across all stages of Data Lifecycle Implement best practices across all Data Analytics Processes Estimate effort, identify risks, and plan execution. Proactively monitor, identify, and escalate issues or root causes of systemic issues. Enable data scientists, business, and product partners to fully leverage our platform. Engage with business stakeholders to understand client requirements and build technical solutions and delivery plans. Evaluate and communicate technical risks effectively and ensure assignments delivery in scheduled time with desired quality. Provide end to end big data solution and design details to data engineering teams. Excellent analytical & problem-solving skills Excellent communication skills, experience communicating with Snr. Business stakeholders Leading technical delivery on use-cases, able to plan and delegate tasks to more junior team members, oversee the work from inception to final product. Key Required Skills: Apache Airflow, Kafka, SQL, Data Engineering, CI CD pipelines , Big Data, Apache Spark, Logical Data Model, Physical Data Model Wish you all the best! Thanks & Regards, VIGNESH 6379146150
