Job Description

Hi Jobseekers, We are Hiring ! Immedidate joiners! We are looking for Immediate joiners for the below position , Job titlePython + Pyspark Exp7+ Notice periodImmediate joiner LocationPune & Hyderabad If interested, Pl let me know Pl mail me @ lavanya j@natobotics com with the below details Exp- CTC_ Exp CTC- Notice period- Location- Preferred Location We assure our best service always JD below, Job Title: Python + PySpark Developer Location: [Location] Employment Type: Full-time / Contract About The Role We are looking for a skilled Python + PySpark Developer to join our team and help design, develop, and optimize data processing workflows You will work on large-scale distributed data processing systems using Apache Spark and PySpark, processing vast amounts of data efficiently This is an excellent opportunity for individuals passionate about big data and advanced analytics Key Responsibilities Design and implement scalable data processing pipelines using PySpark for batch and real-time data processing Develop Python scripts and libraries to integrate, cleanse, and process data from various data sources (databases, APIs, etc ) Optimize PySpark jobs for performance and resource efficiency Collaborate with data engineers and analysts to define data transformation and aggregation requirements Analyze and troubleshoot issues in large-scale distributed data systems Develop custom functions for data manipulation and aggregation using Spark DataFrame API and RDD API Perform data analysis and create insightful visualizations for internal stakeholders Write unit tests, document code, and maintain code quality through peer reviews and best practices Monitor and optimize the performance of Apache Spark clusters Work with cloud platforms such as AWS, Azure, or GCP to deploy Spark jobs and other data processing services Skills And Qualifications Strong experience with Python programming language and libraries such as Pandas, NumPy, Pytest, etc Hands-on experience with Apache Spark and PySpark for distributed data processing Familiarity with data formats such as CSV, JSON, Parquet, Avro, and other big data file formats Solid understanding of Hadoop ecosystem components (HDFS, YARN, etc ) Proficiency with data wrangling, transformation, and aggregation Experience with performance tuning, resource management, and debugging Spark applications Understanding of cloud-based data storage solutions (S3, GCS, etc ) and distributed file systems Familiarity with data warehousing concepts and tools Good knowledge of SQL and relational databases (e g , PostgreSQL, MySQL) Strong problem-solving skills and ability to work with large datasets Excellent communication and collaboration skills, with the ability to work effectively in a team environment Preferred Qualifications Bachelors or Masters degree in Computer Science, Engineering, or a related field Experience with additional big data frameworks like Flink, Kafka, Hive, etc Knowledge of Docker and containerization for deployment Experience with ETL tools like Airflow or Luigi Familiarity with machine learning concepts and frameworks (e g , MLlib, Scikit-learn)