Job
Description
Greetings from Clover Infotech!!! Please review the job details and share the required details if you are interested in proceeding further. If you are not interested, request you to help me reach the right candidate Job Title: Data Engineer (PySpark) Location: Bangalore Employment Type: - Full-Time Experience Required: 3+ years About the Role We are looking for a highly skilled Data Engineer (PySpark) to join our dynamic data engineering team. This position is ideal for a technically sound and detail-oriented professional who has deep expertise in PySpark and the Cloudera Data Platform (CDP) . You will be instrumental in designing, developing, and maintaining scalable data pipelines, ensuring high data quality, and enabling data availability across our organization. The ideal candidate brings hands-on experience with big data technologies, data ingestion, transformation, and optimization using Clouderas ecosystem. Youll collaborate with cross-functional teams to create robust, high-performing data solutions that support business insights and strategic decision-making. Key Responsibilities Data Pipeline Development : Design, build, and manage scalable and high-performance ETL pipelines using PySpark on Cloudera. Data Ingestion : Integrate data from multiple sources (relational databases, APIs, file systems) into CDP environments. Data Processing : Cleanse, transform, and process large datasets to meet business and analytics needs. Performance Optimization : Fine-tune PySpark jobs and Cloudera tools to improve ETL performance and resource utilization. Data Quality : Implement and maintain data validation, error handling, and quality control processes. Workflow Automation : Automate jobs using tools like Apache Oozie, Airflow, or similar orchestration frameworks. Monitoring and Maintenance : Monitor job performance and ensure system reliability through proactive issue identification and resolution. Team Collaboration : Work closely with data analysts, data scientists, and business stakeholders to understand requirements and deliver data-driven solutions. Documentation : Create and maintain clear, comprehensive documentation for pipelines, processes, and systems. Qualifications Education & Experience Bachelor’s or Master’s degree in Computer Science, Information Systems, or a related field. Minimum 3 years of hands-on experience in a Data Engineering role with a focus on PySpark and Cloudera Data Platform. Technical Skills Advanced experience with PySpark (RDDs, DataFrames, performance tuning). Strong knowledge of Cloudera Data Platform and its components: Cloudera Manager, Hive, Impala, HDFS, HBase. Solid understanding of ETL processes , data warehousing , and SQL -based tools. Exposure to Hadoop , Kafka , and other big data frameworks. Experience in workflow orchestration using tools like Apache Oozie , Apache Airflow , etc. Proficient in Linux scripting and automation. Soft Skills Strong analytical and troubleshooting abilities. Effective communication skills – verbal and written. Self-starter with the ability to work independently and collaboratively. Detail-oriented with a commitment to delivering high-quality data solutions. What We Offer An opportunity to work with cutting-edge big data technologies. A collaborative team culture focused on innovation and growth. Career advancement and learning opportunities. Competitive compensation and benefits package. Please share the following details to proceed further. Currently Salary: - Expected Salary: - Notice Period: - Reason for looking for change: - Updated Resume: -Please attach. Job Application Disclaimer” We appreciate your interest in this opportunity. Due to the large number of applications, only shortlisted candidates will be contacted for an interview. However, we will keep your resume on file for future opportunities that match your profile. Thanks Vijin.appukuttan@cloverinfotech.com
