Big Data Developer

Job Description

Role & responsibilities Develop and optimize big data processing pipelines using PySpark and Python. Work with Unix/Shell scripting for automation and data workflow management. Write and optimize SQL/HQL queries for data extraction and transformation on the CDP platform. Manage and process large datasets efficiently within a Big Data ecosystem. Collaborate with data analysts, data scientists, and engineering teams to ensure seamless data flow. Implement best practices in data processing, ETL, and performance tuning. Troubleshoot and resolve issues in data processing pipelines and platform performance. Must-Have Skills Strong experience in PySpark and Python for big data processing. Hands-on experience in Big Data technologies. Unix/Shell scripting skills for automation and scheduling. Proficiency in SQL/HQL queries on the CDP platform. Good-to-Have Skills Experience with Object Storage (S3) for cloud-based data management. Familiarity with Airflow scheduler for workflow automation. Experience using Query Trino tool for distributed SQL queries. Knowledge of Iceberg analytical tools for large-scale data processing. Preferred candidate profile Bachelor's or Master's in Computer Science, Information Technology, or a related field. 5+ years of experience working in Big Data and Data Engineering. Strong problem-solving skills and ability to work in fast-paced environments Perks and benefits Competitive Salary and Benefits Growth Opportunities Be part of a fast-paced and growing startup. Ownership & Responsibility – You will own initiatives and drive results. Continuous Learning & Mentorship – Work closely with senior data scientists and AI experts. Dynamic & Respectful Work Environment – A culture that values and supports its employees.