PySpark Data Engineer

Role Overview: You are a skilled PySpark Data Engineer who will be joining a dynamic team at Viraaj HR Solutions in India. Your role will involve designing and implementing robust data pipelines, optimizing data processing workflows, and collaborating with data scientists and analysts. Your strong analytical skills and ability to work in a fast-paced environment will be crucial for this position. Role Responsibilities: - Design, develop, and maintain robust data pipelines using PySpark. - Implement ETL processes for data extraction, transformation, and loading. - Work with structured and unstructured data sources to ensure data quality. - Optimize data processing workflows for better performance and efficiency. - Collaborate with data scientists and analysts to understand data requirements. - Create and manage data warehousing solutions for end-user access. - Integrate data from various sources into a unified platform. - Monitor and troubleshoot data pipeline performance issues. - Conduct data validation and cleansing to maintain data accuracy. - Document processes and workflows for team transparency. - Stay updated on new technologies and trends in data engineering. - Participate in code reviews and ensure best practices in data engineering. - Assist in technical designs and architecture for data solutions. - Provide technical support and training to junior staff. - Contribute to strategic discussions around data architecture and management. Qualifications: - Bachelor's degree in Computer Science, Engineering, or related field. - Proven experience as a Data Engineer or similar role. - Strong proficiency in PySpark and the Spark ecosystem. - Hands-on experience with SQL and database management systems. - Familiarity with Hadoop and big data technologies. - Experience with cloud platforms like AWS, Azure, or Google Cloud. - Understanding of data modeling and database design concepts. - Proficient in Python for data manipulation and analysis. - Strong analytical and troubleshooting skills. - Excellent communication and teamwork abilities. - Knowledge of data governance and compliance practices. - Experience in implementing ETL and data warehousing solutions. - Able to work in a fast-paced environment and manage multiple priorities. - Familiarity with version control systems (e.g., Git). - Willingness to learn and adapt to new technologies. This is an exciting opportunity to work with a forward-thinking company that values expertise and innovation in the field of data engineering. If you are passionate about data and eager to contribute to transformative projects, we encourage you to apply today.,