Senior Python Pyspark Developer

As a skilled and proactive Python / PySpark Developer at our company, you will join our data engineering or analytics team. Your primary responsibility will be to build scalable data pipelines, perform large-scale data processing, and collaborate with data scientists, analysts, and business stakeholders. Key Responsibilities: - Design, develop, and optimize ETL data pipelines using PySpark on big data platforms (e.g., Hadoop, Databricks, EMR). - Write clean, efficient, and modular code in Python for data processing and integration tasks. - Work with large datasets to extract insights, transform raw data, and ensure data quality. - Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions. - Implement performance tuning and debugging of PySpark jobs. - Monitor and troubleshoot data workflows and batch jobs in production environments. - Document solutions and maintain code repositories (e.g., Git). Required Skills & Qualifications: - Proficient in Python with experience in building data-centric applications. - Strong experience with PySpark and understanding of Spark internals (RDDs, DataFrames, Spark SQL). - Hands-on experience with Hadoop ecosystem, Hive, or cloud-based big data platforms like AWS EMR, Azure Databricks, or GCP DataProc. - Familiarity with workflow orchestration tools like Airflow, Oozie, or Luigi. - Good understanding of SQL and relational databases. - Experience with version control systems like Git. - Strong problem-solving skills and ability to work independently or in a team. - Bachelors degree in Computer Science, Engineering, or a related field. Preferred Qualifications: - Experience with CI/CD pipelines and DevOps practices. - Knowledge of data warehousing and data modeling. - Exposure to streaming technologies Kafka, Spark Streaming. - Familiarity with containerization tools like Docker or Kubernetes. At Virtusa, we embody values like teamwork, quality of life, professional and personal development. When you join our team of 27,000 people globally, you become a part of a community that cares about your growth and provides you with exciting projects and opportunities to work with state-of-the-art technologies throughout your career. We value collaboration, the team environment, and seek to provide a dynamic place for great minds to nurture new ideas and foster excellence.,