Senior Analyst

Role Overview: You will be responsible for designing, developing, and optimizing data pipelines using PySpark and Spark SQL to support large-scale analytics and data integration initiatives. Additionally, you will work with large datasets in Hadoop, Databricks, or AWS/GCP/Azure environments, and collaborate with various teams to define data requirements and integrate data from different sources. Key Responsibilities: - Design, develop, and maintain scalable ETL pipelines using PySpark and Spark SQL. - Work with large datasets in Hadoop, Databricks, or AWS/GCP/Azure environments. - Optimize Spark jobs for performance and reliability. - Collaborate with data engineers, analysts, and business teams to define data requirements. - Integrate data from various sources (structured and unstructured). - Implement best practices for data governance, quality, and security. Qualifications Required: - Strong proficiency in PySpark, Python, and SQL. - Hands-on experience with Big Data ecosystems (HDFS, Hive, Kafka, etc.). - Experience with cloud platforms (AWS EMR, Azure Databricks, or GCP DataProc). - Solid understanding of data modeling, ETL design, and performance tuning. - Familiarity with CI/CD pipelines and version control (Git). Company Details: This section is not included in the provided job description.,