Pyspark Developer

As a PySpark Developer in Bhopal, with 5+ years of experience, your primary responsibilities will include: - Developing and maintaining PySpark pipelines in Azure Databricks for both batch and streaming data processing. - Implementing ETL/ELT workflows by integrating Azure Data Lake, Synapse, and Delta Lake. - Writing efficient Python and SQL code for various data operations like transformations, joins, aggregations, and validations. - Optimising Spark jobs through techniques like partitioning, caching, and tuning to enhance performance and reduce costs. - Integrating pipelines into Azure Data Factory and ensuring CI/CD deployment using Git/DevOps. - Applying monitoring, logging, and documentation standards to ensure the reliability of production pipelines. Your key technical and delivery skills should include: - 5+ years of hands-on experience in PySpark development within the Azure ecosystem. - Proficiency in Azure Databricks (PySpark, Delta Lake, Spark SQL), Azure Data Factory for orchestration, and Azure Synapse Analytics for data integration and consumption. - Experience working with Delta Lake/Parquet for scalable storage and analytics. - Strong debugging and optimization skills for Spark jobs to improve performance and cost efficiency. - Familiarity with CI/CD practices using Azure DevOps, Git versioning, and Agile delivery methodologies. - Demonstrated success in delivering production-grade pipelines in enterprise environments. This is a full-time position based in Bhopal with in-person work location requirements.,