Data Engineer

As a Senior Data Engineer, you will be responsible for designing and implementing high-performance data pipelines to ensure the seamless flow of large datasets across systems. Leveraging your expertise in Databricks, Pyspark, and ETL design, you will play a key role in building efficient data solutions. Moreover, you will contribute to optimizing database performance and ensuring that data systems meet scalability and security standards. Key Responsibilities: - Develop, optimize, and maintain scalable ETL pipelines for processing large datasets using Databricks and Pyspark. - Implement advanced techniques such as chunking and partitioning to handle large file volumes efficiently. - Tune and optimize databases to enhance performance and storage efficiency. - Collaborate with cross-functional teams to ensure that the architecture aligns with business requirements and data quality standards. - Design and implement solutions with scalability and long-term performance in mind. - Utilize Azure services, with exposure to Azure Function Apps, Azure Data Factory, and Cosmos DB (preferred). - Communicate effectively with stakeholders to align on project goals and deliverables. Qualifications Required: - Advanced-level skills in Databricks for data engineering workflows. - Intermediate-level skills in Pyspark for processing big data. - Strong ETL design skills, particularly in partitioning, chunking, and database tuning for large datasets. - Azure experience is a plus, including Function Apps, Cosmos DB, and Azure Data Factory. - Excellent communication skills for effective collaboration with teams and stakeholders.,