Azure Data Engineer (Databricks Specialist)

As an Azure Data Engineer (Databricks Specialist) at CrossAsyst, you will be part of our high-impact Data & AI team, working on critical client-facing projects. With over 5 years of experience, you will utilize your expertise in Azure data services and Databricks to build robust, scalable data pipelines and drive technology innovation for our clients in Pune. Your key responsibilities will include designing, developing, and deploying end-to-end data pipelines using Azure Databricks, Data Factory, and Synapse. You will be responsible for data ingestion, transformation, and wrangling from various sources, optimizing Spark jobs and Databricks notebooks for performance and cost-efficiency. Implementing DevOps best practices for CI/CD, Git integration, and automated testing will be essential in your role. Collaborating with cross-functional teams such as data scientists, architects, and stakeholders, you will design scalable data lakehouse and data warehouse solutions using Delta Lake and Synapse. Ensuring data security, access control, and compliance using Azure-native governance tools will also be a part of your responsibilities. Additionally, you will work closely with data science teams for feature engineering and machine learning workflows within Databricks. Your proactive mindset and strong coding ability in PySpark will be crucial in writing efficient SQL and PySpark code for analytics and transformation tasks. It will also be essential to proactively monitor and troubleshoot data pipelines in production environments. In this role, documenting solution architectures, workflows, and data lineage will contribute to the successful delivery of scalable, secure, and high-performance data solutions. If you are looking to make an impact by driving technology innovation and delivering better and faster outcomes, we welcome you to join our team at CrossAsyst.,