Specialist - Data

As a Senior Data Engineer, you will be an integral part of our team, contributing to the design, implementation, and maintenance of scalable data pipelines utilizing cutting-edge tools like Airflow, PySpark, and Databricks. Your responsibilities will extend to supporting machine learning models in a production environment, crafting data products from vast datasets, and engaging with diverse teams to grasp data requirements effectively. Your primary duties will include designing and deploying robust data pipelines by leveraging Airflow, PySpark, and Databricks. You will also be tasked with ensuring the reliability and performance of machine learning models in production, creating data products that align with business objectives, and collaborating closely with cross-functional teams to address their data-related needs. Furthermore, you will play a key role in implementing process monitoring mechanisms to uphold data quality and reliability, including support for ACID transactions. Additionally, you will be expected to provide technical guidance and assistance to your colleagues whenever necessary. To qualify for this role, you should possess a Bachelor's or Master's degree in Computer Science, Engineering, or a related field, along with a minimum of 5 years of experience in a data engineering capacity. Your background should demonstrate a successful history of delivering scalable and dependable data solutions. Proficiency in Databricks, AWS, and Airflow ETL architectures, particularly in managing data streams from real-time consumer applications, is essential. Moreover, you should exhibit a strong command of Python, PySpark, and Spark SQL for data processing and analysis, coupled with a solid understanding of SQL databases and data modeling principles. Familiarity with Databricks Lakeview and other data visualization tools for monitoring and reporting purposes will be advantageous. Your ability to work autonomously and collaboratively within a team environment, combined with exceptional problem-solving and communication skills, will be critical for excelling in this role.,