Senior Data Engineer - Databricks

While technology is the heart of our business, a global and diverse culture is the heart of our success. We love our people and we take pride in catering them to a culture built on transparency, diversity, integrity, learning and growth.
If working in an environment that encourages you to innovate and excel, not just in professional but personal life, interests you- you would enjoy your career with Quantiphi!

Role: Senior Data Engineer (Databricks)

Experience Level: 3 to 6 Years

Work location: Mumbai, Bengaluru, Trivandrum

Role & Responsibilities:

• Design and implement data solutions (ETL pipelines, Processing and Transformation logic) using Snowflake as a key platform.

• Design Virtual warehouses which are optimized for performance and cost (auto scaling, idle time etc)

• Write SQL , stored procedures to implement business logic, other critical requirements like encryption, data masking etc

• Implement solutions with features like Snowpipe, tasks, streams,dynamic tables etc

• Design the solutions by leveraging key concepts around the architecture of Snowflake, its data distribution and partitioning mechanism, data caching etc.

• Identify queries which take more time , look at the logs etc and provide solutions to optimize those queries

Must have skills:

• 3+ years experience of Hands-on in data structures, distributed systems, Spark, SQL, PySpark, NoSQL Databases

• Strong software development skills in at least one of: Python, Pyspark or Scala.

• Develop and maintain scalable & modular data pipelines using databricks & Apache Spark

• Experience building and supporting large-scale systems in a production environment

• Experience working on either of the cloud storages - AWS S3, Azure data lake, GCS buckets

• Exposure to the ELT tools offered by the cloud platforms like, ADF, AWS Glue, Google dataflow

• Integrate Databricks with other cloud services like AWS, Azure or Google Cloud

• Implement and optimize Spark jobs, data transformations, and data processing workflows in Databricks

• Managing and configuring databricks environments, including clusters and notebooks

• Experience working with different file formats - parquet, databricks delta lake

• Experience working with data governance on databricks, implementing unity catalog, delta share