Data Engineer

As a Data Engineer, you will be responsible for designing, developing, and maintaining robust data pipelines using Azure Data Factory and Databricks workflows. Your role will involve implementing and managing big data solutions using Azure Databricks and designing and maintaining relational databases using Azure Delta Lake. Ensuring data quality and integrity through rigorous testing and validation will be a key aspect of your responsibilities. You will be tasked with monitoring and troubleshooting data pipelines and workflows to ensure seamless operation. Implementing data security and compliance measures in accordance with industry standards will also be part of your job. Your role will involve continuously improving data infrastructure, including CI/CD practices, to enhance scalability and performance. Designing, developing, and maintaining ETL processes to extract, transform, and load data from various sources into Snowflake will be a crucial part of your daily tasks. You will utilize ETL tools such as ADF and Talend to automate and manage data workflows effectively. Developing and maintaining CI/CD pipelines using GitHub and Jenkins for the automated deployment of data models and ETL processes will also be within your scope. Collaborating with stakeholders to understand data requirements and ensuring data integrity will be essential. You will integrate multiple data sources to create data lake/data mart and perform data ingestion and ETL processes using SQL, Scoop, Spark, or Hive. Additionally, monitoring job performances, managing file system/disk-space, cluster & database connectivity, log files, backup/security, and troubleshooting various user issues will be part of your responsibilities. Your role will involve designing, implementing, testing, and documenting performance benchmarking strategies for platforms and different use cases. You will set up, administer, monitor, tune, optimize, and govern large-scale implementations. Driving customer communication during critical events and participating/leading various operational improvement initiatives will also be expected from you.,