Databricks Engineer

As a Data Engineer, you will play a crucial role in designing, developing, and deploying data pipelines using Databricks. Your responsibilities will include: - Designing, developing, and deploying data pipelines using Databricks, focusing on data ingestion, transformation, and loading (ETL) processes. - Maintaining high-quality, scalable, and maintainable Databricks notebooks using Python. - Working with Delta Lake and other advanced features to enhance data processes. - Leveraging Unity Catalog for data governance, access control, and data discovery. - Developing and optimizing data pipelines for improved performance and cost-effectiveness. - Integrating with various data sources such as databases, cloud storage (Azure Blob Storage, ADLS, Synapse), and APIs. - Utilizing Parquet files for efficient data storage and processing. - Working on data integration from Azure Data Factory, Azure Data Lake, and other relevant Azure services. - Performing data quality checks and validation to uphold data accuracy and integrity. - Troubleshooting and resolving data pipeline issues efficiently. - Collaborating with data analysts, business analysts, and stakeholders to understand their data requirements and provide technical solutions. - Participating in code reviews and contributing to best practices within the team. If you have any additional details about the company, please provide them for a more comprehensive job description.,