Databricks Engineer

As a Data Engineer, you will be responsible for the following: - Designing, developing, and deploying data pipelines using Databricks, which includes data ingestion, transformation, and loading (ETL) processes. - Developing and maintaining high-quality, scalable, and maintainable Databricks notebooks using Python. - Working with Delta Lake and other advanced features to ensure efficient data processing. - Leveraging Unity Catalog for data governance, access control, and data discovery. - Developing and optimizing data pipelines for improved performance and cost-effectiveness. - Integrating with various data sources such as databases, cloud storage (Azure Blob Storage, ADLS, Synapse), and APIs. - Utilizing Parquet files for data storage and processing. - Having experience with data integration from Azure Data Factory, Azure Data Lake, and other relevant Azure services. - Performing data quality checks and validation to maintain data accuracy and integrity. - Troubleshooting and resolving data pipeline issues effectively to ensure smooth operations. - Collaborating with data analysts, business analysts, and business stakeholders to understand their data requirements and provide technical solutions. - Participating in code reviews and contributing to best practices within the team. Please note that additional details about the company were not provided in the job description.,