Data Engineer

Job Summary:

We are looking for an experienced Azure Databricks Engineer with expertise in SQL, data migration, and on-premises to cloud transition. The ideal candidate will work on designing, developing, and deploying data solutions using Azure Databricks in combination with SQL-based querying. You will play a critical role in migrating data and workloads from on-premises to Azure cloud environments.

Key Responsibilities:

• Data Engineering with Azure Databricks:
• Design, develop, and manage data pipelines and ETL processes in Azure Databricks.
• Implement and optimize performance of Spark-based applications.
• Write optimized SQL queries for querying data and performing analytics.
• Develop and deploy interactive notebooks in Azure Databricks using SQL and PySpark.
• Cloud Migration (On-premises to Cloud):
• Lead the effort in migrating data from on-premises SQL servers and databases to Azure Cloud solutions.
• Implement data integration strategies for seamless data migration from on-prem systems (SQL Server, Oracle, etc.) to Azure SQL databases and Data Lakes.
• Assist with troubleshooting and resolving migration issues, ensuring minimal downtime and data loss.
• SQL Querying and Data Transformation:
• Use SQL-based solutions to transform, aggregate, and join large datasets from various sources.
• Design and develop SQL queries for data extraction, transformation, and loading into the cloud platform.
• Implement best practices for managing data quality, ensuring accurate and consistent data post-migration.
• Collaboration and Support:
• Work closely with cross-functional teams to understand data requirements and implement the best cloud-based solutions.
• Provide ongoing support and training to team members on Databricks and migration strategies.
• Continuously monitor the performance of Databricks jobs and SQL queries, identifying and resolving any performance bottlenecks.

Required Skills and Experience:

• Proven Experience in Azure Databricks: Hands-on experience with Azure Databricks (using Spark, SQL, and Python).
• SQL Proficiency: Strong command over SQL, including advanced queries, optimization techniques, and working with large datasets.
• On-premises to Cloud Migration: Experience migrating data and workloads from on-premises environments to Azure Cloud (e.g., Azure SQL Database, Azure Data Lake, Azure Synapse Analytics).
• Data Engineering Skills: Experience designing and implementing ETL pipelines and data workflows in Azure Cloud environments.
• Cloud Data Platforms: Familiarity with other Azure tools like Azure Data Factory, Azure SQL, Azure Synapse Analytics, and Azure Blob Storage.
• Data Transformation: Expertise in using SQL and Spark for data transformation, data cleaning, and ensuring data integrity.
• Scripting: Experience with Python or Scala for automating processes and manipulating large datasets in Databricks.

Desired Skills:

• Data Warehousing: Experience with Data Warehousing concepts and tools like Azure Synapse.
• DevOps and CI/CD: Familiarity with CI/CD pipelines and automation in a cloud environment.
• Azure Certification: Certifications such as Azure Data Engineer Associate, Azure Fundamentals, or Azure Databricks Certified Associate are a plus.
• Agile Methodology: Comfortable working in an Agile environment with frequent iteration and communication.

Educational Qualifications:

• Bachelors or Master's degree in Computer Science, Information Technology, or a related field, or equivalent experience.