Data Engineer- Azure Databricks

Role & responsibilities

Job Title: Data Engineer 6+ years of experience

We are seeking a skilled and motivated Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, with expertise in handling Streaming data and large datasets and building efficient data pipelines. You will play a crucial role in ensuring the availability and reliability of data for analysis

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines to ingest streaming and batch data into Azure data Lake and Azure databricks

• Experience in Azure IoT Hub/ Event Hub / Azure IoT operations and IoT Protocol MQTT, EMQX broker

• Data transformation and data preparation using PySpark /Scala

• Handle large datasets and ensure data quality and governance, unity catalog

• Implement data integration and ETL pipelines, scheduling and troubleshooting.

• Develop and optimize data warehousing solutions and dimensional models.

• Collaborate with cross-functional teams to understand data requirements and deliver solutions.

Skills and Qualifications:

• Expert in Azure data factory and Azure Databricks, Spark, Azure Data Lake and IoT

• Proficiency in programming languages such as Python, Java, or Scala.

• Strong knowledge of SQL and databases.

• Experience with data integration and ETL processes.

• Understanding of data warehousing and dimensional modeling.

• Ability to ensure data quality and governance.

• Proficiency in data visualization tools.

• Strong collaboration and communication skills.

• Excellent problem-solving and analytical thinking abilities.

• Knowledge of version control and DevOps practices.

Note- Active Passport is required.