LEAD DATA ENGINEER - Azure Databricks

Years of Experience:

Job Summary:

Responsibilities:

• Design, develop, and maintain scalable data pipelines using Azure Databricks and Azure Data Factory.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions.
• Implement data integration processes to ingest, transform, and store data from various sources into data lakes and warehouses.
• Optimize data workflows and ensure data quality, reliability, and performance.
• Utilize Palantir for data analysis and visualization, providing actionable insights to the business.
• Monitor and troubleshoot data pipeline issues, ensuring timely resolution and minimal downtime.
• Stay updated with the latest industry trends and technologies related to data engineering and cloud computing.
• Mentor junior data engineers and contribute to team knowledge sharing.

Mandatory Skills:

• Proven experience with Azure Databricks (deal breaker skill).
• Strong proficiency in Azure Data Factory for data integration and orchestration.
• Experience with Palantir for data analysis and visualization.
• Solid understanding of data modeling, ETL processes, and data warehousing concepts.
• Proficient in programming languages such as Python, SQL, or Scala.
• Good understanding of Generative AI and Machine Learning concepts, including LLM, RAG, Vector Databases, Embeddings, Token Management, Feature Selection, Model Evaluation, and Neural Network fundamentals

• Good to have

  : Ability to demonstrate practical knowledge of ML algorithms such as KNN, Linear Regression, CNN

Preferred Skills:

• Experience with cloud platforms such as Azure, AWS, or Google Cloud.
• Familiarity with big data technologies like Apache Spark, Hadoop, or Kafka.
• Knowledge of data governance and data security best practices.
• Experience with CI/CD pipelines and DevOps practices in data engineering.

Qualifications:

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• 5-8 years of experience in data engineering or a related role.
• Strong analytical and problem-solving skills.
• Excellent communication and collaboration abilities.

If you are passionate about data engineering and want to work in a fast-paced environment, we encourage you to apply for this exciting opportunity.