Data Engineer – Databricks & Spark

Senior Data Engineer–Big Data & Analytics

Location:

Tech Stack:

Desired Experience:

About the Project:

Join an innovative project focused on revolutionizing how companies manage and optimize their promotional activities. This data-driven solution enables businesses to analyze the effectiveness of promotions, plan more impactful campaigns, and integrate seamlessly with existing ecosystems. Leveraging machine learning and advanced analytics, the platform enhances vendor collaboration, improves deal negotiation, and maximizes return on investment (ROI).

Role:

Senior Data Engineer

Responsibilities:

• Design, build, and optimize scalable data pipelines using

  Databricks, SQL, and Apache Spark

• Develop and maintain robust ETL processes for large-scale data systems
• Collaborate with cross-functional teams to understand business needs and translate them into data solutions
• Ensure high performance and quality of data workflows
• Support machine learning and analytics teams with curated, reliable datasets
• Implement best practices in data engineering, testing, and documentation

Qualifications:

• Deep expertise in

  Databricks, Apache Spark, and SQL

• 3–8 years of hands-on experience in designing and maintaining scalable ETL pipelines, with a strong foundation in data engineering best practices and cloud platforms
• Strong experience with

  data modeling

  and

  ETL pipeline development

• Proficient in

  Python

  for data manipulation and scripting
• Hands-on experience with

  AWS

  (especially

  S3

  and

  ECS

  )
• Comfortable working in a

  Linux/Ubuntu

  environment
• Familiarity with dbt (data build tool) for managing data transformations
• Prior experience working in promotional optimization or the retail analytics domain
• Strong problem-solving skills, preferring logical and innovative solutions over legacy experience
• Excellent communication skills with the ability to clarify requirements directly with stakeholders
• Willingness to engage in pair programming and collaborative problem-solving
• Proactive, self-driven, and able to work independently in a distributed team environment