Senior Data Engineer

Role & responsibilities

• Design, develop and operate high performance, large volume data structures for data-powered products and data science.
• Implement efficient, distributed and scalable pipelines and integrate data from multiple sources to common data models
• Implement design patterns that support data ingestion, data movement, transformation, aggregation, and much more.
• Build and automate ML workflows using Azure, ML Flow and Databricks environment for experimentation tracking, model performance and operationalization
• Convert machine learning models into Spark ML programs to run efficiently on distributed compute clusters.
• Collaborate with Product managers, Software engineers, and Data Scientists, and work towards achieving key results.
• Build high quality production level code, test and deploy data pipelines
• Undertake complex tasks that are large and diverse in scope and/or critical in nature
• Design and develop breakthrough products, services or technological advancements that expand our business
• Participate in code reviews, brainstorm design approaches with peers and mentor junior engineers
• Work in fast paced agile environment, participate in scrum ceremonies and help ensure that the team meets sprint commitments

Preferred candidate profile

• You have a degree in Computer Science, Engineering, or a related field and/or have 6+ years of software and/or data engineering experience
• Experience with big data technologies such as Spark, Databricks, Delta lakes, Hive, etc.
• Experience in productionizing Machine Learning models and Machine Learning Operations (MLOps), a big plus
• 4+ years of experience in core languages such as Python, Scala or Java (Preferably Python)
• Hands-on experience leveraging PaaS offerings in a public cloud environment (Azure preferred)
• Experience with Big Data design patterns and distributed computing tools/frameworks
• Proven expertise building big data pipelines (batch processing, real-time streaming)
• Working knowledge of TDD, CI/CD concepts and tools (Preferably Azure DevOps)
• Advanced understanding of unit testing/integration testing/QA/Validation
• Knowledge of data security and authentication protocols
• Experience in areas of data governance, privacy and regulation and professional experience with architectural approaches to data security
• Have experience integrating multiple data sources into a common set of data assets or common data models