Data Engineer - Optimization Solutions

Job Description

About the Role: We are seeking a highly skilled and passionate Data Engineer to join our growing team dedicated to building and supporting cutting-edge analytical solutions. In this role, you will play a critical part in designing, developing, and maintaining the data infrastructure and pipelines that power our optimization engines. You will work in close collaboration with our team of data scientists who specialize in mathematical optimization techniques. Your expertise in data engineering will be essential in ensuring seamless data flow, enabling the development and deployment of high-impact solutions across various areas of our business. Responsibilities: Design, build, and maintain robust and scalable data pipelines to support the development and deployment of mathematical optimization models. Collaborate closely with data scientists to deeply understand the data requirements for optimization models. This includes: Data preprocessing and cleaning Feature engineering and transformation Data validation and quality assurance Develop and implement comprehensive data quality checks and monitoring systems to guarantee the accuracy and reliability of the data used in our optimization solutions. Optimize data storage and retrieval processes for highly efficient model training and execution. Work effectively with large-scale datasets, leveraging distributed computing frameworks when necessary to handle data volume and complexity. Contribute to the development and maintenance of thorough data documentation and metadata management processes. Stay up to date on the latest industry best practices and emerging technologies in data engineering, particularly in the areas of optimization and machine learning. Qualifications: Education: Bachelor's degree in computer science, Data Engineering, Software Engineering, or a related field is required. Master's degree in a related field is a plus. Experience: 3+ years of demonstrable experience working as a data engineer, specifically focused on building and maintaining complex data pipelines. Proven track record of successfully working with large-scale datasets, ideally in environments utilizing distributed systems. Technical Skills - Essential: Programming: High proficiency in Python is essential. Experience with additional scripting languages (e.g., Bash) is beneficial. Databases: Extensive experience with SQL and relational database systems (PostgreSQL, MySQL, or similar). You should be very comfortable with: Writing complex and efficient SQL queries Understanding performance optimization techniques for databases Applying schema design principles Data Pipelines: Solid understanding and practical experience in building and maintaining data pipelines using modern tools and frameworks. Experience with the following is highly desirable: Workflow management tools like Apache Airflow Data streaming systems like Apache Kafka Cloud Platforms: Hands-on experience working with major cloud computing environments such as AWS, Azure, or GCP. You should have a strong understanding of: Cloud-based data storage solutions (Amazon S3, Azure Blob Storage, Google Cloud Storage) Cloud compute services Cloud-based data warehousing solutions (Amazon Redshift, Google Big Query, Snowflake) Technical Skills - Advantageous (Not Required, But Highly Beneficial): NoSQL Databases: Familiarity with NoSQL databases like MongoDB, Cassandra, and DynamoDB, along with an understanding of their common use cases. Containerization: Understanding of containerization technologies such as Docker and container orchestration platforms like Kubernetes. Infrastructure as Code (IaC): Experience using IaC tools such as Terraform or CloudFormation. Version Control: Proficiency with Git or similar version control systems. Soft Skills: Communication: Excellent verbal and written communication skills. You'll need to effectively explain complex technical concepts to both technical and non-technical audiences. Collaboration: You'll collaborate closely with data scientists and other team members, so strong teamwork and interpersonal skills are essential. Problem-Solving: You should possess a strong ability to diagnose and solve complex technical problems related to data infrastructure and data pipelines. Adaptability: The data engineering landscape is constantly evolving. A successful candidate will be adaptable, eager to learn new technologies, and embrace change. Additional Considerations: Industry Experience: While not a strict requirement, experience working in industries with a focus on optimization, logistics, supply chain management, or similar domains would be highly valuable. Machine Learning Operations (MLOps): Familiarity with MLOps concepts and tools is increasingly important for data engineers in machine learning-focused environments.