Data Engineer II

Job responsibilities

• Design, develop and maintain scalable and large-scale data processing pipelines and infrastructure on the cloud following engineering standards, goverce standards and technology best practices
• Develop and optimize data models for large-scale datasets, ensuring efficient storage, retrieval, and analytics while maintaining data integrity and quality.
• Collaborate with cross-functional teams to translate business requirements into scalable and effective data engineering solutions.
• Demonstrate a passion for innovation and continuous improvement in data engineering, proactively identifying opportunities to enhance data infrastructure, data processing and analytics capabilities.

Required qualifications, capabilities, and skills

• Strong analytical problem solving and critical thinking skills
• Proficiency in at least one programming language ( Python, if not Java or Scala)
• Proficiency in at least one distributed data processing framework ( Spark or similar)
• Proficiency in at least one cloud data Lakehouse platforms (AWS Data lake services or Databricks, alternatively Hadoop),
• Proficiency in at least one scheduling/orchestration tools ( Airflow, if not AWS Step Functions or similar)
• Proficiency with relational and NoSQL databases.
• Proficiency in data structures, data serialization formats (JSON, AVRO, Protobuf, or similar), and big-data storage formats (Parquet, Iceberg, or similar),
• Experience working in teams following Agile methodology
• Experience with test-driven development (TDD) or behavior-driven development (BDD) practices, as well as working with continuous integration and continuous deployment (CI/CD) tools.