Data Engineer - AI | ML

Data Engineer

We're seeking a skilled Data Engineer to design, build, and maintain scalable data pipelines and infrastructure that enable data-driven decision making across the organization.

Key Responsibilities:

• Design and implement robust ETL/ELT pipelines for data ingestion, transformation, and loading
• Build and maintain data orchestration workflows using tools like Apache Airflow, NiFi, or similar platforms
• Perform comprehensive data cleansing, validation, and quality assurance processes
• Implement data security best practices including encryption, access controls, and compliance measures
• Configure and manage authentication systems and identity management solutions
• Monitor system performance, logs, and observability metrics across data infrastructure
• Develop and optimize data models for analytics and reporting needs
• Troubleshoot data flow issues and maintain distributed system coordination

Required Skills & Experience:

• 3+ years of experience in data engineering or related field
• Strong proficiency in Python, SQL, and at least one other programming language (Java, Scala, or Go)
• Hands-on experience with Apache NiFi for data flow automation and management
• Expert knowledge of ETL processes and data transformation techniques
• Experience with data orchestration tools (Apache Airflow, Prefect, or Dagster)
• Solid understanding of data security principles and implementation
• Experience with authentication and identity management systems (Keycloak, LDAP, OAuth)
• Proficiency with logging and observability tools (Graylog, SigNoz, ELK stack)
• Knowledge of distributed system coordination (Apache Zookeeper, Consul)
• Proficiency with open source frameworks: Apache Spark, Kafka, Hadoop ecosystem, dbt
• Experience with both relational and NoSQL databases
• Knowledge of containerization (Docker, Kubernetes) and infrastructure as code
• Familiarity with cloud platforms (AWS, Azure, GCP) alongside open source alternatives
• Experience with webscraping complex data structures

Preferred:

• Experience with real-time streaming data processing
• Knowledge of data governance and lineage tools
• Background in data quality frameworks and monitoring systems
• Experience with service mesh and API gateway implementations

We value candidates who demonstrate expertise across both enterprise and open source technologies, bringing flexibility and cost-effective solutions to our data infrastructure while maintaining robust security and observability practices.