Data Engineer

Job Summary:

We are seeking a highly skilled and motivated Data Engineer to design, develop, and optimize our critical data infrastructure. You will play a pivotal role in building scalable batch and real-time data pipelines, ensuring data quality, and collaborating with cross-functional teams to deliver impactful data solutions. If you are passionate about data, possess strong technical skills, and thrive in a collaborative environment, we encourage you to apply.

Responsibilities:

• Design, implement, and optimize robust batch and real-time data pipelines using technologies such as Spark, Python, and Airflow.
• Leverage AWS native tools (e.g., S3, EMR, Glue) effectively to build and manage scalable data solutions.
• Collaborate cross-functionally with data scientists, data analysts, product managers, and engineering teams to gather requirements, deliver high-quality data solutions, and ensure overall project success.
• Lead initiatives focused on enhancing data quality, establishing data governance frameworks, implementing robust security measures, and setting up comprehensive data monitoring systems.
• Evaluate, recommend, and integrate various cloud, big data, and DataOps tools, including SQL/NoSQL databases, streaming systems, workflow orchestration platforms, and observability tools.

Required Qualifications:

• 2+ years of practical experience in data pipeline development, encompassing both batch and streaming processes.
• Advanced knowledge and hands-on experience with Spark, Python, and SQL.
• Proven expertise in designing and implementing ETL/ELT processes.
• Strong practical experience with AWS (specifically EMR, S3, Glue) or other major cloud platforms (e.g., Azure, GCP).
• Solid understanding and experience with data modeling, data warehousing concepts, and metadata management.
• Demonstrated problem-solving abilities, strong analytical skills, and a highly collaborative mindset.

Preferred Qualifications:

• Experience with CI/CD pipelines and MLOps lifecycle.
• Exposure to NoSQL databases, data lake architectures, or streaming technologies (e.g., Kafka, Kinesis).
• Experience utilizing API tools such as Postman and Swagger.
• Awareness and understanding of data security, privacy, and compliance best practices.