Data Engineer

You will join the team as a Data Engineer, responsible for building and maintaining data pipelines, as well as developing data visualizations that drive company-wide analytics.

A strong understanding of good API design, data pipelining, and a commitment to data security is essential for this role. Proficiency in SQL is a must, as you'll regularly work with complex queries, data transformations, and optimizations to support scalable, high-performance data workflows.

Responsibilities:
Data Pipeline Development

• Design and implement scalable, production-grade data pipelines using PySpark and Python
• Develop ETL/ELT workflows within the AWS ecosystem, leveraging services like AWS Glue, Lambda, and Step Functions.
• Ingest and transform data from a variety of sources, including files, APIs, SQL databases (e.g., PostgreSQL, MySQL), NoSQL databases (e.g., DynamoDB, Cassandra) , events and streaming data
• Write robust unit and integration tests to validate pipeline logic and ensure data quality.
• Monitor, optimize, and troubleshoot pipeline performance, with logging and alerting for failures and delays

Data Lake Table Management (Hudi & Iceberg)

• Work with modern table formats like Apache Hudi and Apache Iceberg to enable incremental processing, upserts, and time-travel queries.
• Implement efficient data modelling strategies using these formats to support both batch and streaming data needs
• Optimize table partitioning, compaction, and schema evolution in large-scale data lake environments

Data Visualization & Reporting

• Create impactful dashboards and data visualizations using tools like QuickSight,Power BI, Tableau or similar tools
• Translate complex data into actionable insights for business stakeholders
• Provide support and training to stakeholders on accessing and using analytics tools and data assets

Collaboration & Stakeholder Engagement

• Partner with product, data scientists, and business teams to gather data requirements and deliver integrated solutions
• Translate business logic into efficient data transformations and visual outputs
• Optimize application performance for speed and responsiveness

Data Governance & Infrastructure

• Manage cloud-based data infrastructure (e.g., AWS Glue, Redshift, S3, EMR) ensuring security, reliability, and scalability
• Ensure compliance with data governance policies, privacy regulations, and access control standards
• Maintain proper data documentation, versioning, and lineage tracking.

Deployment & Release Management

• Design, implement, and maintain the build, deployment, and release process for data pipelines using AWS CloudFormation (CFN) or Terraform.
• Collaborate with team members to integrate code changes into CI/CD pipelines and ensure smooth deployment across multiple environments

Debugging, Collaboration & Growth

• Mentor and support junior developers by guiding them on creating and maintaining pipelines.
• Debug and troubleshoot cross-platform issues efficiently
• Collaborate with developers, DevOps, and stakeholders to deliver end-to-end features
• Stay updated on industry trends in the data and generative BI world

Knowledge, Education and Experience:

• Bachelor's or Master's degree in computer science or a related field (preferred)
• 3–5 years of experience Data engineering and pipeline development using Pyspark and python, SQL, No-SQL and understanding of modern development patterns
• Understanding and familiarity of modern concepts like data lake, lake house, open table formats
• Strong knowledge and experience to work with complex SQL queries
• Hands-on with AWS or similar cloud platforms; knowledge of cloud architecture (serverless)
• Experience with Git, CI/CD tools
• Basic experience in Microservice architecture, web services (Restful API's).
• Experience with Redshift will be an added advantage.
• Strong problem-solving, analytical, communication, and collaboration skills
• Familiarity with Agile software development methodologies.