Job
Description
Job Overview As a Data Engineer, you will be responsible for designing, developing, and optimizing data pipelines and workflows that handle large volumes of data. You will work closely with our data science and analytics teams to ensure data is readily available for analysis and reporting. This role requires expertise in AWS services, particularly Kinesis, Glue, Lambda, Step Functions, and Redshift. Additionally, the ideal candidate should have strong SQL skills and experience with one of the following programming languages, with a preference for Node.js. Role & responsibilities Design, develop, and maintain data pipelines: Create scalable and efficient data pipelines using AWS services like Kinesis, Glue, Lambda, Step Functions, and Redshift. Data integration: Integrate data from various sources into a centralized data warehouse, ensuring data consistency, quality, and security. ETL processes: Develop and manage ETL processes using AWS Glue and other relevant tools to transform and load data into Redshift or other databases. Real-time data processing: Implement real-time data processing solutions using AWS Kinesis and Lambda to handle streaming data. Automation: Automate data workflows and processes using AWS Step Functions and other orchestration tools. Performance optimization: Optimize SQL queries and database performance for efficient data retrieval and reporting. Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver data solutions that meet their needs. Data quality and governance: Ensure high standards of data quality and implement data governance best practices across all data pipelines. Documentation: Maintain clear and comprehensive data dictionary along with documentation of data pipelines, architectures, and processes. Preferred candidate profile 4+ years of experience in data engineering or a related field. Hands-on experience with AWS services, specifically Kinesis, Glue, Lambda, Step Functions, and Redshift. Expert-level SQL knowledge, with a proven track record of writing complex queries and optimizing database performance. Proficiency in a programming language, like python or scala or Node.js. Strong understanding of ETL processes, data warehousing concepts, and real-time data processing. Experience with data modelling, schema design, and database optimization. Ability to work with large datasets and troubleshoot data issues. Familiarity with data governance, data quality, and security best practices. Excellent problem-solving skills and attention to detail. Strong communication skills, with the ability to explain technical concepts to non-technical stakeholders. Ability to work independently and as part of a team in a fast-paced environment.
