Senior Data Engineer

Senior Data Engineer:

Incedo is a US-based consulting, data science and technology services firm

work with industry leading client organizations, deep technology and domain experts, and global teams

Location:

Experience:

Notice Period:

Role Description:

We are seeking a skilled professional to maintain and support batch jobs in a legacy environment. The role involves managing and monitoring ETL processes, addressing issues, and enhancing existing PL/SQL scripts. The ideal candidate will have strong expertise in Informatica, SQL Server, and data warehousing concepts, along with experience in troubleshooting and improving batch job performance.

Key Responsibilities:

• Design and implement robust ETL pipelines using AWS Glue, Redshift, Lambda, and S3.
• Monitor and optimize the performance of data workflows and batch processing jobs.
• Troubleshoot and resolve issues related to data pipeline failures, inconsistencies, and performance bottlenecks.
• Collaborate with cross-functional teams to define data requirements and ensure data quality and accuracy.
• Develop and maintain automated solutions for data transformation, migration, and integration tasks.
• Implement best practices for data security, data governance, and compliance within AWS environments.
• Continuously improve and optimize AWS Glue jobs, Lambda functions, and S3 storage management.
• Maintain comprehensive documentation for data pipeline architecture, job schedules, and issue resolution processes.

Required Skills and Experience:

• Strong experience with Data Engineering practices.
• Experience in AWS services, particularly AWS Redshift, Glue, Lambda, S3, and other AWS data tools.
• Proficiency in SQL, python , Pyspark, numpy etc and experience in working with large-scale data sets.
• Experience in designing and implementing ETL pipelines in cloud environments.
• Expertise in troubleshooting and optimizing data processing workflows.
• Familiarity with data warehousing concepts and cloud-native data architecture.
• Knowledge of automation and orchestration tools in a cloud-based environment.
• Strong problem-solving skills and the ability to debug and improve the performance of data jobs.
• Excellent communication skills and the ability to work collaboratively with cross-functional teams.
• Good to have knowledge of DBT & Snowflake

Preferred Qualifications:

• Bachelor’s degree in Computer Science, Information Technology, Data Engineering, or a related field.
• Experience with other AWS data services like Redshift, Athena, or Kinesis.
• Familiarity with Python or other scripting languages for data engineering tasks.
• Experience with containerization and orchestration tools like Docker or Kubernetes.

We are an Equal Opportunity Employer:

We value diversity at Incedo. We do not discriminate based on race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.