Data Engineer

Client C

Data Engineer (AWS Integration)

Key Responsibilities:

Data Lake Architecture:

-Design, implement, and maintain a data lake on AWS using Iceberg tables on S3, ensuring data consistency, high availability, and easy querying.

-Work with AWS Glue for ETL processes, creating Glue jobs to process, transform, and load data from SQL Server and other sources into Iceberg tables.

SQL Server to AWS Migration:

-Lead the migration of on-premises SQL Server data to AWS S3 using AWS Glue, DMS, and other relevant tools.

-Design and implement CDC (Change Data Capture) pipelines to enable real-time data replication from SQL Server to AWS services like S3, Iceberg, and Redshift.

Data Integration & Management:

-Design and implement data pipelines for batch and real-time data processing, leveraging AWS DMS, Kinesis, and Glue Streaming.

-Automate data loading, transformation, and archiving processes with Glue and ensure the correct data schema management and data governance.

-Set up and maintain data cataloging systems using AWS Glue Data Catalog to organize and manage metadata for Iceberg tables.

Querying with Athena:

-Use AWS Athena for querying Iceberg tables on S3, enabling high-performance, low-latency SQL querying on large datasets stored in Iceberg format.

-Optimize Athena queries to improve performance and manage large-scale data sets effectively.

-Design efficient partitioning strategies and manage the Iceberg tables using Athena's native support for Iceberg.

Continuous Improvement & Optimization:

-Implement data compaction strategies to reduce small file issues in Iceberg and optimize storage and query performance.

-Continuously monitor, troubleshoot, and optimize data pipelines for performance, cost efficiency, and data quality.

Collaboration & Documentation:

-Work closely with data scientists, analysts, and business teams to understand data needs and develop solutions that support their goals.

-Provide clear documentation on architecture, solutions, and workflows to ensure smooth knowledge transfer and maintainability.

Required Skills & Qualifications:

Experience:

-Proven experience (3+ years) in data engineering roles with expertise in SQL Server, AWS services, and Iceberg.

AWS Glue, AWS DMS, S3, and Kinesis

-Experience with Apache Iceberg for building data lakes and managing large-scale, versioned data on S3.

-Strong knowledge of ETL processes, data modeling, and data transformation using Spark and Glue.

-AWS Athena experience for querying and managing Iceberg tables in S3.

Technical Skills:

-SQL Server: Expertise in database administration, SQL queries, stored procedures, and performance tuning.

-AWS Glue: Experience in creating, debugging, and deploying Glue jobs and managing Glue Data Catalog.

-AWS DMS: Experience with migrating and replicating data from SQL Server to AWS cloud services, including CDC.

-Apache Iceberg: Practical knowledge of Iceberg table formats, partitioning, compaction, and schema evolution.

-AWS S3: Experience managing large datasets in S3, implementing partitioning strategies, and optimizing storage.

-Kinesis: Familiarity with real-time streaming data pipelines using Kinesis Data Streams and Glue Streaming.

-AWS Athena: Strong experience querying Iceberg tables and optimizing large-scale data queries.

Tools & Technologies:

-Apache Spark (for Glue and standalone Spark jobs)

-SQL, Python, AWS SDK for automation, and scripting tasks.

-Data governance: Experience with AWS Lake Formation for security and access management in a data lake environment.

Desired Soft Skills:

-Strong problem-solving skills with the ability to troubleshoot complex data issues.

-Ability to collaborate effectively with cross-functional teams (data scientists, analysts, DevOps, and stakeholders).

-Strong communication skills to document and explain technical concepts clearly to both technical and non-technical audiences.

-Detail-oriented with a focus on data quality, integrity, and performance.

Experience:

Work Timings:

Work Mode:

Salary:

About HR Ways: