Aws Data Engineer

Role Overview

Senior Data Engineer

Key Responsibilities:

• Data Pipeline Development:

  Design, develop, and maintain complex ETL/ELT pipelines to build high-value data assets.

• Legacy Modernization:

  Lead the

  code refactorization

  of legacy codebases, improving readability, maintainability, and performance.

• System Optimization:

  Perform deep code optimization using

  Spark SQL

  and

  PySpark

  to handle large-scale datasets efficiently.

• Quality Assurance:

  Implement a

  Test-Driven Development (TDD)

  approach, writing comprehensive unit tests to ensure functionality and catch bugs early in the lifecycle.

• Complex Problem Solving:

  Isolate and resolve difficult bugs, including those related to performance bottlenecks, concurrency issues, and complex logic flaws.

• Cloud Architecture:

  Design and deploy solutions utilizing the full AWS stack, explaining the trade-offs and benefits of specific services for various use cases.

Technical Requirements:

Core Programming & Data Engineering

• 7+ years of experience

  in hands-on programming with

  Python

  and

  PySpark

  .
• Expertise in

  Boto3

  and various Python frameworks and libraries, adhering strictly to Python best practices (PEP 8).
• Strong experience in

  Spark SQL

  and PySpark optimization techniques (e.g., partitioning, caching, broadcast joins).

Cloud & Infrastructure (AWS)

• Deep architectural knowledge of

  AWS services

  , including:

• S3, EC2, Lambda, Redshift, CloudFormation

DevOps & Tools

• Advanced understanding of

  Git

  (branching strategies, PR reviews).
• Experience with

  JFrog Artifactory

  for dependency management and artifact storage.
• Proficiency in CI/CD pipelines and automated testing frameworks.

Professional Attributes:

• Analytical Mindset:

  Ability to debug complex, non-obvious issues in distributed systems.

• Clean Coder:

  Passion for writing "clean code" and mentoring junior engineers on maintainability.

• Architectural Thinking:

  Ability to explain the "why" behind choosing specific AWS components over others.