Data Engineer (Python, Pyspark, AWS Services, data pipelines,data transformation, data quality & Spark SQL & DataFrame)

Must Have:

• Must have 6+Years of experience
• Programming Language: Python
• AWS Services: IAM, Networking, Lake Formation, EMR, MWAA, Athena, Glue ETL, Glue Catalog, S3, Lambda, API Gateway, Event Bridge, SQS, SNS, Kinesis, DMS
• Automation: Terraform (IaC), GitHub Actions
• UI Tech (for a few resources): Angular
• Architecture: Data Mesh, Micro Services

Strong Experience in PySpark:

• Hands-on expertise in building scalable

  data pipelines

  using

  PySpark.

• Proficiency in using

  Spark SQL, DataFrame, and RDD APIs t

  o implement complex business logic.

Proficient Programming Skills:

• Solid coding skills in

  Python

  (preferred), with strong fundamentals in

  data structures, algorithms,

  and software engineering principles.

Data Pipeline Development:

• Proven experience designing, developing, and maintaining batch and streaming data pipelines.
• Understanding of

  ETL/ELT

  processes and best practices for

  data transformation, data quality

  , and performance optimization.

Knowledge of Modern Data Engineering Ecosystem:

• Familiarity with the current data engineering landscape including distributed data processing, storage systems, and workflow orchestration tools.
• Tools/technologies could include Apache Airflow, dbt, Delta Lake, etc.

Cloud Data Platform Experience (Preferably AWS)

• Experience working with cloud services such as:

• AWS S3 (data lake)

• AWS Glue / EMR (for Spark workloads)

• AWS Lambda, Step Functions, or similar for orchestration/integration

• Redshift or other cloud data warehouses

Spark API Expertise for Business Logic Implementation

• Ability to choose and apply the right

  Spark APIs (Data Frame, Dataset, RDD)

  for performance-efficient implementation of business logic at scale.