Data Engineer

Overview

Key Responsibilities

• Design, develop, and maintain end-to-end data pipelines using Databricks.
• Work extensively with SQL and Python to support data transformations, processing, and analysis.
• Implement and optimize data workflows across AWS services such as S3, Secrets Manager, and other foundational cloud components.
• Leverage AWS Athena, Glue, and Redshift for data extraction, cataloging, processing, and warehousing.
• Develop and maintain shell scripts for automation and operational workflows.
• Work with PostgreSQL databases for data modeling, querying, and optimization.
• Collaborate with cross-functional teams to understand data requirements and ensure data quality, consistency, and availability.
• Troubleshoot data issues, optimize query performance, and ensure efficient data processing.
• Apply strong data engineering and data understanding skills to solve complex business challenges.
  

Required Skills & Experience

• 4–5 years of experience in Data Engineering.
• Strong hands-on experience with:
• Databricks (end-to-end development)
• SQL (advanced querying, performance tuning)
• Python (data processing, automation)
• Solid knowledge of AWS Cloud services including S3, Secrets Manager, and other basic AWS components.
• Experience with Athena, Glue, and Redshift.
• Basic understanding of Shell scripting.
• Strong data modeling, data pipeline design, and data troubleshooting skills.
• Experience with PostgreSQL.
  

Nice to Have

• Exposure to data governance, data validation, or CI/CD for data pipelines.
• Experience with any orchestration tools (e.g., Airflow, AWS Step Functions).