Job
Description
We are seeking a skilled and experienced Data Engineer Lead to join our team. The ideal candidate will have expertise in Apache Spark, PySpark, Python , and AWS services (particularly AWS Glue ). You will be responsible for designing, building, and optimizing ETL processes and data workflows in the cloud, specifically on the AWS platform . Your work will focus on leveraging Spark-based frameworks, Python, and AWS services to efficiently process and manage large datasets. Experience Range - 5 to 7 years Key Responsibilities: Spark & PySpark Development : Design and implement scalable data processing pipelines using Apache Spark and PySpark to support large-scale data transformations. ETL Pipeline Development : Build, maintain, and optimize ETL processes for seamless data extraction, transformation, and loading across various data sources and destinations. AWS Glue Integration : Utilize AWS Glue to create, run, and monitor serverless ETL jobs for data transformations and integrations in the cloud. Python Scripting : Develop efficient, reusable Python scripts to support data manipulation, analysis, and transformation within the Spark and Glue environments. Data Pipeline Optimization : Ensure that all data workflows are optimized for performance, scalability , and cost-efficiency on the AWS Cloud platform. Collaboration : Work closely with data analysts , data scientists , and other engineering teams to create reliable data solutions that support business analytics and decision-making . Documentation & Best Practices : Maintain clear documentation of processes, workflows, and code while adhering to best practices in data engineering , cloud architecture , and ETL design . Required Skills: Expertise in Apache Spark and PySpark for large-scale data processing and transformation. Hands-on experience with AWS Glue for building and managing ETL workflows in the cloud. Strong programming skills in Python , with experience in data manipulation, automation, and integration with Spark and Glue. In-depth knowledge of ETL principles and data pipeline design, including optimization techniques. Proficiency in working with AWS services , such as S3 , Glue , Lambda , and Redshift . Strong skills in writing optimized SQL queries , with a focus on performance tuning. Ability to translate complex business requirements into practical technical solutions . Familiarity with Apache Airflow for orchestrating data workflows. Knowledge of data warehousing concepts and cloud-native analytics tools. Required Skills Aws Glue,Pyspark,Python.
