AWS data engineer

As an AWS Data Engineer with over 5 years of experience, you will be working in Chennai (WFO) and will be expected to attend a Face to Face interview on 26 July (Saturday) during IST hours. Your preferred domain should be in Life Sciences / Pharma, which is good to have. Your mandate includes possessing key skill sets in AWS, Python, Databricks, Pyspark, and SQL. Your primary responsibilities will involve designing, building, and maintaining scalable data pipelines for ingesting, processing, and transforming large datasets from diverse sources into usable formats. You will also be responsible for integrating data from multiple sources, ensuring accurate transformation and storage in optimal formats such as Delta Lake, Redshift, and S3. Additionally, you will optimize data processing and storage systems for cost efficiency and high performance while managing compute resources and cluster configurations. Automation of data workflows using tools like Airflow, Databricks APIs, and other orchestration technologies to streamline data ingestion, processing, and reporting tasks will be a crucial part of your role. Implementing data quality checks, validation rules, and transformation logic to guarantee accuracy, consistency, and reliability of data will also be essential. In terms of cloud platform management, you will manage and optimize cloud infrastructure (AWS, Databricks) for data storage, processing, and compute resources, ensuring seamless data operations. Leading migrations from legacy data systems to modern cloud-based platforms and implementing cost optimization strategies will also be part of your responsibilities. Ensuring data security by implementing IAM roles and policies, adhering to data security best practices, and enforcing compliance with organizational standards will be vital. Lastly, collaborating with data scientists, analysts, and business teams to understand data requirements and provide support for data-related tasks will be key to your success in this role.,