Data Engineer

Greetings from TCS!

Job Title: Azure Data Engineer

Required Skillset:

Location: PAN INDIA

Experience Range: 5-14 years

Must Have:

• Experience: More than 4 years in data integration, pipeline development, and data warehousing, with a strong focus on AWS/AZURE Databricks.
• Proficiency in Databricks platform, management, and optimization.
• Strong experience in AWS/AZURE Cloud, particularly in data engineering and administration, with expertise in Apache Spark/Azure Synapse Analytics Spark pool, S3, Athena, Glue/AZURE Data factory, Kafka, Lambda/Azure functions, Redshift, and RDS.
• Proven experience in data engineering performance tuning and analytical understanding in business and program contexts.
• Solid experience in Python development, specifically in pySpark within the AWS/AZURE Cloud environment, including experience with Terraform.
• Knowledge of databases (Oracle, SQL Server, PostgreSQL, Redshift, MySQL, or similar) and advanced database querying.
• Experience with source control systems (Git, Bitbucket) and Jenkins for build and continuous integration.
• Understanding of continuous deployment (CI/CD) processes.
• Experience with Airflow and additional Apache Spark knowledge is advantageous.
• Exposure to ETL tools, including Informatica.

Good To Have:

• Experience with Airflow and additional Apache Spark knowledge is advantageous.
• Exposure to ETL tools, including Informatica.

Key Responsibilities:

• Administer, manage, and optimize the Databricks environment to ensure efficient data processing and pipeline development.
• Perform advanced troubleshooting, query optimization, and performance tuning in a Databricks environment.
• Collaborate with development teams to guide, optimize, and refine data solutions within the Databricks ecosystem.
• Ensure high performance in data handling and processing, including the optimization of Databricks jobs and clusters.
• Engage with and support business teams to deliver data and analytics projects effectively.

Thanks & Regards,

Ria Aarthi A.