Lead II - Data Engineering

7 - 9 Years
1 Opening
Trivandrum

Role Proficiency:

This role requires proficiency in developing data pipelines including coding and testing for ingesting wrangling transforming and joining data from various sources. The ideal candidate should be adept in ETL tools like Informatica Glue Databricks and DataProc with strong coding skills in Python PySpark and SQL. This position demands independence and proficiency across various data domains. Expertise in data warehousing solutions such as Snowflake BigQuery Lakehouse and Delta Lake is essential including the ability to calculate processing costs and address performance issues. A solid understanding of DevOps and infrastructure needs is also required.

Outcomes:

Measures of Outcomes:

Outputs Expected:

Code:

Documentation:

Configure:

Test:

Domain Relevance:

Manage Project:

Manage Defects:

Estimate:

Manage Knowledge:

Release:

Design:

Interface with Customer:

Manage Team:

Certifications:

Skill Examples:

Knowledge Examples:

Knowledge Examples

Additional Comments:

Job Title: Databricks and PySpark Developer Role Focus: Design, build, and optimize data pipelines and create notebooks on the Databricks platform. Ensure performance, scalability, and security of Databricks workloads. Perform operations support to deploy and manage Databricks for large-scale data processing. Key Responsibilities: Integrate and manage Databricks with AWS cloud services. Optimize query performance and cluster configurations in Databricks. Document processes, standards, and best practices for Databricks development. Apply strong knowledge of Apache Spark and distributed data processing. Build and maintain scalable data pipelines for both batch and streaming workloads. Perform data ingestion within Databricks to ingest, transform, and deliver data across development, UAT, and Production environments Develop solutions using Python, SQL, PySpark, and Airflow for data engineering and analytics. Work with CI/CD pipelines, DevOps/MLOps practices, and automation tools. Leverage knowledge of PostgreSQL and DB2 for data integration and optimization. Tools & Technologies: AWS ecosystem (EC2, S3, IAM, Lambda, RDS, etc.) Infrastructure as Code: CDK (TypeScript), CloudFormation (YAML) Version Control & Automation: GitHub, Bash/SH scripting Databricks, Python, Spark, Ariflow Snowflake (nice to have) Other Requirements: Strong awareness of data security practices and compliance standards. Ability to maintain data quality, operational stability, and reliability in production environments. Excellent problem-solving, documentation, and collaboration skills.

Databricks,PaySpark,Aws Cloud,Python,sql