Lead Data Engineer Hungry Bird It Consulting Services

5.0 - 10.0 years

30 - 45 Lacs

hyderabad, pune

Work from Office

Hello Candidate, Greetings from Hungry Bird IT Consulting Services Pvt Ltd. We are hiring Lead Data Engineer for our client. Job Title: Lead Data Engineer Location: Hyderabad Job Type: Full-Time Experience: 5+ years (2+ years in a leadership role) Role Overview: We are looking for a highly skilled and experienced Lead Data Engineer to oversee the design, development, and management of scalable data infrastructure and pipelines. You will work closely with cross-functional teams, lead a team of engineers, and drive best practices in data engineering using modern cloud and big data technologies like Databricks , AWS , Airflow , and FastAPI . Key Responsibilities: Lead the design, development, and optimization of robust ETL pipelines and data workflows Architect scalable and secure data solutions using Databricks and AWS Manage and optimize data storage using AWS S3 , following data lake and warehouse best practices Implement centralized data governance using Databricks Unity Catalog Develop data APIs using FastAPI for internal and external integrations Use Airflow for workflow orchestration and automation Work with columnar databases like Rockset , Clickhouse , and CrateDB Ensure data quality, security, and compliance across all systems Collaborate with data scientists and analysts to meet analytical and ML needs Mentor and guide junior data engineers, promoting code quality and reusability Stay updated with new tools and technologies to enhance data infrastructure Contribute to cross-functional projects with technical leadership Required Qualifications: Bachelors/Masters in Computer Science, Engineering, or a related field 5+ years of experience in data engineering, with 2+ years in a lead role Strong expertise in Python , PySpark , and SQL Hands-on experience with Databricks , AWS (S3, IAM, Lambda, ECR) Experience with Airflow , FastAPI , and columnar databases Deep understanding of ETL pipelines , data modeling , and data warehousing Familiarity with Git, CI/CD, and Agile methodologies Knowledge of data governance and compliance standards Preferred Qualifications: Experience with real-time data processing/streaming Familiarity with MLOps and machine learning workflows Certifications in Databricks and/or AWS Exposure to data mesh or data fabric architectures Understanding of metadata management and data lineage Tech Stack: Languages & Frameworks: Python, PySpark, SQL, FastAPI Cloud: AWS (S3, IAM, Lambda, ECR) Big Data: Databricks Orchestration: Airflow Databases: Rockset, Clickhouse, CrateDB Governance: Databricks Unity Catalog (Interested candidates can share their CV with us at or reach us at aradhana@hungrybird.in ) PLEASE MENTION THE RELEVANT POSITION IN THE SUBJECT LINE OF THE EMAIL. Example: KRISHNA, HR MANAGER, 7 YEARS, 20 20DAYS NOTICE. Name: Position applying for: Total experience: Notice period: Current Salary: Expected Salary: Thanks and Regards Aradhana +919959417171

Posted 1 week ago

Apply

Data Engineer - Senior/Lead Fission Labs

5.0 - 12.0 years

0 Lacs

hyderabad, telangana

On-site

As a Senior Data Engineer at our company based in Hyderabad, India, you will be responsible for leading the design, development, and maintenance of data pipelines and ETL processes. Your role will involve architecting and implementing scalable data solutions using Databricks and AWS, optimizing data storage and retrieval systems using Rockset, Clickhouse, and CrateDB, and developing data APIs using FastAPI. Furthermore, you will orchestrate and automate data workflows using Airflow, collaborate with data scientists and analysts, and ensure data quality, security, and compliance across all data systems. Mentoring junior data engineers, evaluating and implementing new data technologies, and participating in cross-functional projects will also be part of your responsibilities. Additionally, managing and optimizing data storage solutions using AWS S3 and implementing Databricks Unity Catalog for centralized data governance and access control will be crucial aspects of your role. To qualify for this position, you should hold a Bachelor's or Master's degree in Computer Science, Engineering, or a related field, along with a minimum of 5 years of experience in data engineering, including 2-3 years in a lead role. Proficiency in Python, PySpark, SQL, Databricks, AWS cloud services, Airflow, FastAPI, and columnar databases like Rockset, Clickhouse, and CrateDB is essential. You should also have a solid understanding of data modeling, data warehousing, ETL processes, version control systems (e.g., Git), and CI/CD pipelines. Strong problem-solving skills, the ability to work in a fast-paced environment, excellent communication skills, and knowledge of data governance, security, and compliance best practices are also required. Experience in designing and implementing data lake architectures using AWS S3 and familiarity with Databricks Unity Catalog or similar data governance tools are preferred. In terms of skills and experience, you should be proficient in the following tech stack: Databricks, Python, PySpark, SQL, Airflow, FastAPI, and AWS services such as S3, IAM, ECR, and Lambda, as well as Rockset, Clickhouse, and CrateDB. Working with us will provide you with various benefits, including the opportunity to work on business challenges from global clientele, self-development opportunities, sponsored certifications, tech talks, industry events, retirement benefits, flexible work hours, and a supportive work environment that encourages exploring passions beyond work. This role offers an exciting opportunity for you to contribute to cutting-edge solutions and advance your career in a dynamic and collaborative environment.,

Posted 2 weeks ago

Apply

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.