Data Engineer

Brillio

11 - 21 years

11 - 21 Lacs

pune bengaluru

Posted:6 days ago| Platform:

Apply

Skills Required

pyspark databricks aws sql

Work Mode

Hybrid

Job Type

Full Time

Job Description

Role & responsibilities

Design, build, and maintain scalable data pipelines to collect, process, and store from multiple datasets.
Optimize data storage solutions for better performance, scalability, and cost-efficiency.
Develop and manage ETL/ELT processes to transform data as per schema definitions, apply slicing and dicing, and make it available for downstream jobs and other teams.
Collaborate closely with cross-functional teams to understand system and product functionalities, pace up feature development, and capture evolving data requirements.
Engage with stakeholders to gather requirements and create curated datasets for downstream consumption and end-user reporting.
Automate deployment and CI/CD processes using GitHub workflows, identifying areas to reduce manual, repetitive work.
Ensure compliance with data governance policies, privacy regulations, and security protocols.
Utilize cloud platforms like AWS and work on Databricks for data processing with S3 Storage.
Work with distributed systems and big data technologies such as Spark, SQL, and Delta Lake.
Integrate with SFTP to push data securely from Databricks to remote locations.
Analyze and interpret spark query execution plans to fine-tune queries for faster and more efficient processing.
Strong problem-solving and troubleshooting skills in large-scale distributed systems.

Preferred candidate profile

BE / B Tech in Engineering.

Min 4+ years of IT Experience.

QL, Python, AWS, Databricks

Tableau exposure would be optional, good to have.

Have couple of projects having work exposure on Analytics and Insights based on Big Data.

Exposure on building Data sets on complex Big Data would be an advantage.

More Jobs at Brillio

React Js Frontend Developer

Pune, Bengaluru

4 - 8 yrs

INR 14 - 20 Lacs

Java Full Stack Developer

Pune, Bengaluru

3 - 8 yrs

INR 14 - 20 Lacs

Senior Software Development Lead - R01536938

Gurgaon, Haryana, India

Experience: Not specified

Salary: Not disclosed

Angular Lead

Hyderabad, Pune, Bengaluru

6 - 10 yrs

INR 14 - 20 Lacs

Network Perimeter Browsing - R01542500

Bengaluru, Karnataka, India

Experience: Not specified

Salary: Not disclosed

Mock Interview

Practice Video Interview with JobPe AI

Start PySpark Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.