WeAssemble logo

Big Data Engineer - ETL/PySpark

0 years

0 Lacs

mumbai metropolitan region

Posted:1 week ago| Platform: Linkedin logo

Apply

Skills Required

data etl pyspark design processing pipeline development apache spark aws redshift architecture database relational nosql optimization automation monitoring workflow management airflow programming python sql orchestration storage azure kafka

Work Mode

On-site

Job Type

Full Time

Job Description

Description

Job Summary :We are looking for an experienced Big Data Engineer to design, build, and optimize data pipelines and infrastructure supporting large-scale data processing and analytics.You will be instrumental in migrating our legacy systems to a modern, scalable data platform.

Key Responsibilities

  • ETL/ELT Pipeline Development: Design, construct, and maintain robust, scalable, and fault-tolerant ETL/ELT pipelines for ingesting and transforming high-volume, high-velocity data using Apache Spark or Databricks.
  • Data Warehousing: Implement and manage modern data warehousing solutions on the cloud, utilizing platforms like Snowflake, AWS Redshift, or Google BigQuery.
  • Data Architecture: Evaluate and implement optimal database solutions (relational, NoSQL, data lake technologies like Delta Lake or Hudi) for different use cases.
  • Performance Optimization: Tune and optimize Spark jobs and database queries for maximum performance and cost efficiency.
  • Automation & Monitoring: Integrate data pipelines with workflow management tools like Apache Airflow and establish monitoring and alerting for data quality and pipeline health.

Required Technical Skills

  • Big Data Frameworks: Expert proficiency with Apache Spark (PySpark or Scala).
  • Programming: Strong skills in Python or Scala.
  • Databases/Warehousing: Expertise in SQL and hands-on experience with modern data warehouses (Snowflake, Redshift, BigQuery).
  • Workflow Orchestration: Hands-on experience with Apache Airflow or similar tools.
  • Cloud & Storage: Experience with cloud storage services (AWS S3, Azure Data Lake Storage, Google Cloud Storage) and cloud platform services.
  • Streaming (Bonus): Familiarity with real-time streaming technologies like Kafka or Kinesis
(ref:hirist.tech)

More Jobs at WeAssemble

Full Stack Python Developer - 1 years Expereince - Mumbai

Mumbai, Maharashtra, India

1.0 - 1.0 yrs
|

Salary: Not disclosed

Social Media Executive

Mumbai, Maharashtra

Experience: Not specified

|

INR 0 - 0 Lacs

Social Media Executive

Mumbai, Maharashtra, India

Experience: Not specified

|

Salary: Not disclosed

Senior Dev Ops- Site Reliability Engineer

Mumbai Metropolitan Region

Experience: Not specified

|

Salary: Not disclosed

Inside Sales Representative (ISR) – IT Services & Staff Augmentation

Mumbai, Maharashtra

3.0 - 3.0 yrs
|

Salary: Not disclosed

Mock Interview

Practice Video Interview with JobPe AI

Start PySpark Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

Google Play App Store
coding practice

Enhance Your Python Skills

Practice Python coding challenges to boost your skills

Start Practicing Python Now
WeAssemble logo
WeAssemble

RecommendedJobs for You

WeAssemble logo

Big Data Engineer - ETL/PySpark

WeAssemble

mumbai metropolitan region

WeAssemble logo

Big Data Engineer - ETL/PySpark

WeAssemble

mumbai metropolitan region