Jobs
Interviews

Senior Data Engineer PySpark/Hadoop

0 - 2 years

2 - 4 Lacs

Kolkata

Posted:4 months ago| Platform: Naukri logo

AI Match Score
Apply

Skills Required

redis sem rdbms nosql database creation continuous integration ci/cd pyspark docker sql database design postgresql spark mysql hadoop etl mongodb data lake cache python github data processing data engineering query optimization aws etl process

Work Mode

Work from Office

Job Type

Full Time

Job Description

Role :Senior Data Engineer. Experience :4+ yrs. Work Mode :Remote. Joining Time Frame :Immediate to 15 days. Key Responsibilities. Design and architect robust data pipelines for structured, semi-structured, and unstructured data.
  • Develop, manage, and optimize databases, including RDMS (MySQL, PostgreSQL), NoSQL (MongoDB), and data lakes (S3).
  • Implement efficient ETL processes using tools like PySpark and Hadoop to transform and prepare data for analytics and #AI use cases.
  • Optimize database performance, including query tuning, indexing, and caching strategies using tools like Redis and AWS-specific caching databases.
  • Build and maintain CI/CD pipelines, manage YML files, and use GitHub for version control and collaboration.
  • Leverage Docker for containerized deployment, with hands-on experience in running Docker commands for database and pipeline management.
  • Ensure solutions adhere to best practices in system design, focusing on trade-offs, security, performance, and efficiency.
  • Monitor, maintain, and troubleshoot database infrastructure to ensure high availability and performance.
  • Collaborate with engineering teams to design scalable solutions for large-scale data processing.
  • Stay updated on the latest database technologies and implement best practices for database design and management.
  • Qualifications. 4+ years of experience in database architecture and optimization.
  • Expertise in RDMS, NoSQL, and semi-structured databases (MySQL, PostgreSQL, MongoDB).
  • Proficiency in programming languages for database integration and optimization (Python preferred).
  • Strong knowledge of distributed data processing tools like PySpark and Hadoop.
  • Hands-on experience with AWS services for data storage and processing, including S3.
  • Strong familiarity with Redis for caching and query optimization.
  • Proven experience with Docker for containerized deployments and writing CI/CD pipelines using YML files.
  • (ref:hirist.tech). Show more Show less

    • Mock Interview

    Practice Video Interview with JobPe AI

    Start PySpark Interview
    cta

    Start Your Job Search Today

    Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

    Job Application AI Bot

    Job Application AI Bot

    Apply to 20+ Portals in one click

    Download Now

    Download the Mobile App

    Instantly access job listings, apply easily, and track applications.

    Google Play App Store
    coding practice

    Enhance Your Python Skills

    Practice Python coding challenges to boost your skills

    Start Practicing Python Now
    Exult

    Outsourcing and Offshoring Consulting

    RecommendedJobs for You

    Software Developer

    Anantya.ai

    Bhatgaon

    Software Developer

    Ericsson

    Bengaluru, Karnataka, India

    Software Developer

    IBM

    Hyderabad, Telangana, India

    Software Developer

    Ericsson

    Noida, Uttar Pradesh, India

    Software Developer

    Chemelex

    Mumbai Metropolitan Region

    Full-stack Developer

    Dc Consultants

    Thane, Maharashtra, India

    Backend Engineer

    Robotics Technologies

    Gurgaon, Haryana, India

    Python Backend

    Robotics Technologies

    Delhi, Delhi, India

    Java developer

    Robotics Technologies

    Gurgaon, Haryana, India

    Java Developer

    INNOVA SOLUTIONS

    Hyderabad, Telangana, India