Data Engineer - Senior Software Engineer

Prodapt Solutions

3 - 6 years

6 - 10 Lacs

Bengaluru

Posted:2 months ago| Platform:

Apply

Skills Required

database design data modeling etl data processing pipeline hive information technology scala pyspark sql docker java spark software engineering mysql hadoop big data mongodb snowflake python airflow microsoft azure data engineering nosql kafka aws

Work Mode

Work from Office

Job Type

Full Time

Job Description

Data Engineer - Senior Software Engineer Bangalore, India Information Technology 16750 Overview We are seeking a skilled and experienced Data Engineer who has expertise in playing a vital role in supporting, data ingestion/migration, creating data pipelines, creating data marts and managing, monitoring the data using tech stack Azure, SQL, Python, PySpark, Airflow and Snowflake. Responsibilities 1. Data Ingestion/MigrationCollaborate with cross-functional teams to Ingest/migrate data from various sources to staging area. Develop and implement efficient data migration strategies, ensuring data integrity and security throughout the process. 2. Data Pipeline DevelopmentDesign, develop, and maintain robust data pipelines that extract, transform, and load (ETL) data from different sources into GCP. Implement data quality checks and ensure scalability, reliability, and performance of the pipelines. 3. Data ManagementBuild and maintain data models and schemas, ensuring optimal storage, organization, and accessibility of data. Collaborate with requirement team to understand their data requirements and provide solutions by creating data marts to meet their needs. 4. Performance OptimizationIdentify and resolve performance bottlenecks within the data pipelines and data services. Optimize queries, job configurations, and data processing techniques to improve overall system efficiency. 5. Data Governance and SecurityImplement data governance policies, access controls, and data security measures to ensure compliance with regulatory requirements and protect sensitive data. Monitor and troubleshoot data-related issues, ensuring high availability and reliability of data systems. 7. Documentation and CollaborationCreate comprehensive technical documentation, including data flow diagrams, system architecture, and standard operating procedures. Collaborate with cross-functional teams, analysts, and software engineers, to understand their requirements and provide technical expertise. Requirements Qualifications: - Bachelor's or Master's degree in Computer Science, Information Systems, or a related field. - Proven experience as a Data Engineer with a focus on output driven. - Knowledge and hands-on experience with Azure, SQL, Python, PySpark, Airflow, Snowflake and related tools. - Proficiency in data processing and pipeline development. Solid understanding of data modeling, database design, and ETL principles. - Experience with data migration projects, including data extraction, transformation, and loading. - Familiarity with data governance, security, and compliance practices. - Good communication and interpersonal skills, with the ability to articulate technical concepts to non-technical stakeholders.

Mock Interview

Practice Video Interview with JobPe AI

Start PySpark Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

Enhance Your Java Skills

Practice Java coding challenges to boost your skills

Start Practicing Java Now

Prodapt Solutions

Software Development

Chennai Tamilnadu

Login to

Please Verify Your Phone or Email

Confirm Action

Search

Profile

Upskill and Grow with AI

Data Engineer - Senior Software Engineer