1166 Data Pipeline Jobs - Page 5

Experience : 2+ years Shift : (GMT+05:30) Asia/Kolkata (IST) Opportunity Type : Office (Mumbai) Placement Type : Full time Permanent Position (*Note: This is a requirement for one of Uplers' client- Shaadi.com) Required skills: Kafka and Redshift Your Superpowers (a.k.a. What You'll Do) Build data pipelines that could carry a whale Stream data in real time like a Netflix show on 5G Clean, organize, and serve data so clean you could eat off it Make insights magically appear (okay, not magically, but with great logic) Write ETL code that makes even your past self say, Whoa Partner with teams across the org and become their data superhero Tools in Your Utility Belt Python (or you talk to snakes...

Experience : 2+ years Shift : (GMT+05:30) Asia/Kolkata (IST) Opportunity Type : Office (Mumbai) Placement Type : Full time Permanent Position (*Note: This is a requirement for one of Uplers' client- Shaadi.com) Required skills: Kafka and Redshift Your Superpowers (a.k.a. What You'll Do) Build data pipelines that could carry a whale Stream data in real time like a Netflix show on 5G Clean, organize, and serve data so clean you could eat off it Make insights magically appear (okay, not magically, but with great logic) Write ETL code that makes even your past self say, Whoa Partner with teams across the org and become their data superhero Tools in Your Utility Belt Python (or you talk to snakes...

Skills- AWS +Pyspark Experience: 4 to 9 years Location: AIA-Pune As an AWS Data Engineer, you will design, build, and manage robust data pipelines on the AWS cloud platform. You will leverage your expertise in AWS data services and programming languages to ingest, process, transform, and store data efficiently, enabling data-driven insights and decision-making. Key Responsibilities : - Data Pipeline Development: Design, implement, and maintain scalable data pipelines using AWS services such as S3, Glue, Lambda, EMR, and Kinesis. - Data Processing and Transformation: Utilize PySpark, Spark, and SQL to perform complex data transformations and aggregations on large datasets within the AWS ecosy...

Skills- Databricks+ Pyspark Experience: 4 to 13 years Location: AIA-Pune We are looking for a highly skilled Data Engineer with expertise in PySpark and Databricks to design, build, and optimize scalable data pipelines for processing massive datasets. Key Responsibilities: Build & Optimize Pipelines: Develop high-throughput ETL workflows using PySpark on Databricks. Data Architecture & Engineering: Work on distributed computing solutions, optimize Spark jobs, and build efficient data models. Performance & Cost Optimization: Fine-tune Spark configurations, optimize Databricks clusters, and reduce compute/storage costs. Collaboration: Work closely with Data Scientists, Analysts, and DevOps tea...

Skills- Pyspark Experience: 6 to 9 years Location: AIA-Pune Responsibilities Develop and maintain scalable data pipelines using Python and PySpark. Collaborate with data engineers and data scientists to understand and fulfil data processing needs. Optimize and troubleshoot existing PySpark applications for performance improvements. Write clean, efficient, and well-documented code following best practices. Participate in design and code reviews. Develop and implement ETL processes to extract, transform, and load data. Ensure data integrity and quality throughout the data lifecycle. Stay current with the latest industry trends and technologies in big data and cloud computing.

Skills- Azure Databricks Experience: 9 to 13 years Location: AIA-Pune Responsibilities Design, develop, and deploy data processing solutions using Azure Databricks. Collaborate with data engineers, architects, and business analysts to gather and analyze requirements. Develop, optimize, and manage complex data pipelines and workflows. Ensure data quality, integrity, and security across various data sources. Troubleshoot and resolve performance issues related to Databricks solutions. Document technical specifications and maintain comprehensive project documentation. Provide technical guidance and mentorship to junior developers. Qualifications Minimum of 5 years of experience in data engineeri...

We are seeking a skilled Data Engineer with strong SQL and ETL/ELT experience to join a dynamic team supporting one of the largest insurance providers in the United States. This role involves designing and building data pipelines, transforming raw data into actionable insights, and collaborating with cross-functional teams to deliver high-quality data solutions. Key Responsibilities Design, build, test, and implement scalable data products and pipelines. Write and optimize complex SQL queries for data transformation, validation, and extraction. Collaborate with data analysts, scientists, and business teams to understand data needs. Maintain and enhance existing ETL workflows for performance ...

We are hiring a Data Engineer with strong experience in AWS cloud services, Python, and data pipeline development. The ideal candidate will work on designing scalable data solutions, building ETL pipelines, and collaborating with cross-functional teams to deliver high-quality data products. Key Responsibilities: Design and develop data pipelines using AWS Glue, Lambda, Airflow, and S3. Build dimensional data models and dashboards using Power BI. Implement CI/CD pipelines for automated deployments. Work with APIs, JSON/XML data formats, and data governance practices. Collaborate with data scientists and business teams to deliver insights.

Prescience Decision Solutions - A Movate Company Role: Data Engineer (Operations) Location: Bangalore(Hybrid) Job Description: We are seeking a skilled Ops Data Engineer to maintain robust data infrastructure and pipelines that support our operational analytics and business intelligence needs. Candidates will bridge the gap between data engineering and operations, ensuring reliable, scalable, and efficient data systems that enable data-driven decision making across the organization. Apply : Immediate Joiners/ Serving notice period only - preferred Key Responsibilities: Maintain ETL/ELT pipelines using modern data engineering tools and frameworks Requires On-call support data pipeline health,...

Dear Candidate , Please find the details below: Skills - Data Lead (MS Fabric - Mandatory Skill) Exp - 8 to 12 years Location Pune-Viman Nagar (WFO) Please find the job description below: Design and build data pipelines, lakehouse architectures (Bronze–Silver–Gold) , and semantic models in Microsoft Fabric. Develop data ingestion and transformation workflows using Data Factory, Dataflows Gen2, SQL, and PySpark. Integrate Fabric with Power BI for analytics and reporting, ensuring data quality, governance, and performance optimization. Implement DevOps (CI/CD, Git) practices for Fabric assets and stay updated with Microsoft Fabric roadmap (Cortex AI, Copilot, Real-Time Intelligence). Kindly Sh...

Skills-GCP Data Engineer Experience: 6 to 12 years Location: AIA-Gurgaon Hands-on experience with GCP services, specifically BigQuery, Cloud Storage, and Composer for data pipeline orchestration Proficiency in Databricks platform with PySpark for building and optimizing large-scale ETL/ELT processes Expertise in writing and tuning complex SQL queries for data transformation, aggregation, and reporting on large datasets Experience integrating data from multiple sources such as APIs, cloud storage, and databases into a central data warehouse Familiarity with workflow orchestration tools like Apache Airflow or Cloud Composer for scheduling, monitoring, and managing data jobs Knowledge of versio...

What will you drive? As a BI team player, you will be responsible for overseeing the planning, execution, and delivery of complex and challenging technical projects involved in Business Intelligence, within our organization. Following are your key responsibilities for the role: Data-Driven Decision Making: At CARS24, we thrive on insights derived from data. Your role would involve harnessing data-backed insights to inform decisions across all functions. Whether its refining strategies, optimizing processes, or implementing innovative solutions, your analytical prowess will be instrumental. Core of Business Efforts: The BI team holds a pivotal position, bridging data with business context to ...

Are you ready to design, build, and scale next-gen data solutions? Do you thrive on turning data into decisions and guiding teams toward success? What Were Looking For: 10+ years in Datawarehousing & ETL Hands-on with Azure, Databricks, ADF, Synapse, PySpark/Python Strong in STTM (Source-to-Target Mapping), design & documentation mentor at heart who can guide junior talent Good to Have: Certifications in Azure/Data Engineering Banking exposure in Risk & Regulatory / Commercial / Credit Cards / Retail What You'll Achieve With Us: Architect & optimize cloud-first data ecosystems Solve real-world enterprise-scale data challenges Be the trusted expert in a high-impact role Grow into leadership a...

Job Title: Senior Data Engineer AWS | Python | Data Pipelines Experience: 6 - 10 Years Location: Hyderabad Employment Type: Full-time | Permanent Notice Period: Immediate About the Role: We are seeking a highly skilled Senior Data Engineer with deep expertise in AWS cloud services and Python-based data engineering . In this role, you will architect and build scalable, automated, and high-performance data platforms that power analytics, AI, and business intelligence across the organization. Key Responsibilities: Architect and implement end-to-end data pipelines using AWS Glue, Lambda, EMR, Step Functions, and Redshift . Design and manage data lakes and warehouses on Amazon S3, Redshift, and A...

Role & responsibilities • Serve as the Subject Matter Expert (SME) for data engineering solutions within a high-stakes production support context . • Provide expert-level troubleshooting, diagnosis, and resolution for complex incidents affecting data pipelines built on Azure Databricks, PySpark, SQL, Azure Data Factory (ADF) and other related Azure data services (e.g., Azure Data Lake Storage, Azure Synapse Analytics, Azure SQL Database). • Conduct thorough root cause analysis (RCA) for recurring data issues, implementing sustainable long-term solutions to minimize downtime and prevent data discrepancies. • Proactively monitor, optimize, and enhance existing data pipelines for performance, s...

Company Introduction Coditas is a new-age, offshore product development organization, offering services pertaining to the entire software development life cycle. Headquartered in Pune, Coditas works with clients across the globe. We attribute our organic growth to an engineering-driven culture and steadfast philosophies around writing clean code, designing intuitive user experiences, and letting the work speak for itself. Job Description We are looking for data engineers who have the right attitude, aptitude, skills, empathy, compassion, and hunger for learning. Build products in the data analytics space. A passion for shipping high-quality data products, interest in the data products space;...

Hiring an ETL Developer to design, develop, and maintain robust data integration pipelines. The ideal candidate will have strong SQL skills and hands-on experience with ETL tools like SSIS, Informatica, or Talend. Apply now recruiter2@rankskills.in Perks and benefits Mediclaim policy & Personal Accident Policy

Role Proficiency: This role requires proficiency in data pipeline development including coding and testing data pipelines for ingesting wrangling transforming and joining data from various sources. Must be skilled in ETL tools such as Informatica Glue Databricks and DataProc with coding expertise in Python PySpark and SQL. Works independently and has a deep understanding of data warehousing solutions including Snowflake BigQuery Lakehouse and Delta Lake. Capable of calculating costs and understanding performance issues related to data solutions. Outcomes: Act creatively to develop pipelines and applications by selecting appropriate technical options optimizing application development mainten...

SUMMARY W issen Technology is Hiring for .Net + AWS Developer About Wissen Technology: At Wissen Technology, we deliver niche, custom-built products that solve complex business challenges across industries worldwide. Founded in 2015, our core philosophy is built around a strong product engineering mindset ensuring every solution is architected and delivered right the first time. Today, Wissen Technology has a global footprint with 2000+ employees across offices in the US, UK, UAE, India, and Australia . Our commitment to excellence translates into delivering 2X impact compared to traditional service providers. How do we achieve this? Through a combination of deep domain knowledge, cutting-ed...

About the Role This role is part of the MarTech Data Engineering POD supporting a large-scale ESP (Email Service Provider) migration program from Oracle Responsys to Adobe Journey Optimizer. The initiative aims to build a scalable, real-time marketing data foundation integrating customer journeys, preference orchestration, and omnichannel personalization across email, mobile, and direct channels. The offshore team will focus on designing, building, and maintaining data pipelines across SAS , OES , Wunderkind , and AEP (Adobe Experience Platform), enabling real-time campaign execution and advanced targeting capabilities. Key Responsibilities Design and develop data ingestion, transformation, ...

•Provide technical leadership and mentorship to a team of data engineers. •Design, architect, and implement highly scalable, resilient, and performant data pipelines, using GCP technologies is a plus (e.g., Dataproc, Cloud Composer, Pub/Sub, BigQuery). •Guide the team in adopting best practices for data engineering, including CI/CD, infrastructure-as-code, and automated testing. •Conduct code reviews, design reviews, and provide constructive feedback to team members. •Stay up-to-date with the latest technologies and trends in data engineering, Data Pipeline Development: •Develop and maintain robust and efficient data pipelines to ingest, process, and transform large volumes of structured and...

We are looking for a skilled and experienced Databricks Engineer to join our team at our Hyderabad (Madhapur) office. The ideal candidate will have a strong background in data engineering, hands-on experience with Databricks, and a solid understanding of cloud platforms (AWS, Azure, or GCP). This is a great opportunity to work on cutting-edge data projects in a collaborative and fast-paced environment. Key Responsibilities : - Design and implement scalable and efficient data pipelines, ETL/ELT processes, and data integration solutions using Databricks. - Work with large-scale structured and unstructured datasets using Python, PySpark, SQL, and NoSQL technologies. - Collaborate with cross-fun...

Role: Sr. Data Engineer - Snowflake Experience: 5-8 years Location: Hyderabad Job Overview: We are seeking a highly skilled and experienced Snowflake Data Engineer with a strong background in SQL, Python, and a minimum of 3 years of hands-on Snowflake development experience. The ideal candidate will be certified, with a proven track record in data warehousing, data modeling, performance optimization, and implementing scalable ETL/ELT pipelines using industry-standard tools. Experience in leading small delivery tracks or mentoring junior engineers is highly desirable. Primary Roles & Responsibilities: • Design, develop, and optimize scalable data pipelines and ETL/ELT processes in Snowflake. ...

Must-Have Skills: Strong hands-on experience with Python, ETL processes, and Snowflake Expertise in PySpark and Big Data Engineering Proven ability to create and manage data pipelines and data systems

The AI Engineer plays a critical role in advancing LabVantages AI capabilities by designing, evaluating, and integrating state-of-the-art AI and generative models into our product suite. This includes selecting and benchmarking large language models (LLMs), developing applications using Retrieval-Augmented Generation (RAG) and agentic AI, and collaborating with engineering teams to deliver innovative, scalable, and robust AI-driven features. Additional responsibilities include building and maintaining data pipelines, developing APIs for model access, and staying at the forefront of AI innovation to ensure LabVantage remains a leader in applied AI. The ideal candidate has a strong understandi...