1569 Data Pipeline Jobs - Page 2

Azure Data Factory and Azure Synapse Analytics. SQL and ETL concepts. Azure Data Lake Storage (ADLS) , Medallion Architecture Python and Azure functions for data transformation CI/CD pipelines, version control (Git), Agile methodologies.

Role & responsibilities Overall 6+ years and 5+ years of experience in Palantir Hyper automation Data Pipeline Development: Design, develop, and optimize scalable data pipelines to process large datasets. Infrastructure Maintenance: Support and enhance existing data infrastructure, ensuring data integrity, reliability, and performance. Greenfield Projects: Lead the development of new data systems and solutions from scratch, leveraging Palantir Foundry. Data Integration: Ensure seamless integration, data quality, and security in alignment with Palantir's capabilities. Collaboration: Work with product teams, analysts, and other engineers to define data requirements and ensure successful delive...

Role & responsibilities Overall 6+ years and 5+ years of experience in Palantir Hyper automation Data Pipeline Development: Design, develop, and optimize scalable data pipelines to process large datasets. Infrastructure Maintenance: Support and enhance existing data infrastructure, ensuring data integrity, reliability, and performance. Greenfield Projects: Lead the development of new data systems and solutions from scratch, leveraging Palantir Foundry. Data Integration: Ensure seamless integration, data quality, and security in alignment with Palantir's capabilities. Collaboration: Work with product teams, analysts, and other engineers to define data requirements and ensure successful delive...

Overview : We are seeking a highly skilled and experienced Senior Data Engineer (L4) to join the program in supporting critical customer engagement. We are looking for a candidate to design and build scalable data pipelines on GCP, responsible for replacing end-of-life legacy ETL tools with modern cloud frameworks and building core data models. This role is ideal for a self-starter who possesses strong analytical problem-solving skills, specifically the ability to reverse-engineer undocumented legacy code and map it to modern logic without losing business context. The ideal candidate adapts quickly between different technologies (Java, SQL, Python) and maintains rigorous attention to detail ...

Job Position : GCP Data Engineer Experience : 6+ Yrs into Gcp Data Engineer Notice Period : Immediate - 15 days only JD- Key Responsibilities : We are seeking a skilled Data Engineer to design, build, and maintain data pipelines and data models that support analytical and business intelligence needs. The ideal candidate will have hands-on experience with Python or SQL, Google Cloud Platform (GCP), and a strong understanding of data management, quality, and security best practices. Key Responsibilities : - Build and maintain moderately complex data pipelines, ensuring data flow, transformation, and usability for analytical projects. - Design and implement data models, optimizing for performan...

Job Position : GCP Data Engineer Experience : 6+ Yrs into Gcp Data Engineer Location : Chennai, Pune, Trivandrum, Kochi Notice Period : Immediate - 15 days only JD- Key Responsibilities : We are seeking a skilled Data Engineer to design, build, and maintain data pipelines and data models that support analytical and business intelligence needs. The ideal candidate will have hands-on experience with Python or SQL, Google Cloud Platform (GCP), and a strong understanding of data management, quality, and security best practices. Key Responsibilities : - Build and maintain moderately complex data pipelines, ensuring data flow, transformation, and usability for analytical projects. - Design and imp...

Skills- Azure Databricks +Pyspark Experience: 9 to 13 years Location: AIA-Pune Responsibilities Design, develop, and deploy data processing solutions using Azure Databricks. Collaborate with data engineers, architects, and business analysts to gather and analyze requirements. Develop, optimize, and manage complex data pipelines and workflows. Ensure data quality, integrity, and security across various data sources. Troubleshoot and resolve performance issues related to Databricks solutions. Document technical specifications and maintain comprehensive project documentation. Provide technical guidance and mentorship to junior developers. Qualifications Minimum of 5 years of experience in data ...

Position Overview : We are looking for a seasoned Data Architect with deep expertise in the Azure and Microsoft ecosystem to design, build, and optimize enterprise-grade data solutions across hybrid environments. You will play a pivotal role in defining the data architecture strategy, leading complex data initiatives, and ensuring our data platforms are secure, scalable, and aligned with business objectives. Key Responsibilities : Architecture & Design : - Design and implement end-to-end data architectures leveraging Azure cloud and on-premises Microsoft technologies. - Develop data strategy, standards, and governance frameworks to ensure data consistency, integrity, and availability. - Crea...

Description : Industry & Sector : Technology consulting firm operating in the blockchain, fintech and distributed ledger technology (DLT) space. We deliver enterprise-grade DLT integrations, ledger analytics and data engineering solutions for global clients, transforming on-chain and off-chain data into actionable business insights. Role : Senior Data Engineer (DLT) Remote (India). Role & Responsibilities : - Design and build scalable ETL/ELT pipelines to ingest, process and transform on-chain and off-chain ledger data using streaming and batch frameworks. - Integrate with DLT nodes, RPC endpoints and indexers to capture transactions, blocks and smart-contract events into analytical stores. ...

About The Opportunity : A fast-scaling technology company operating in the Enterprise AI / Generative AI sector, building production-grade LLM-driven products and intelligent automation for global clients. We deliver secure, low-latency generative systems that power conversational AI, summarization, code generation, and retrieval-augmented applications across cloud-native environments. We are hiring a Senior Generative AI Engineer (7+ years) to own architecture, model development, and production deployment of advanced generative systems. This is a fully remote role for candidates based in India. Role & Responsibilities : - Lead end-to-end design and delivery of Generative AI/LLM solutionsdat...

We are looking for a motivated SQL/Python Developer with hands-on experience in SQL, Python, and basic exposure to Snowflake. The ideal candidate should have a good understanding of querying databases, working with datasets, and supporting data pipelines. Basic knowledge of Git for version control is a plus. Key Responsibilities : - Write, execute, and optimize SQL queries for data extraction, transformation, and analysis. - Develop Python scripts to automate data processing and reporting tasks. - Work with Snowflake to access and run queries on cloud-based data warehouses. - Support data integration, validation, and ETL processes. - Collaborate with team members to troubleshoot data issues ...

Job description 12+ years' of relevant experience, with 3+ years with Snowflake. Snowflake Solution Architect: You are an expert with hands-on solution design and development experience You will be an influencer and thought leader with in-depth technical expertise, credibility, and field experience to establish yourself as a subject-matter expert in a company that is leading innovation within the integration services industry You have a passion for continuous learning You enjoy mentoring and developing the skills of those around you. Job Responsibilities: Lead and Architect migration of data analytics environment to Snowflake with performance and reliability. Assess and understand the ETL jo...

Position Overview : We are looking for an experienced Data Scientist with a strong academic background in Data Science and 10+ years of hands-on experience in Data science projects. The ideal candidate should be skilled in developing predictive models, performing data analysis, and applying advanced machine learning and AI techniques to solve Roles and Responsibilities : - Analyze large datasets to extract meaningful insights and patterns. - Develop, implement, and optimize ML/AI models for predictive and prescriptive analytics - Collaborate with engineering, product, and business teams to define and deliver data-driven solutions. - Design and build data pipelines and perform data wrangling ...

Roles and Responsibilities : - Create and maintain efficient and scalable data models as per business needs - Create and maintain optimal data pipelines against multiple data sources lie SQL, BigData on Azure / AWS cloud; - Assemble and process large, complex data sets to meet both functional and non-functional business requirements; - Analyze and improve existing data models, pipelines, related infrastructure and processes - Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics; - Monitor and improve data quality and data governance policies - Collaborate with stakehold...

Role & responsibilities Overall 6+ years and 5+ years of experience in Palantir Hyper automation Data Pipeline Development: Design, develop, and optimize scalable data pipelines to process large datasets. Infrastructure Maintenance: Support and enhance existing data infrastructure, ensuring data integrity, reliability, and performance. Greenfield Projects: Lead the development of new data systems and solutions from scratch, leveraging Palantir Foundry. Data Integration: Ensure seamless integration, data quality, and security in alignment with Palantir's capabilities. Collaboration: Work with product teams, analysts, and other engineers to define data requirements and ensure successful delive...

SUMMARY W issen Technology is Hiring fo r EDP and Integration Lead About Wissen Technology: At Wissen Technology, we deliver niche, custom-built products that solve complex business challenges across industries worldwide. Founded in 2015, our core philosophy is built around a strong product engineering mindset ensuring every solution is architected and delivered right the first time. Today, Wissen Technology has a global footprint with 2000+ employees across offices in the US, UK, UAE, India, and Australia . Our commitment to excellence translates into delivering 2X impact compared to traditional service providers. How do we achieve this? Through a combination of deep domain knowledge, cutti...

Role & responsibilities • Define and uphold data hygiene and data stewardship best practices for the company. • Lead in the design, development, and maintenance of complex data pipelines using Python, Fivetran, dbt, Snowflake, and Hightouch. • Set coding standards and best practices for the data engineering team. • Conduct code reviews to ensure quality and consistency. • Solve complex data puzzles through innovative design and architecture. • Mentor less experienced engineers, providing technical guidance and career development support. • Lead initiatives to mature existing capabilities and introduce new capabilities into company's data practice. • Conduct reverse ETL processes to push metr...

Strong exp with AWS Glue (Jobs, Crawlers, Workflows) • Hands-on knowledge of AWS Glue Data Catalog • Exp building AWS data pipelines and VPC, IAM, and security best practices • Proficiency in Python (PySpark) • Exp with S3-based data lakes

The Data Engineer will be responsible for designing, building, and maintaining scalable data pipelines and data platforms to support analytics, reporting, forecasting, and advanced data science use cases. The role requires strong technical expertise in modern data engineering tools, cloud platforms, and data processing frameworks. The engineer will work closely with data scientists, analysts, and business teams to ensure reliable, high-quality data availability

Role : Senior Data Engineer Location : Hyderabad Employment Type : Full-Time Job Description: The Senior Data Engineer will build and maintain the core data infrastructure for an enterprise AI platform. This role focuses on designing scalable data pipelines, developing knowledge graphs, and preparing structured and unstructured data for AI and LLM-based applications. Roles & Responsibilities : Data Pipeline Development : - Design and build scalable data ingestion pipelines from enterprise systems (ERP, documentation tools, version control, and project management tools) - Develop connectors for structured, semi-structured, and unstructured data - Implement incremental data loads, change data ...

Google Cloud Platform (GCP) Data Engineer to design, develop, and manage scalable and efficient data pipelines, integrations, and analytics solutions. The ideal candidate will leverage GCP's suite of tools to optimize data processes and enable business intelligence. Key Responsibilities: Data Pipeline Development: Design and implement ETL/ELT workflows using GCP services such as Dataflow, BigQuery, and Cloud Composer (Apache Airflow). Develop real-time and batch data processing pipelines. Data Integration and Management: Integrate data from various structured and unstructured sources using APIs, Pub/Sub, and Cloud Storage. Perform data modeling and database design for optimal performance. Da...

Exp:9-15yrs Location: Hyderabad Shifts: 6:30 am onwards Proffered Technologies knowledge: Azure/ Snowflake/ Power BI/ Tableau/ Python/ GEN AI Responsibility: The work includes data extraction, data transformation, data load, data quality management, and data audit process. Designed and optimized data warehouses using clusteringmicro-partitioning, and query performance Design, develop and implement data integration (ETL) processes to transform structured data, unstructured data, disparate source data into the target data warehouse or other source applications Work with service & support team to resolve escalated operational issues and able to work Designed and developed interactive Power BI d...

Infosys Hirings for PYSPARK Developer Experience: 5-10 years Mandatory Skills: Pyspark, Spark, Python Education: B.Tech/BE/M.Tech/ME/BSC/MSC/BCA/MCA Location: PAN INDIA

Purpose/Objective To establish a robust analytics-driven decision-making capability for NMDPL, enabling the business to achieve its strategic objectives in both short and long term. The role involves developing Analytics and AI roadmaps for each business vertical, aligned with overall business and digital strategies. It includes designing, building, and optimizing data infrastructure and systems for efficient data storage, organization, and access, while ensuring seamless integration of AI solutions. The position provides a strategic and technical foundation for data management, driving operational efficiency, and delivering AI-driven solutions that enhance key value drivers and business out...

Salary: 15 to 25 LPA Exp: 4 to 7 years Location: Gurgaon/Pune/Bengalore Notice: Immediate to 30 days..!! Job Profile: Experienced Data Engineer with a strong foundation in designing, building, and maintaining scalable data pipelines and architectures. Skilled in transforming raw data into clean, structured formats for analytics and business intelligence. Proficient in modern data tools and technologies such as SQL, T-SQL, Python, Databricks, and cloud platforms (Azure). Adept at data wrangling, modeling, ETL/ELT development, and ensuring data quality, integrity, and security. Collaborative team player with a track record of enabling data-driven decision-making across business units. As a Dat...