1578 Data Pipeline Jobs - Page 19

Job role: Data Engineer Location: Hyderabad/Chennai (Hybrid) Employment Type: Full-time Team: Data Engineering No-of Positions: 4 JD: Hands-on expertise in GCP data stack: BigQuery, Dataflow (Apache Beam), Dataproc, Cloud Storage, Pub/Sub, Cloud Composer (Airflow) . Strong Spark (PySpark or Scala) for batch processing on Dataproc. Solid Airflow DAG design (idempotent tasks, backfills, retries, SLAs). Advanced SQL and data modeling (star/snowflake, slowly changing dimensions, partition strategies). Proficiency in Python (preferred) or Scala/Java for data engineering. Experience with Git and CI/CD (Cloud Build/GitHub Actions/GitLab CI). Familiarity with security & governance on GCP (IAM, servi...

Role & responsibilities Design and build robust, maintainable software systems to support data pipelines, analytics, and AI applications. Collaborate with domain experts to scope, build, and iterate on AI-driven tools tailored to business needs. Develop efficient data ingestion, transformation, and validation pipelines using Python, SQL, and modern libraries. Integrate large language models (LLMs) and generative AI APIs (e.g., OpenAI, HuggingFace) into internal tooling. Create clean, intuitive user interfaces and data visualizations with libraries like Plotly and Dash. Optimize database access and performance for large structured and semi-structured datasets. Ensure tools are scalable, secur...

Job Summary Data Engineer II perform development activities within a data engineering team and help guide, onboard, and train Data Engineer I. You will work closely with account management, ETL, data warehouse, business intelligence, and reporting teams as you develop data pipelines and enhancements and investigate and troubleshoot issues. You possess an understanding of multiple data structures, including relational and non-relational data models. Essential Functions/Responsibilities/Duties Extracting, cleansing, and loading data. Building data pipelines using Snowflake, SQL, Kafka, and other technologies. Investigation and documentation of new data sets Triage incoming bugs and incidents. ...

The Role We are seeking an experienced Senior Software Data Engineer to join the Data Integrations Team, a critical component of the Addepar Platform team. The Addepar Platform is a comprehensive data fabric that provides a single source of truth for our product set, encompassing a centralized and self-describing repository, API driven data services, integration pipeline, analytics infrastructure, warehousing solutions, and operating tools. The Data Integrations team is responsible for the acquisition, conversion, cleansing, reconciliation, modeling, tooling, and infrastructure related to the integration of market and security master data from third-party data providers. This team plays a cr...

Through this role you have an opportunity to develop intuitive software that meets and exceeds the needs of the customer and the company. You also get to collaborate with team members to develop best practices and client requirements for the software. In this role it would be important for you to professionally maintain all codes and create updates regularly to address the customer s and company s concerns You will show your skills in analyzing and testing programs/products before formal launch to ensure flawless performance Troubleshooting coding problems quickly and efficiently will offer you a chance to grow your skills in a high-pace, high-impact environment. Software security is of prim...

In this role, your responsibilities will include:Owning the design, development, and maintece of scalable solutions for ongoing metrics, reports, analyses, dashboards, etc. to support analytical and business needs. Development of queries and visualizations for ad-hoc requests and projects, as well as ongoing reporting. Specifically, own all SQL and Tableau ad hoc, recurring, trouble-shooting. Working with team leadership to understand customer needs and concerns as well as define solutions. Translating basic business problem statements into analysis requirements. Work with internal customers to define best output based on expressed stakeholder needs. Using analytical and statistical rigor to...

1.Relevant accredited certifications for Azure, AWS, Cloud Engineering, and/or Databricks Data Engineering expertise in Big Data Management using Cloudera, Python, Hive, Scala, Data Warehouse, Data Lake, AWS, Azure 1 Designing and implementing data ingestion pipelines from multiple sources using Azure Databricks 2 Ensure data pipelines run smoothly and efficiently 3 Developing scalable and re-usable frameworks for ingesting of data sets 4 Integrating the end-to-end data pipeline - to take data from source systems to target data repositories ensuring the quality and consistency of data is maintained at all times 5 Working with event based / streaming technologies to ingest and process data 6 ...

Design, develop, and maintain scalable data pipelines and systems. Collaborate with data scientists and software engineers to integrate data solutions. Optimize and improve existing data systems for performance and reliability. Implement data security and privacy measures to protect sensitive information. Monitor and troubleshoot data pipeline issues to ensure data integrity. Required Qualifications Bachelors degree in Computer Science, Engineering, or a related field. Proven experience as a Data Engineer or in a similar role. Strong knowledge of SQL and database management systems. Experience with data pipeline and workflow management tools. Proficiency in programming languages such as Pyth...

Collaborative Solution Development- Engage with a diverse range of stakeholders, including analysts, software engineers, product managers, and customers, to understand their requirements and craft effective solutions. State-of-the-Art Pipelines and Architecture- Design, develop, deploy and operate high-quality production ELT pipelines and data architectures. Integrate data from various sources and formats, ensuring compatibility, consistency, and reliability. Design scalable architectures. Champion Data Best Practices- Proactively establish and advocate for best practices in performance, code quality, data validation, data governance, and discoverability across Adyen through mentoring, train...

Design, develop, and maintain robust data pipelines using Python Work with Pandas for data manipulation, transformation, and modeling Integrate data from various sources into Azure Data Lake , Azure Data Factory , or AWS/GCP ecosystems Build and maintain RESTful services using frameworks like Django , FastAPI , or Flask Collaborate with data scientists and analysts to support AI workflows (prompting experience is a plus) Develop and test scripts in Jupyter Notebook (preferred) or VSCode Ensure high performance, availability, and security of data systems Write clean, scalable, and well-documented code

Owning the design, development, and maintece of scalable solutions for ongoing metrics, reports, analyses, dashboards, etc. to support analytical and business needs. Development of queries and visualizations for ad-hoc requests and projects, as well as ongoing reporting. Specifically, own all SQL and Tableau ad hoc, recurring, trouble-shooting. Working with team leadership to understand customer needs and concerns as well as define solutions. Translating basic business problem statements into analysis requirements. Work with internal customers to define best output based on expressed stakeholder needs. Using analytical and statistical rigor to solve complex problems and drive business decisi...

Role: Data Engineer Years of Experience: 5-10 years Key Skills: Pyspark, SQL, Python, Data Bricks, Azure, AWS, GCP, Key responsibilities Design and build data infrastructure: Create and maintain scalable systems, databases, and data pipeline architectures to support data collection, storage, and processing. Transform and prepare data: Develop algorithms and methods to clean, combine, and transform raw data into a structured format suitable for analytics and machine learning. Ensure data quality and security: Implement data validation methods, monitor workflows, and ensure compliance with data governance and security policies. Collaborate with teams: Work with data scientists, analysts, and b...

SUMMARY Wissen Technology is Hirin g for Python + Database Developer About Wissen Technology: At Wissen Technology, we deliver niche, custom-built products that solve complex business challenges across industries worldwide. Founded in 2015, our core philosophy is built around a strong product engineering mindset ensuring every solution is architected and delivered right the first time. Today, Wissen Technology has a global footprint with 2000+ employees across offices in the US, UK, UAE, India, and Australia . Our commitment to excellence translates into delivering 2X impact compared to traditional service providers. How do we achieve this? Through a combination of deep domain knowledge, cut...

Skill: Data Engineer Experience: 7+ Years Location: Warangal, Bangalore, Chennai, Hyderabad, Mumbai, Pune, Delhi, Noida, Gurgaon, Kolkata, Jaipur, Jodhpur Notice Period: Immediate - 15 Days Job Description: Design & Build Data Pipelines Develop scalable ETL/ELT workflows to ingest, transform, and load data into Snowflake using SQL, Python, or data integration tools. Data Modeling Create and optimize Snowflake schemas, tables, views, and materialized views to support business analytics and reporting needs. Performance Optimization Tune Snowflake compute resources (warehouses), optimize query performance, and manage clustering and partitioning strategies. Data Quality & Validation Security & A...

About the Role We are seeking a highly experienced Data Architect to design, implement, and manage large-scale data architecture solutions. The ideal candidate will have deep expertise in data modeling , cloud data platforms (Azure/AWS) , and data governance , with a strong ability to architect secure, scalable, and high-performance data ecosystems. Key Responsibilities Design and develop enterprise-grade data architectures to support analytics , AI , and business intelligence initiatives. Create and maintain conceptual, logical, and physical data models aligned with organizational strategies. Architect and optimize ETL/ELT pipelines for efficient, reliable data movement and transformation. ...

Seeking a Sr. Data Validation Engineer with 5+ years’ experience in SQL, Python, data validation, and ETL pipelines. Role involves building scalable data solutions on AWS and ensuring data integrity for a growing SaaS platform.

Salary: 15 to 25 LPA Exp: 4 to 7 years Location: Gurgaon/Pune/Bengalore Notice: Immediate to 30 days..!! Job Profile: Experienced Data Engineer with a strong foundation in designing, building, and maintaining scalable data pipelines and architectures. Skilled in transforming raw data into clean, structured formats for analytics and business intelligence. Proficient in modern data tools and technologies such as SQL, T-SQL, Python, Databricks, and cloud platforms (Azure). Adept at data wrangling, modeling, ETL/ELT development, and ensuring data quality, integrity, and security. Collaborative team player with a track record of enabling data-driven decision-making across business units. As a Dat...

Position summary: We are seeking a Senior Software Development Engineer – Data Engineering with 3-5 years of experience to design, develop, and optimize data pipelines and analytics workflows using Snowflake, Databricks, and Apache Spark. The ideal candidate will have a strong background in big data processing, cloud data platforms, and performance optimization to enable scalable data-driven solutions. Key Responsibilities: Work with cloud-based data solutions (Azure, AWS, GCP). Implement data modeling and warehousing solutions. Developing and maintaining data pipelines for efficient data extraction, transformation, and loading (ETL) processes. Designing and optimizing data storage solutions...

Role & responsibilities Total Experience of 4-8years Hands on experience in python coding is must . Experience in data engineering which includes laudatory account Hands-on experience in Big Data cloud platforms like AWS(redshift, Glue, Lambda), Data Lakes, and Data Warehouses, Data Integration, data pipeline. Experience in SQL, writing code in spark engine using python,pyspark.. Experience in data pipeline and workflow management tools ( such as Azkaban, Luigi, Airflow etc.) Key Personal Attributes Business focused, Customer & Service minded Strong Consultative and Management skills Good Communication and Interpersonal skills Preferred candidate profile Interested Candidate Share your CV on...

Job Title : Senior-Level Data Engineer Healthcare Domain Location: Remote Option Experience: 5+ Years Employment Type: Full-Time About the Role We are looking for a Senior Data Engineer with extensive experience in healthcare data ecosystems and Databricks-based pipelines . The ideal candidate brings deep technical expertise in building large-scale data platforms, optimizing performance, and ensuring compliance with healthcare data standards (e.g., HIPAA, EDI, HCC). This role requires the ability to lead data initiatives, mentor junior engineers, and work cross-functionally with product, analytics, and compliance teams. Key Responsibilities Architect, develop, and manage large-scale, secure,...

Job Description We are seeking a highly skilled and experienced Machine Learning Engineer to lead the design and implementation of cutting-edge ML solutions across our organization. You will be responsible for prototyping and taking models all the way from proof-of-concept to production deployment. This role requires a deep understanding of ML algorithms, data processing pipelines, model optimization, and production-grade engineering practices. Role & responsibilities Build and validate ML prototypes to solve real business problems Develop, test, and optimize ML models using structured and unstructured data Design and implement scalable data pipelines and model serving infrastructure Continu...

Role: Solutions Architect - AWS Data and Analytics GTM Experience: 8+ Years, Minimum 5 years on AWS Work location: Mumbai/Bangalore/ Trivandrum (Hybrid) Responsibilities Be a trusted technical advisor to customers and solutions for complex Cloud & Data related technical challenges. Be a thought leader in architecture design and development of cloud data analytics solutions. Liaison with internal and external stakeholders to design optimized data analytics solutions on AWS cloud Partner with SMEs and Solutions Architects from leading cloud providers to present solutions to customers Support Quantiphi Sales and GTM teams from a technical perspective in building proposals and SOWs Lead discover...

Role - GCP Data Engineer Experience: 6~13 years Preferred - Immediate to 30 Days joiners Location - Bangalore, Chennai, Pune, Kolkata, Hyderabad Job Requirement: Have Implemented and Architected solutions on Google Cloud Platform using the components of GCP Experience with Apache Beam/Google Dataflow/Apache Spark in creating end to end data pipelines. Experience in some of the following: Python, Hadoop, Spark, SQL, Big Query, Big Table Cloud Storage, Datastore, Spanner, Cloud SQL, Machine Learning. Experience programming in Java, Python, etc. Expertise in at least two of these technologies: Relational Databases, Analytical Databases, NoSQL databases. Certified in Google Professional Data Eng...

Job Description: We are seeking a skilled Cloud Engineer to join our team. The ideal candidate will provide Level 2 (L2) and Level 3 (L3) operational support for our cloud infrastructure. You will be responsible for monitoring data pipelines to ensure optimal performance and reliability. Additionally, you will work on enhancing automation processes through AIOps, implementing solutions that leverage artificial intelligence for IT operations. You will collaborate with cross-functional teams to troubleshoot and resolve issues, improve system performance, and deploy new solutions in the cloud environment. Responsibilities: - Provide L2 and L3 operational support for cloud services and infrastru...

Role & responsibilities Preferred candidate profile