1166 Data Pipeline Jobs - Page 2

We are looking for a highly skilled Machine Learning Engineer with 2 to 7 years of experience to join our team at Apple India Pvt Ltd. The ideal candidate will have expertise in machine learning and natural language processing. Roles and Responsibility Design, develop, and deploy machine learning models using various algorithms and techniques. Collaborate with cross-functional teams to integrate machine learning solutions into existing systems. Develop and maintain large-scale data pipelines and architectures. Conduct research and experimentation to improve model performance and efficiency. Work closely with data engineers to ensure seamless data flow and processing. Implement automated test...

Job Title: Senior Data Engineer (Kafka & AWS) Responsibilities: Develop and maintain real-time data pipelines using Apache Kafka (MSK or Confluent) and AWS services . Configure and manage Kafka connectors , ensuring seamless data flow and integration across systems. Demonstrate strong expertise in the Kafka ecosystem , including producers, consumers, brokers, topics, and schema registry. Design and implement scalable ETL/ELT workflows to efficiently process large volumes of data. Optimize data lake and data warehouse solutions using AWS services such as Lambda, S3, and Glue . Implement robust monitoring, testing, and observability practices to ensure reliability and performance of data platf...

Key Responsibilities Design, develop, and optimize ETL pipelines using PySpark on Google Cloud Platform (GCP) . Work with BigQuery , Cloud Dataflow , Cloud Composer (Apache Airflow) , and Cloud Storage for data transformation and orchestration. Develop and optimize Spark-based ETL processes for large-scale data processing . Implement best practices for data governance, security, and monitoring in a cloud environment. Collaborate with data engineers, analysts, and business stakeholders to understand data requirements. Troubleshoot performance bottlenecks and optimize Spark jobs for efficient execution. Automate data workflows using Apache Airflow or Cloud Composer . Ensure data quality, valid...

Role: Sr AWS Data engineer - Data Migration Experience: 6+ years Location: Hyderabad Fission Labs , headquartered in Sunnyvale, with offices in Dallas & Hyderabad, Fission Labs is a leading software development company, specializing in crafting flexible, agile, and scalable solutions that propel businesses forward. With a comprehensive range of services, including product development, cloud engineering, big data analytics, QA, DevOps consulting, and AI/ML solutions, we empower clients to achieve sustainable digital transformation that aligns seamlessly with their business goals. Responsibilities Experience leading and driving large scale data migration projects Experience in working with all...

Job Description Job Summary: We are looking for a proactive and technically strong Lead Gen AI Engineer 10+ years exp to lead a high-performing offshore team working on Gen AI solutions in the real estate domain. This role requires close collaboration with the Gen AI Architect and direct accountability for delivery, technical execution, and mentoring the team across various Gen AI initiatives. Key Responsibilities: - Lead the offshore team in developing, testing, and deploying Gen AI and NLP-based applications. - Collaborate closely work with the Lead Architect to translate high-level designs into actionable workstreams. - Develop and guide the team on best practices in GPT-based development...

Role Proficiency: This role requires proficiency in data pipeline development including coding and testing data pipelines for ingesting wrangling transforming and joining data from various sources. Must be skilled in ETL tools such as Informatica Glue Databricks and DataProc with coding expertise in Python PySpark and SQL. Works independently and has a deep understanding of data warehousing solutions including Snowflake BigQuery Lakehouse and Delta Lake. Capable of calculating costs and understanding performance issues related to data solutions. Outcomes: Act creatively to develop pipelines and applications by selecting appropriate technical options optimizing application development mainten...

Role Proficiency: This role requires proficiency in data pipeline development including coding and testing data pipelines for ingesting wrangling transforming and joining data from various sources. Must be adept at using ETL tools such as Informatica Glue Databricks and DataProc with coding skills in Python PySpark and SQL. Works independently and demonstrates proficiency in at least one domain related to data with a solid understanding of SCD concepts and data warehousing principles. Outcomes: Collaborate closely with data analysts data scientists and other stakeholders to ensure data accessibility quality and security across various data sources.rnDesign develop and maintain data pipelines...

We are looking for a skilled and experienced Databricks Engineer to join our team at our Hyderabad (Madhapur) office. The ideal candidate will have a strong background in data engineering, hands-on experience with Databricks, and a solid understanding of cloud platforms (AWS, Azure, or GCP). This is a great opportunity to work on cutting-edge data projects in a collaborative and fast-paced environment. Key Responsibilities : - Design and implement scalable and efficient data pipelines, ETL/ELT processes, and data integration solutions using Databricks. - Work with large-scale structured and unstructured datasets using Python, PySpark, SQL, and NoSQL technologies. - Collaborate with cross-fun...

We’re looking for a talented Business Intelligence Developer with 3 to 4 years of experience in Power BI . The ideal candidate will design data pipelines using Power BI Dataflows, develop and maintain dynamic dashboards and reports. A strong analytical mindset and the ability to turn raw data into actionable insights are essential.

Role Overview: As an ETL Developer at the company, you will be responsible for developing and maintaining ETL processes using Python and PySpark. Your role will involve designing and implementing data pipelines and workflows, optimizing data processing applications, and collaborating with data scientists and analysts to understand data requirements. You will play a key role in ensuring data integrity and consistency across systems, writing clean, scalable, and maintainable code, as well as monitoring and troubleshooting data processing jobs. Key Responsibilities: - Get to the heart of customer issues, diagnose problem areas, design innovative solutions, and facilitate deployment resulting in...

Lead AI & Data strategy, manage engineering teams, define enterprise data & AI architecture, drive innovation through GenAI and advanced analytics, and ensure scalable, compliant, cloud-native delivery. Required Candidate profile Tech leader with 12–20 yrs exp managing AI, ML & Data teams. Proven expertise in cloud data platforms, AI frameworks, strategy, and stakeholder management in enterprise-scale environments.

Backend SQL Dev (7–10 yrs), Data Curation, ETL (PL/SQL,T-SQL), Salesforce/NetSuite, Data Pipelines, Dimensional Modeling, Query Optimization, Python/Shell, Jira/GitHub. C2H via TE Infotech, Convertible 2 Permanent, Bengaluru @ssankala@toppersedge.com

Salary: 35 to 50 LPA Exp: 7 to 12 years Location: Gurgaon/Pune/Bengalore Notice: Immediate to 30 days..!! Job Profile: Experienced Data Engineer with a strong foundation in designing, building, and maintaining scalable data pipelines and architectures. Skilled in transforming raw data into clean, structured formats for analytics and business intelligence. Proficient in modern data tools and technologies such as SQL, T-SQL, Python, Databricks, and cloud platforms (Azure). Adept at data wrangling, modeling, ETL/ELT development, and ensuring data quality, integrity, and security. Collaborative team player with a track record of enabling data-driven decision-making across business units. As a Da...

Job role: Data Engineer Location: Hyderabad/Chennai (Hybrid) Employment Type: Full-time Team: Data Engineering No-of Positions: 4 JD: Hands-on expertise in GCP data stack: BigQuery, Dataflow (Apache Beam), Dataproc, Cloud Storage, Pub/Sub, Cloud Composer (Airflow) . Strong Spark (PySpark or Scala) for batch processing on Dataproc. Solid Airflow DAG design (idempotent tasks, backfills, retries, SLAs). Advanced SQL and data modeling (star/snowflake, slowly changing dimensions, partition strategies). Proficiency in Python (preferred) or Scala/Java for data engineering. Experience with Git and CI/CD (Cloud Build/GitHub Actions/GitLab CI). Familiarity with security & governance on GCP (IAM, servi...

Role & responsibilities Design and build robust, maintainable software systems to support data pipelines, analytics, and AI applications. Collaborate with domain experts to scope, build, and iterate on AI-driven tools tailored to business needs. Develop efficient data ingestion, transformation, and validation pipelines using Python, SQL, and modern libraries. Integrate large language models (LLMs) and generative AI APIs (e.g., OpenAI, HuggingFace) into internal tooling. Create clean, intuitive user interfaces and data visualizations with libraries like Plotly and Dash. Optimize database access and performance for large structured and semi-structured datasets. Ensure tools are scalable, secur...

Job Summary Data Engineer II perform development activities within a data engineering team and help guide, onboard, and train Data Engineer I. You will work closely with account management, ETL, data warehouse, business intelligence, and reporting teams as you develop data pipelines and enhancements and investigate and troubleshoot issues. You possess an understanding of multiple data structures, including relational and non-relational data models. Essential Functions/Responsibilities/Duties Extracting, cleansing, and loading data. Building data pipelines using Snowflake, SQL, Kafka, and other technologies. Investigation and documentation of new data sets Triage incoming bugs and incidents. ...

The Role We are seeking an experienced Senior Software Data Engineer to join the Data Integrations Team, a critical component of the Addepar Platform team. The Addepar Platform is a comprehensive data fabric that provides a single source of truth for our product set, encompassing a centralized and self-describing repository, API driven data services, integration pipeline, analytics infrastructure, warehousing solutions, and operating tools. The Data Integrations team is responsible for the acquisition, conversion, cleansing, reconciliation, modeling, tooling, and infrastructure related to the integration of market and security master data from third-party data providers. This team plays a cr...

Through this role you have an opportunity to develop intuitive software that meets and exceeds the needs of the customer and the company. You also get to collaborate with team members to develop best practices and client requirements for the software. In this role it would be important for you to professionally maintain all codes and create updates regularly to address the customer s and company s concerns You will show your skills in analyzing and testing programs/products before formal launch to ensure flawless performance Troubleshooting coding problems quickly and efficiently will offer you a chance to grow your skills in a high-pace, high-impact environment. Software security is of prim...

In this role, your responsibilities will include:Owning the design, development, and maintece of scalable solutions for ongoing metrics, reports, analyses, dashboards, etc. to support analytical and business needs. Development of queries and visualizations for ad-hoc requests and projects, as well as ongoing reporting. Specifically, own all SQL and Tableau ad hoc, recurring, trouble-shooting. Working with team leadership to understand customer needs and concerns as well as define solutions. Translating basic business problem statements into analysis requirements. Work with internal customers to define best output based on expressed stakeholder needs. Using analytical and statistical rigor to...

1.Relevant accredited certifications for Azure, AWS, Cloud Engineering, and/or Databricks Data Engineering expertise in Big Data Management using Cloudera, Python, Hive, Scala, Data Warehouse, Data Lake, AWS, Azure 1 Designing and implementing data ingestion pipelines from multiple sources using Azure Databricks 2 Ensure data pipelines run smoothly and efficiently 3 Developing scalable and re-usable frameworks for ingesting of data sets 4 Integrating the end-to-end data pipeline - to take data from source systems to target data repositories ensuring the quality and consistency of data is maintained at all times 5 Working with event based / streaming technologies to ingest and process data 6 ...

Design, develop, and maintain scalable data pipelines and systems. Collaborate with data scientists and software engineers to integrate data solutions. Optimize and improve existing data systems for performance and reliability. Implement data security and privacy measures to protect sensitive information. Monitor and troubleshoot data pipeline issues to ensure data integrity. Required Qualifications Bachelors degree in Computer Science, Engineering, or a related field. Proven experience as a Data Engineer or in a similar role. Strong knowledge of SQL and database management systems. Experience with data pipeline and workflow management tools. Proficiency in programming languages such as Pyth...

Collaborative Solution Development- Engage with a diverse range of stakeholders, including analysts, software engineers, product managers, and customers, to understand their requirements and craft effective solutions. State-of-the-Art Pipelines and Architecture- Design, develop, deploy and operate high-quality production ELT pipelines and data architectures. Integrate data from various sources and formats, ensuring compatibility, consistency, and reliability. Design scalable architectures. Champion Data Best Practices- Proactively establish and advocate for best practices in performance, code quality, data validation, data governance, and discoverability across Adyen through mentoring, train...

Design, develop, and maintain robust data pipelines using Python Work with Pandas for data manipulation, transformation, and modeling Integrate data from various sources into Azure Data Lake , Azure Data Factory , or AWS/GCP ecosystems Build and maintain RESTful services using frameworks like Django , FastAPI , or Flask Collaborate with data scientists and analysts to support AI workflows (prompting experience is a plus) Develop and test scripts in Jupyter Notebook (preferred) or VSCode Ensure high performance, availability, and security of data systems Write clean, scalable, and well-documented code

Owning the design, development, and maintece of scalable solutions for ongoing metrics, reports, analyses, dashboards, etc. to support analytical and business needs. Development of queries and visualizations for ad-hoc requests and projects, as well as ongoing reporting. Specifically, own all SQL and Tableau ad hoc, recurring, trouble-shooting. Working with team leadership to understand customer needs and concerns as well as define solutions. Translating basic business problem statements into analysis requirements. Work with internal customers to define best output based on expressed stakeholder needs. Using analytical and statistical rigor to solve complex problems and drive business decisi...

Role: Data Engineer Years of Experience: 5-10 years Key Skills: Pyspark, SQL, Python, Data Bricks, Azure, AWS, GCP, Key responsibilities Design and build data infrastructure: Create and maintain scalable systems, databases, and data pipeline architectures to support data collection, storage, and processing. Transform and prepare data: Develop algorithms and methods to clean, combine, and transform raw data into a structured format suitable for analytics and machine learning. Ensure data quality and security: Implement data validation methods, monitor workflows, and ensure compliance with data governance and security policies. Collaborate with teams: Work with data scientists, analysts, and b...