1578 Data Pipeline Jobs - Page 20

Loction :Mumbai Job Description: Looking for Candidates with 8 to 10 years of experience Hands on experience of implementing data pipelines using traditional DWH, Big data & Cloud ecosystem Good exposure to data architecture design, cost and size estimation Good understanding of handling Realtime/streaming pipelines Have experience of Data Quality and Data Governance Have experience of handling & interacting clients and managing vendors Having knowledge on AI/ML, GenAI is a plus Exposure of managing and leading teams

We are seeking an AI/ML Engineer to join our software project team. The ideal candidate will be responsible for designing, developing, and integrating artificial intelligence models and algorithms into applications. This role requires collaboration with data scientists and developers to implement machine learning, natural language processing, or computer vision solutions tailored to project requirements. Key Responsibilities: Design and develop AI models and algorithms for integration into applications. Collaborate with data scientists and developers to implement machine learning, natural language processing, and computer vision solutions. Ensure AI models are trained, tested, and optimized ...

Job role: Data Engineer Location: Hyderabad/Chennai (Hybrid) Employment Type: Full-time Team: Data Engineering No-of Positions: 4 JD: Hands-on expertise in GCP data stack: BigQuery, Dataflow (Apache Beam), Dataproc, Cloud Storage, Pub/Sub, Cloud Composer (Airflow) . Strong Spark (PySpark or Scala) for batch processing on Dataproc. Solid Airflow DAG design (idempotent tasks, backfills, retries, SLAs). Advanced SQL and data modeling (star/snowflake, slowly changing dimensions, partition strategies). Proficiency in Python (preferred) or Scala/Java for data engineering. Experience with Git and CI/CD (Cloud Build/GitHub Actions/GitLab CI). Familiarity with security & governance on GCP (IAM, servi...

6 years relevant in Data Engineering & Data Architecture Good understanding of Data Architecture. Experience in AWS mandatory Experience in Python & pyspark. Required Candidate profile Experience in ETL Development, ETL pipeline. Experience in Data Modelling & Data Pipeline & Data Lake. Experience in SQL. Experience in working as Individual Contributor + Lead.

SUMMARY Wissen Technology is Hirin g for Data Engineer About Wissen Technology: At Wissen Technology, we deliver niche, custom-built products that solve complex business challenges across industries worldwide. Founded in 2015, our core philosophy is built around a strong product engineering mindset ensuring every solution is architected and delivered right the first time. Today, Wissen Technology has a global footprint with 2000+ employees across offices in the US, UK, UAE, India, and Australia . Our commitment to excellence translates into delivering 2X impact compared to traditional service providers. How do we achieve this? Through a combination of deep domain knowledge, cutting-edge tech...

Job Title: GCP Data Engineer Location: Pune, India Experience: 3 to 7 Years Job Type: Full-Time Job Summary: We are looking for a highly skilled GCP Data Engineer with 3 to 7 years of experience to join our data engineering team in Pune . The ideal candidate should have strong experience working with Google Cloud Platform (GCP) , including Dataproc , Cloud Composer (Apache Airflow) , and must be proficient in Python , SQL , and Apache Spark . The role involves designing, building, and optimizing data pipelines and workflows to support enterprise-grade analytics and data science initiatives. Key Responsibilities: Design and implement scalable and efficient data pipelines on GCP , leveraging D...

Role Proficiency: This role requires proficiency in data pipeline development including coding and testing data pipelines for ingesting wrangling transforming and joining data from various sources. Must be skilled in ETL tools such as Informatica Glue Databricks and DataProc with coding expertise in Python PySpark and SQL. Works independently and has a deep understanding of data warehousing solutions including Snowflake BigQuery Lakehouse and Delta Lake. Capable of calculating costs and understanding performance issues related to data solutions. Outcomes: Act creatively to develop pipelines and applications by selecting appropriate technical options optimizing application development mainten...

Role Proficiency: This role requires proficiency in developing data pipelines including coding and testing for ingesting wrangling transforming and joining data from various sources. The ideal candidate should be adept in ETL tools like Informatica Glue Databricks and DataProc with strong coding skills in Python PySpark and SQL. This position demands independence and proficiency across various data domains. Expertise in data warehousing solutions such as Snowflake BigQuery Lakehouse and Delta Lake is essential including the ability to calculate processing costs and address performance issues. A solid understanding of DevOps and infrastructure needs is also required. Outcomes: Act creatively ...

Role Proficiency: This role requires proficiency in data pipeline development including coding and testing data pipelines for ingesting wrangling transforming and joining data from various sources. Must be adept at using ETL tools such as Informatica Glue Databricks and DataProc with coding skills in Python PySpark and SQL. Works independently and demonstrates proficiency in at least one domain related to data with a solid understanding of SCD concepts and data warehousing principles. Outcomes: Collaborate closely with data analysts data scientists and other stakeholders to ensure data accessibility quality and security across various data sources.rnDesign develop and maintain data pipelines...

Role Proficiency: This role requires proficiency in developing data pipelines including coding and testing for ingesting wrangling transforming and joining data from various sources. The ideal candidate should be adept in ETL tools like Informatica Glue Databricks and DataProc with strong coding skills in Python PySpark and SQL. This position demands independence and proficiency across various data domains. Expertise in data warehousing solutions such as Snowflake BigQuery Lakehouse and Delta Lake is essential including the ability to calculate processing costs and address performance issues. A solid understanding of DevOps and infrastructure needs is also required. Outcomes: Act creatively ...

Role Proficiency: This role requires proficiency in data pipeline development including coding and testing data pipelines for ingesting wrangling transforming and joining data from various sources. Must be skilled in ETL tools such as Informatica Glue Databricks and DataProc with coding expertise in Python PySpark and SQL. Works independently and has a deep understanding of data warehousing solutions including Snowflake BigQuery Lakehouse and Delta Lake. Capable of calculating costs and understanding performance issues related to data solutions. Outcomes: Act creatively to develop pipelines and applications by selecting appropriate technical options optimizing application development mainten...

Job description Hiring for PySpark Specialist | Data Pipeline Mandatory Skills: PySpark, Python, SQL Responsibilities A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to actively aid the consulting team in different phases of the project including problem definition, effort estimation, diagnosis, solution generation and design and deployment You will explore the alternatives to the recommended solutions based on research that includes literature surveys, information available in public domains, vendor evaluation information, etc. and build POCs You will create requirement specifications from the business needs, define the to-be-processes an...

Job description Hiring for PySpark Specialist | Data Pipeline Mandatory Skills: PySpark, Python, SQL Responsibilities A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to actively aid the consulting team in different phases of the project including problem definition, effort estimation, diagnosis, solution generation and design and deployment You will explore the alternatives to the recommended solutions based on research that includes literature surveys, information available in public domains, vendor evaluation information, etc. and build POCs You will create requirement specifications from the business needs, define the to-be-processes an...

Job description Hiring for PySpark Specialist | Data Pipeline Mandatory Skills: PySpark, Python, SQL Responsibilities A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to actively aid the consulting team in different phases of the project including problem definition, effort estimation, diagnosis, solution generation and design and deployment You will explore the alternatives to the recommended solutions based on research that includes literature surveys, information available in public domains, vendor evaluation information, etc. and build POCs You will create requirement specifications from the business needs, define the to-be-processes an...

As a Machine Learning Engineer at our company, you will be responsible for SAS to Python code conversion, acquiring skills for building machine learning models, and deploying them for production. Your role will involve feature engineering, exploratory data analysis, pipeline creation, model training, and hyperparameter tuning with both structured and unstructured datasets. Additionally, you will develop and deploy cloud-based applications, including LLM/GenAI, into production. Key Responsibilities: - Hands-on experience working with SAS to Python conversions - Strong mathematics and statistics skills - Skilled in AI-specific utilities like ChatGPT, Hugging Face Transformers, etc - Ability to...

1. Cloud Platform Management : Manage and optimize cloud infrastructure (GCP), ensuring scalability, security, and performance. 2. Data Engineering : Design and implement data pipelines, data warehousing, and data processing solutions. 3. Kubernetes and GKE : Develop and deploy applications using Kubernetes and Google Kubernetes Engine (GKE). 4. Python Development : Develop and maintain scripts and applications using Python. What You Need to Be Successful 1. Experience: 6-9 years of experience in cloud computing, data engineering, and DevOps. 2. Technical Skills: 1. Strong understanding of GCP (Google Cloud Platform) or Azure. 2. Experience with Kubernetes and GKE. 3. Proficiency in Python p...

Min 3 Yrs experience in Advance SQL, ETL/DW testing, Data Automation. Expertise in Validating data pipelines , ETL processes to ensure data accuracy, consistency, integrity. Perform data quality checks and anomaly detection to identify and rectify data issues. Good experience in Writing complex SQL Queries to validate data quality and business transformations. Good experience in reporting tools like Tableau, Power BI Any exposure on Data Automation tools like DataGaps, Query surge or any Automation frameworks will be advantage. Exposure working on Databases like SQL Server / Oracle etc is desirable. Exposure working in Agile based delivery model is desirable Good written / spoken communicati...

Design and implement data pipelines for data ingestion, transformation, and loading using Microsoft Fabric tools. Develop and maintain data warehouses and data lakes within the Fabric environment. Create and manage data models for efficient data storage and retrieval. Build and deploy data science models using Fabrics machine learning capabilities. Develop real-time analytics solutions for streaming data processing. Create and maintain Power BI reports and dashboards for data visualization and analysis. Collaborate with cross-functional teams to understand data requirements and provide data solutions. Monitor and optimize the performance of data solutions. Ensure data security and compliance...

The Microsoft Fabric role involves designing, implementing, and managing data solutions using Microsoft Fabric This includes data integration, data warehousing, data science, real-time analytics, and business intelligence The role requires a strong understanding of data engineering principles, cloud computing, and the Microsoft Fabric platform Responsibilities:Design and implement data pipelines for data ingestion, transformation, and loading using Microsoft Fabric tools Develop and maintain data warehouses and data lakes within the Fabric environment Create and manage data models for efficient data storage and retrieval Build and deploy data science models using Fabrics machine learning cap...

Responsibilities: -Experience testing AML pipelines (pipelines/jobs/components), & message-driven integrations (Service Bus/Event Hubs). -focused experience on ML/Data systems (data pipelines + model validation) Python automation for automated ML QA.

Key Responsibilities: The ideal candidate will have strong expertise in Snowflake, Hadoop ecosystem, PySpark, and SQL, and will play a key role in enabling data-driven decision-making across the organization. Design, develop, and optimize robust data pipelines using PySpark and SQL. • Implement and manage data warehousing solutions using Snowflake. Work with large-scale data processing frameworks within the Hadoop ecosystem. Collaborate with data scientists, analysts, and business stakeholders to understand data requirements. Ensure data quality, integrity, and governance across all data platforms. Monitor and troubleshoot data pipeline performance and reliability. Automate data workflows an...

Designing and developing Data Pipeline Ensure data integrity and quality Design, develop, and maintain the data models Providing clean, reliable, and timely data Define high-quality data for Data Science and Analytics use-case Required Candidate profile Expert in Complex SQL & PostgreSQL Quires, Python, AWS(ECS, EMR Cloud watch, Event Bridge, Step Functions, Far Gate), Streaming Kafka, Data Modeling, Data Build Tool(DBT) 4 an Analytical Saas Product

Job Requirements At Quest Global, it’s not just what we do but how and why we do it that makes us different. With over 25 years as an engineering services provider, we believe in the power of doing things differently to make the impossible possible. Our people are driven by the desire to make the world a better place—to make a positive difference that contributes to a brighter future. We bring together technologies and industries, alongside the contributions of diverse individuals who are empowered by an intentional workplace culture, to solve problems better and faster. Key Responsibilities Design and implement data pipelines to collect, clean, and transform data from a variety of sources. ...

Job Description: Complete data ingestion, data pipeline, data lineage , data quality , data wearhouse, data governance and data reconciliation. Essential Skills: Must have Data architect experience and knowledge. Data Architect with over 10 +years of hands-on experience in designing, developing, and managing large-scale data solutions. Proven expertise in building and optimizing ETL pipelines. Strong in data preprocessing, and enhancing data quality.Extracting events, processing large datasets (5 billion+ records) within (a Spark-Hadoop) cluster. Automated data processing tasks for DAAS (Data as a Service) project, streamlining workflow efficiency. Configured file and client setups, ensuring...

About us: We are building a modern, scalable, end-to-end automated on-premises data platform designed to handle complex data workflows, including data ingestion, ETL processes, physics-based calculations and machine learning predictions. Our platform integrates with multiple data sources, edge devices, and storage systems. We are using core Python as programming language, Docker as deployment technology and Dagster as orchestrator. We are a small cross-functional team sharing a wide range of tasks from database operations to data science. We are looking for a data platform developer with expert Python and Docker knowledge who would help to develop, maintain, and optimize our platform. Key Re...