1586 Data Pipeline Jobs - Page 36

Develop full-stack forecasting and simulation tools using React, Node.js, and Python on AWS Modernize Excel/VBA tools into scalable cloud-native web applications Deploy solutions efficiently with AWS services, CI/CD, and infrastructure-as-code tools Food allowance Annual bonus Health insurance Provident fund

Role & responsibilities Summary In this role, you will be a part of the centralised global office based in India and work closely with each of our markets globally to understand the clients communication objectives, access multiple data sources and visualise it using Tableau / Datorama, support on of ETL process using MSSQL / Alteryx Flow and has sound knowledge of Excel VBA. Key responsibilities Own the design, development, and maintenance of ongoing metrics, reports, analyses, dashboards on the key drivers of our business Partner with operations/business teams to consult, develop and implement KPIs, automated reporting/process solutions and data infrastructure improvements to meet business...

We are looking for a Software Engineering Manager to join our team at Collective Data Solutions (The Collective) , part of Kochava. This role involves both technical contributions, product development and team management , with a focus on building and scaling high-performance distributed systems for real-time data solutions in AdTech. Responsibilities : Lead and mentor a team of backend engineers delivering scalable, high-quality solutions. Technical Contributions in design, architecture, product roadmap and coding Drive roadmap execution and ensure delivery timelines. Build and maintain distributed systems and data pipelines at scale. Collaborate with global stakeholders to define technical...

Preferred candidate profile 7-11 years of Professional experience designing/building/maintaining highly available data and analytics platform. Hands-on experience with AWS or similar cloud platform building data engineering solutions for analytics and science. Must have experience building complex data pipelines batch and/or real time event-based processing Strong experience in designing, building and maintaining data warehouse in Redshift or similar cloud-based solutions. Experience building complex ETL using Spark (Scala or Python) for event based big data processing. Must have strong hands-on experience in SQL. Strong hands-on experience in modern scripting languages using Python. Strong ...

Hello Candidate, Greetings from Hungry Bird IT Consulting Services Pvt Ltd. We are hiring Lead Data Engineer for our client. Job Title: Lead Data Engineer Location: Hyderabad Job Type: Full-Time Experience: 5+ years (2+ years in a leadership role) Role Overview: We are looking for a highly skilled and experienced Lead Data Engineer to oversee the design, development, and management of scalable data infrastructure and pipelines. You will work closely with cross-functional teams, lead a team of engineers, and drive best practices in data engineering using modern cloud and big data technologies like Databricks , AWS , Airflow , and FastAPI . Key Responsibilities: Lead the design, development, a...

Must Have Skills Total Experience Data Engineer Python AWS Airflow API Data pipelines Job Description (Full-stack Data Engineer) Duties Understand governance and adopter APIs, and how those support data lineage and veracity. Review object model, metadata, for data sourcing and governance, creating integration patterns. Work with data partners to gain required governance insights and data catalogue requirements. Provide subject matter expertise on the integration needs for project deliveries, scope, acceptance, installation, and deployment. Design and develop pipelines and reports to visualize analysis results using BI tools for internal governance team use. Contribute to the design, developm...

Responsibilities: * Design, implement & optimize Azure infrastructure solutions. * Collaborate with cross-functional teams on project delivery. * Manage Azure networks, VMs, storage, DAF, DBs & data pipelines. *Azure Data Migration Food allowance Health insurance Provident fund Annual bonus

Location: Pune, Onsite(WFO) Department: Engineering / Data & AI Reports To: Associate Director of Engineering Job Summary We are seeking an Associate Machine Learning Engineer of exposure and experience in developing AI solutions, including Generative AI, classical ML models, and data engineering pipelines. You will play a hands-on role in building and deploying scalable ML and GenAI applications. Key Responsibilities - Generative AI Development - Design and implement applications using large language models (LLMs). - Optimize prompts using techniques such as Chain of Thought, Few-Shot Learning, and RAG. - Build scalable APIs and pipelines for GenAI-powered features. Machine Learning Enginee...

Urgent Need for Databricks Engineer with 5-10 Yrs of experience for Chennai & Pune Location, need candidates who can join us in short notice . 1. Overview We are building a scalable data ingestion and streaming platform that ingests change data capture (CDC) events from diverse source systems (databases and applications), processes them in real time and lands curated data into our analytics lake. The platform uses Confluent connectors (Debezium/Oracle CDC) to emit Parquet files into cloud storage and leverages Databricks Auto Loader to incrementally ingest, deduplicate and write this data into Delta Lake Bronze table. To ensure broad applicability, the following job description emphasizes ge...

We are looking for a skilled and experienced Databricks Engineer to join our team at our Hyderabad (Madhapur) office. The ideal candidate will have a strong background in data engineering, hands-on experience with Databricks, and a solid understanding of cloud platforms (AWS, Azure, or GCP). This is a great opportunity to work on cutting-edge data projects in a collaborative and fast-paced environment. Key Responsibilities : - Design and implement scalable and efficient data pipelines, ETL/ELT processes, and data integration solutions using Databricks. - Work with large-scale structured and unstructured datasets using Python, PySpark, SQL, and NoSQL technologies. - Collaborate with cross-fun...

Job Title: GCP Data Engineer Location: Pune, India Experience: 3 to 7 Years Job Type: Full-Time Job Summary: We are looking for a highly skilled GCP Data Engineer with 3 to 7 years of experience to join our data engineering team in Pune . The ideal candidate should have strong experience working with Google Cloud Platform (GCP) , including Dataproc , Cloud Composer (Apache Airflow) , and must be proficient in Python , SQL , and Apache Spark . The role involves designing, building, and optimizing data pipelines and workflows to support enterprise-grade analytics and data science initiatives. Key Responsibilities: Design and implement scalable and efficient data pipelines on GCP , leveraging D...

Our world is transforming, and PTC is leading the way.Our software brings the physical and digital worlds together, enabling companies to improve operations, create better products, and empower people in all aspects of their business. Our people make all the difference in our success. Today, we are a global team of nearly 7,000 and our main objective is to create opportunities for our team members to explore, learn, and grow – all while seeing their ideas come to life and celebrating the differences that make us who we are and the work we do possible. Job Overview: We are looking for a motivated and detail-oriented Junior Data Engineer with at least 2 years of experience in building and main...

Key Responsibilities: Works in an agile framework to deliver solution artifacts (code, documentation, unit testing) with a focus on frontend functionality Actively contributes to populating the backlog with technical features and system enhancements Programming of robust, modular and user-friendly interfaces, development of data pipelines, using Azure DevOps as environment Aligns objectives, priorities, and business requirements with the Product Owner and Project Manager Support in conception and implementation of frontend projects, from defining the frontend architecture to programming solutions, giving support to the realization of scalable and high-performing enterprise solutions Contribu...

Join our team as a Data Engineer, where you will develop and deliver comprehensive data solutions to meet business requirements. As a member of our dynamic team, your responsibilities will include creating and optimizing data pipelines and data models for data lakes and data warehouses towards specific microservices. The role encompasses responsibility for data ingress into data lake/DWH, ensuring the content, format, and integrity sustain throughout the data lifecycle. You will develop, test, and enhance data solution components and design detailed data solution interfaces and integration specifications. Propose implementation strategies based on designed solutions, estimating effort requir...

Databricks Developer (5–7 yrs exp) skilled in Apache Spark, Design & Implementation, PySpark, Delta Lake, and cloud platforms (Azure/AWS). Must excel in data pipelines, ETL, governance, and performance tuning. Unity Catalog, ML, and CI/CD experience.

Roles and Responsibilities : - Create and maintain efficient and scalable data models as per business needs - Create and maintain optimal data pipelines against multiple data sources lie SQL, BigData on Azure / AWS cloud; - Assemble and process large, complex data sets to meet both functional and non-functional business requirements; - Analyze and improve existing data models, pipelines, related infrastructure and processes - Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics; - Monitor and improve data quality and data governance policies - Collaborate with stakehold...

• Strong proficiency in Python and data libraries like Pandas. • Experience with web frameworks like Django, FastAPI, or Flask. • Hands-on experience with MongoDB or other NoSQL databases • Proficiency in working with RESTful APIs and JSON.

Responsibilities : - Develop and maintain data pipelines using GCP. - Write and optimize queries in BigQuery. - Utilize Python for data processing tasks. - Manage and maintain SQL Server databases. Must-Have Skills : - Experience with Google Cloud Platform (GCP). - Proficiency in BigQuery query writing. - Strong Python programming skills. - Expertise in SQL Server. Good to Have : - Knowledge of MLOps practices. - Experience with Vertex AI. - Background in data science. - Familiarity with any data visualization tool.

About the Opportunity : Join a dynamic leader in the cloud data engineering sector, specializing in advanced data solutions and real-time analytics for enterprise clients. This role offers an on-site opportunity in India to work on cutting-edge AWS infrastructures where innovation is at the forefront of business transformation. This opportunity is ideal for professionals with 4+ years of proven experience in AWS data engineering, Python, and PySpark. You will contribute to designing, optimizing, and maintaining scalable data pipelines that drive business intelligence and operational efficiency. Role & Responsibilities : - Design, develop, and maintain robust AWS-based data pipelines using Py...

We are looking for a skilled and experienced Data Engineer with experience in building scalable data solutions on the Microsoft Azure ecosystem . The ideal candidate must have strong hands-on experience with Azure Databricks , along with Spark , Python and SQL expertise. Familiarity with Data Lake , Data Warehouse concepts, and end-to-end data pipelines is essential. Key Responsibilities: Lead the design end-to-end data engineering solutions across cloud platforms. Develop and oversee robust data pipelines and ETL workflows using Python and Apache Spark. Implement scalable Delta Lake solutions for structured and semi-structured data. Orchestrate complex workflows using Databricks Workflows o...

Responsibilities: * Design, develop & maintain data pipelines using Snowflake, DBT,SQL, ETL tools. * Collaborate with cross-functional teams on project requirements & deliverables. Office cab/shuttle

Responsibilities: * Collaborate with cross-functional teams on project delivery. * Design, implement & optimize Azure solutions using certified expertise. * Ensure compliance, security & scalability standards met. Food allowance Health insurance Provident fund Annual bonus

2+ Yrs of relevant experience Strong Python programming skills Experience with Generative AI / LLMs Hands-on with LangChain / Hugging Face / OpenAI APIs Knowledge of ML workflows & data pipelines Opportunity to work on cutting-edge AI solutions

2+ Yrs of relevant experience Strong Python programming skills Experience with Generative AI / LLMs Hands-on with LangChain / Hugging Face / OpenAI APIs Knowledge of ML workflows & data pipelines Opportunity to work on cutting-edge AI solutions

About this role: Aurigo is leading the charge in transforming enterprise SaaS for infrastructure, and AI is at the core of this revolution. With Aurigo Lumina, our AI copilot, we are redefining how infrastructure owners interact with Masterworks, driving smarter, faster, and more efficient decision-making. We have set up AI Labs to build AI-powered software and transform Aurigo into an AI-first company. We have AI initiatives across the products and platforms. As Platform and Applied AI Manager based in Bengaluru, you will lead a world-class team to build cutting-edge products/solutions, production grade AI/Gen AI platforms, build AI agents while addressing critical challenges such as AI gov...