10863 Airflow Jobs - Page 25

Key Responsibilities : Design, develop, and maintain scalable data pipelines using modern ETL/ELT tools and frameworks Implement efficient data models and warehousing solutions to support business intelligence initiatives Write high-quality, documented code for data integration, transformation, and automation Design and optimize database schemas and query performance Develop and maintain data quality monitoring systems and procedures Implement data governance practices and ensure data security compliance Troubleshoot complex data pipeline issues and performance bottlenecks Collaborate with stakeholders to gather requirements and translate them into technical specifications Create and maintai...

GitLab is an open-core software company that develops the most comprehensive AI-powered DevSecOps Platform, used by more than 100,000 organizations. Our mission is to enable everyone to contribute to and co-create the software that powers our world. When everyone can contribute, consumers become contributors, significantly accelerating human progress. Our platform unites teams and organizations, breaking down barriers and redefining what's possible in software development. Thanks to products like Duo Enterprise and Duo Agent Platform, customers get AI benefits at every stage of the SDLC. The same principles built into our products are reflected in how our team works: we embrace AI as a core ...

Key Skills: Informatica MDM, Informatica IICS, ETL, Informatica, Airflow, DB2, Python Roles and Responsibilities: Lead the offshore delivery team responsible for developing and maintaining integration pipelines. Design and implement robust data workflows using Informatica IDMC, Apache Airflow, and open-source ELT tools like Meltano, Singer taps, or DBT. Provide technical direction, mentorship, and code reviews to ensure delivery quality. Collaborate with onshore and client teams to gather requirements and translate them into actionable integration solutions. Own the offshore delivery scope including planning, execution, and reporting. Ensure best practices in data orchestration, error handli...

We are looking for a skilled PySpark Lead with 5-8 years of experience to join our team in Bangalore. The ideal candidate will have expertise in designing and developing ETL/ Data Engineering Pipelines using PySpark. Roles and Responsibility Design and develop scalable data pipelines using PySpark. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data processing systems using PySpark. Ensure high-quality data processing and analysis by implementing robust testing and validation frameworks. Optimize data pipeline performance and efficiency by analyzing system metrics and identifying areas for improvement. Provide technic...

Job Title:Senior Data Engineer Location: Pune, India Job Type: Full-Time As a Senior Data Engineer at Assent, you will play a critical role in advancing our data engineering practices and serving as a hands-on technical expert. You will contribute significantly to the vision and execution of our data engineering efforts, ensuring the development of secure, robust, scalable, and high-performing data platforms. Your role will involve driving key technical initiatives, collaborating on architectural decisions, and executing complex data engineering tasks to align our data infrastructure with Assent's business goals. You will actively participate in the design, development, and implementation of...

We are looking for a skilled PySpark Lead with 5 to 8 years of experience, based in Bangalore. The ideal candidate will have expertise in PySpark, SQL, ETL/Data Engineering Pipeline, Cloud understanding, CI/CD, Data Modelling, Airflow, Kafka, EMR/Databricks, Snowflake, and Hive. Roles and Responsibility Design and develop scalable data pipelines using PySpark and other relevant technologies. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data systems, ensuring high performance and reliability. Troubleshoot and resolve complex technical issues related to data processing and analysis. Implement automated testing and val...

We are looking for a skilled Data Engineer with 6 to 11 years of experience to join our team in Bangalore. The ideal candidate will have expertise in SQL, ETL, data modeling, and programming languages such as Python, C++, C#, Scala, etc. Roles and Responsibility Design and develop scalable data pipelines using Big Data/MPP analytics platforms. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale databases and data systems. Ensure data quality and integrity by implementing robust testing and validation procedures. Optimize data processing workflows for improved performance and efficiency. Participate in code reviews and cont...

Job Title: Senior Data Engineer Employment Type: Full-Time Location: Ahmedabad, Onsite Experience Required: 5+ Years About Techiebutler Techiebutler is looking for an experienced Data Engineer to develop and maintain scalable, secure data solutions. You will collaborate closely with data science, business analytics, and product development teams, deploying cutting-edge technologies and leveraging best-in-class third-party tools. You will also ensure compliance with security, privacy, and regulatory standards while aligning data solutions with industry best practices. Tech Stack Languages: SQL, Python Pipeline Orchestration: Dagster (Legacy: Airflow) Data Stores: Snowflake, Clickhouse Platfor...

Frequence is the only end-to-end platform for media companies and agencies to grow and automate their advertising sales and operations while integrating owned and operated media. Through its full-stack proposal, workflow, and campaign-management software, Frequence drives revenue with best-in-class tools to sell, optimize, and report omnichannel advertising campaigns. Frequence is a Madhive Company. Madhive is the leading independent and fully customizable operating system built to help local media organizations build profitable, differentiated, and efficient businesses. Learn more about how Madhive and Frequence work together here. We are seeking an experienced Engineering Manager to lead o...

About the Role We are seeking a highly skilled Snowflake Developer to join our team in Bangalore. The ideal candidate will have extensive experience in designing, implementing, and managing Snowflake-based data solutions. This role involves developing data architectures and ensuring the effective use of Snowflake to drive business insights and innovation. Key Responsibilities: Design and implement scalable, efficient, and secure Snowflake solutions to meet business requirements. Develop data architecture frameworks, standards, and principles, including modeling, metadata, security, and reference data. Implement Snowflake-based data warehouses, data lakes, and data integration solutions. Mana...

Job Description NIQ is looking for a Senior Data Engineer to join our Financial Services Engineering team. At NIQ, the Financial Services team uses alternative datasets to help global public equity investors (hedge funds, mutual funds, pension funds) make better investment decisions. We work with some of the largest hedge funds in the world. As a Senior Data Engineer, you will be at the cutting edge of the alternative data space where you will help maintain and improve our data infrastructure, which enables us to develop market research products and delivery data to our customers. In this role, you would also get the opportunity to work with world-class big data and cloud services, such as: ...

R1 RCM Inc. is a leading provider of technology-enabled revenue cycle management services which transform and solve challenges across health systems, hospitals and physician practices. Headquartered in Chicago, R1 is a publicly-traded organization with employees throughout the US and multiple INDIA locations. Our mission is to be the one trusted partner to manage revenue, so providers and patients can focus on what matters most. Our priority is to always do what is best for our clients, patients and each other. With our proven and scalable operating model, we complement a healthcare organization's infrastructure, quickly driving sustainable improvements to net patient revenue and cash flows ...

BlitzenX is hiring a Senior ETL Developer to lead enterprise-grade data integration initiatives across complex, high-throughput environments. This is not a maintenance role we're looking for a builder , a problem-solver , and a technical authority who can design, optimize, and deliver production-ready ETL solutions in real time. Key Responsibilities Design, build, and deploy scalable ETL pipelines across cloud and on-prem environments. Integrate structured and semi-structured data from diverse sources into centralized data platforms. Tune performance of large-scale data workflows for speed, resilience, and cost-efficiency. Collaborate with cross-functional teams to translate business rules i...

Job Title: Technical Lead, Data Engineering Location: Pune, India Job Type: Full-Time About Assent Assent is the global leader in supply chain sustainability management , empowering manufacturers to navigate complex regulations and drive responsible business practices. Our comprehensive solutions span Product Compliance, ESG, and Trade Compliance , combining powerful cloud-based software with expert services to help companies map, monitor, and manage their suppliers, products, and materials. In early 2024, we hit a huge milestone: $100M in Annual Recurring Revenue (ARR) ! This makes us the first Certified B Corporation in North Americas SaaS sustainability industry to reach this level. Whats...

About The Role Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : MongoDB Good to have skills : NA Minimum 3 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating data pipelines, ensuring data quality, and implementing...

About The Role Project Role : Cloud Services Engineer Project Role Description : Act as liaison between the client and Accenture operations teams for support and escalations. Communicate service delivery health to all stakeholders and explain any performance issues or risks. Ensure Cloud orchestration and automation capability is operating based on target SLAs with minimal downtime. Hold performance meetings to share performance and consumption data and trends. Must have skills : Hitachi Data Systems (HDS) Good to have skills : NA Minimum 3 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a Cloud Services Engineer, you will serve as a vit...

Position summary: A user shall work with the development team and responsible for development task as individual contribution .He/she should be technical sound and able to communicate with client perfectly . Key duties & responsibilities: Work as Specialist Data engineering project for E2E Analytics. Ensure Project delivery on time. Mentor other team mates and guide them. Will take the requirement from client and communicate as well. Ensure Timely documents creation for knowledge base, user guides, and other various communications systems. Ensures delivery against Business needs, team goals and objectives, i.e., meeting commitments and coordinating overall schedule. Works with large datasets...

Description: We are seeking a Staff Software Engineer(5I) (ETL) with 8-10 years of experience to join our ETL Development team. This role will report to the Engineering Manager and the candidate will be involved in the planning, design, and implementation of our centralized data warehouse solution for data acquisition, Ingestion and large data processing and automation/optimization across all the company products. About the Role: Candidate will play a crucial role in designing, developing, and leading the implementation of ETL processes and data architecture solutions. Will collaborate with various stakeholders to ensure the seamless integration, transformation, and loading of data to suppor...

Position summary: A Specialist shall work with the development team and responsible for development task as individual contribution. He/she should be able to mentor team and able to help in resolving issues. He/she should be technical sound and able to communicate with client perfectly. Key duties & responsibilities Work as Lead Developer Data engineering project for E2E Analytics. Ensure Project delivery on time. Mentor other team mates and guide them. Will take the requirement from client and communicate as well. Ensure Timely documents creation for knowledge base, user guides, and other various communications systems. Ensures delivery against Business needs, team goals and objectives, i.e...

About The Role Project Role : Application Lead Project Role Description : Lead the effort to design, build and configure applications, acting as the primary point of contact. Must have skills : Google BigQuery Good to have skills : NA Minimum 5 year of experience is required Educational Qualification : 15 years full time education Summary :As an Application Lead, you will lead the effort to design, build, and configure applications, acting as the primary point of contact. Your typical day will involve collaborating with various teams to ensure project milestones are met, facilitating discussions to address challenges, and guiding your team through the development process while ensuring align...

About The Role Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : Data Engineering Good to have skills : NA Minimum 5 year of experience is required Educational Qualification : 15 years full time education Summary :As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating data pipelines, ensuring data quality, and implem...

Expert in building and scaling data pipelines for ML/GenAI using Spark, Airflow, PostgreSQL, and cloud-native practices. Must have experience to handle end-to-end GenAI projects: NLP model usage, embeddings, pipelines Hands on experience with LLMs with Spark/Airflow ETL Hands-on experience with tool stack: GitHub, Docker, FastAPI, Redis, Ray Serve, Kubernetes and GitHub Actions. Should have experience in tokenization and embedding generation work using Hugging Face. Should be able to generate prompts for the GPT and Should be able to fine-tune GPT for a classification task. Should have experience in building a RAG pipeline using LangChain + FAISS Should be able to integrate FastAPI with Open...

7+ years of experience as a Data Engineer / Data Architect or similar role: Proven track record of designing and implementing data solutions on Google Cloud Platform. Strong analytical and problem-solving skills: Ability to analyze complex data challenges and provide innovative solutions. Excellent communication and collaboration skills: Proven ability to effectively communicate technical concepts to both technical and non-technical audiences. Passion for data and technology: A genuine interest in data, its potential, and the latest advancements in data technology. Continuous learner: A proactive and enthusiastic approach to learning new technologies and techniques. Location : Hyderabad Hybr...

Role Overview: An AWS SME with a Data Science Background is responsible for leveraging Amazon Web Services (AWS) to design, implement, and manage data-driven solutions. This role involves a combination of cloud computing expertise and data science skills to optimize and innovate business processes. Key Responsibilities: Data Analysis and Modelling: Analyzing large datasets to derive actionable insights and building predictive models using AWS services like SageMaker, Bedrock, Textract etc. Cloud Infrastructure Management: Designing, deploying, and maintaining scalable cloud infrastructure on AWS to support data science workflows. Machine Learning Implementation: Developing and deploying mach...

If you are looking for a career at a dynamic company with a people-first mindset and a deep culture of growth and autonomy, ACV is the right place for you! Competitive compensation packages and learning and development opportunities, ACV has what you need to advance to the next level in your career. We will continue to raise the bar every day by investing in our people and technology to help our customers succeed. We hire people who share our passion, bring innovative ideas to the table, and enjoy a collaborative atmosphere. Who We Are ACV is a technology company that has revolutionized how dealers buy and sell cars online. We are transforming the automotive industry. ACV Auctions Inc. (ACV)...