683 Apache Airflow Jobs - Page 22

As a Data Engineer at our company, you will be responsible for building and maintaining scalable data pipelines and ETL processes using Python and related technologies. Your primary focus will be on developing efficient data pipelines to handle large volumes of data and optimize processing times. Additionally, you will collaborate closely with our team of data scientists and engineers at Matrix Space. To qualify for this role, you should have 2-5 years of experience in data engineering or a related field, with a strong proficiency in Python programming. You must be well-versed in libraries such as Pandas, NumPy, and SQL Alchemy, and have hands-on experience with data engineering tools like A...

You are a talented and driven Machine Learning Engineer with 2-5 years of experience, looking to join a dynamic team in Chennai. Your expertise lies in machine learning principles and hands-on experience in building, deploying, and managing ML models in production environments. In this role, you will focus on MLOps practices and orchestration to ensure robust, scalable, and automated ML pipelines. Your responsibilities will include designing, developing, and implementing end-to-end MLOps pipelines for deploying, monitoring, and managing machine learning models in production. You will use orchestration tools such as Apache Airflow, Kubeflow, AWS Step Functions, or Azure Data Factory to automa...

At Rhythm, our values serve as the cornerstone of our organization. We are deeply committed to customer success, fostering innovation, and nurturing our employees. These values shape our decisions, actions, and interactions, ensuring that we consistently create a positive impact on the world around us. Rhythm Innovations is currently looking for a skilled and enthusiastic Machine Learning (ML) Developer to conceptualize, create, and implement machine learning models that enhance our supply chain risk management and other cutting-edge solutions. As an ML Developer, you will collaborate closely with our AI Architect and diverse teams to construct intelligent systems that tackle intricate busin...

DifferentByte Technologies Pvt Ltd, a GENAI consultancy firm located in Cochin, Kerala, is in immediate need of a talented Python Developer/Backend Developer to join their team. The primary focus of this position is to contribute to the development of an AI-based high-frequency trading platform. This role presents a unique opportunity for individuals looking to establish a rewarding career in a technology-oriented trading organization. As a Python Developer at DifferentByte Technologies Pvt Ltd, your responsibilities will include writing effective and scalable code, enhancing back-end components to optimize responsiveness and performance, integrating user-facing features into applications, a...

We empower our people to stay resilient and relevant in a constantly changing world. We are looking for individuals who are always seeking creative ways to grow and learn, individuals who aspire to make a real impact, both now and in the future. If this resonates with you, then you would be a valuable addition to our dynamic international team. We are currently seeking a Senior Software Engineer - Data Engineer (AI Solutions). In this role, you will have the opportunity to: - Design, build, and maintain data pipelines to cater to the requirements of various stakeholders, including software developers, data scientists, analysts, and business teams. - Ensure that the data pipelines are modular...

Golden Eagle IT Technologies Pvt. Ltd. is looking for a skilled Data Engineer with 2 to 4 years of experience to join the team in Indore. The ideal candidate should have a solid background in data engineering, big data technologies, and cloud platforms. As a Data Engineer, you will be responsible for designing, building, and maintaining efficient, scalable, and reliable data pipelines. You will be expected to develop and maintain ETL pipelines using tools like Apache Airflow, Spark, and Hadoop. Additionally, you will design and implement data solutions on AWS, leveraging services such as DynamoDB, Athena, Glue Data Catalog, and SageMaker. Working with messaging systems like Kafka for managin...

We are looking for a Data Engineer to join our data team. You will be responsible for managing our master data set, developing reports, and troubleshooting data issues. To excel in this role, attention to detail, experience as a data analyst, and a deep understanding of popular data analysis tools and databases are essential. Your responsibilities include: - Building, maintaining, and managing data pipelines for efficient data flow between systems. - Collaborating with stakeholders to design and manage customized data pipelines. - Testing various ETL (Extract, Transform, Load) tools for data ingestion and processing. - Assisting in scaling the data infrastructure to meet the organization's g...

Job Description 15 to 18 years. with at-least 3 to 4 years as expertise in ETL, data engineering and Cloud Technologies, with a proven ability to orchestrate cutting-edge technology to connect various applications within the cloud environment in a large development project. Primary Technical Skills: ETL, Apache Spark, AWS EMR, EKS, Serverless, Data Engineering, Distributed Computing, Data Lineage, Apache Airflow, Java 17+, Springboot/ Quarkus, Hibernate ORM, REST, Postgres or any RDBMS, Microservices, Cloud-native development, Secondary Technical Skills Devops : Docker, Kubernetes, CI / CD stack Jenkins or Gitlab CI, Maven, GIT, SonarQube, Nexus, AWS, expertise in at least one Data Engineeri...

As a Data Engineer specializing in ETL, you should possess a minimum of 7 to 8 years of relevant experience in the field. This position is open across Pan India, and immediate joiners are highly preferred. You will be expected to demonstrate expertise in a range of mandatory skills, including ETL Developer, Synapse, Pyspark, ADF, SSIS, Databricks, SQL, Apache Airflow, and proficiency in Azure & AWS. It is important to note that proficiency in all the mentioned skills is a prerequisite for this role. The selection process for this position involves a total of three rounds - L1 with the External Panel, L2 with the Internal Panel, and L3 with the Client Round. Your responsibilities will include...

We are searching for a Senior Data Engineer with significant experience in developing ETL processes utilizing PySpark Notebooks and Microsoft Fabric, as well as supporting existing legacy SQL Server environments. The perfect candidate will have a solid foundation in Spark-based development, showcase advanced SQL skills, and feel at ease working autonomously, collaboratively within a team, or guiding other developers when necessary, all while possessing excellent communication abilities. The ideal candidate will also demonstrate expertise with Azure Data Services, such as Azure Data Factory, Azure Synapse, or similar tools, familiarity with creating DAG's, implementing activities, and running...

We are looking for a skilled and motivated Data Engineer with at least 4 years of experience in GCP, Teradata, and Data Warehousing. The ideal candidate should have hands-on expertise in developing robust data engineering solutions on Google Cloud Platform (GCP) and working experience with Teradata. You must be proficient in designing and automating scalable data pipelines and possess excellent leadership, communication, and collaboration skills. Your responsibilities will include analyzing source systems, profiling data, and resolving data quality issues. You will be required to gather and comprehend business requirements for data transformation, design, develop, test, and deploy ETL/data p...

You have the opportunity to apply for the position of Senior ETL and Feature Engineer at PrivaSapien, based in Bangalore. PrivaSapien is at the forefront of Privacy Enhancing & Responsible AI Technologies, where you will play a crucial role in setting up the big data ecosystem for the world's first privacy red teaming and blue teaming platform. As an individual contributor, you will work on cutting-edge privacy platform requirements with clients globally, spanning across various industry verticals. Joining as one of the early employees, you will receive a significant ESOP option and collaborate with brilliant minds from prestigious institutions such as IISc and IIMs. Your responsibilities wi...

As a Data Engineer at our company, your primary responsibility will be the development and maintenance of scalable and efficient data pipelines and ETL processes using Python and related technologies. You will play a crucial role in optimizing the performance of these pipelines and queries to handle large volumes of data and improve processing times. Collaboration is key in this role, as you will closely work with our team of data scientists and engineers at Matrix Space. To excel in this position, you should have 2-5 years of experience in data engineering or a related field with a strong focus on Python. Proficiency in Python programming is a must, including knowledge of libraries such as ...

You are a Sr. Data Engineer with a strong background in building ELT pipelines and expertise in modern data engineering practices. You are experienced with Databricks and DBT, proficient in SQL and Python, and have a solid understanding of data warehousing methodologies such as Kimball or Data Vault. You are comfortable working with DevOps tools, particularly within AWS, Databricks, and GitLab. Your role involves collaborating with cross-functional teams to design, develop, and maintain scalable data infrastructure and pipelines using Databricks and DBT. Your responsibilities include designing, building, and maintaining scalable ELT pipelines for processing and transforming large datasets ef...

As a Data Engineer at Synoptek, you will be responsible for designing, developing, and maintaining robust and scalable data pipelines on the Google Cloud Platform (GCP). You will leverage your hands-on experience with GCP services such as BigQuery, Jitterbit, Cloud Dataflow, Cloud Pub/Sub, and Cloud Storage to build efficient data processing solutions. Collaborating with cross-functional teams, you will translate their data needs into technical requirements, ensuring data quality, integrity, and security throughout the data lifecycle. Your role will involve developing and optimizing ETL/ELT processes to extract, transform, and load data from various sources into data warehouses and data lake...

As a Data Engineer, you will be responsible for designing and building efficient data pipelines using Azure Databricks (PySpark). You will implement business logic for data transformation and enrichment at scale, as well as manage and optimize Delta Lake storage solutions. Additionally, you will develop REST APIs using FastAPI to expose processed data and deploy them on Azure Functions for scalable and serverless data access. Your role will also involve developing and managing Airflow DAGs to orchestrate ETL processes, ingesting and processing data from various internal and external sources on a scheduled basis. You will handle data storage and access using PostgreSQL and MongoDB, writing op...

As a Senior Developer specializing in SnapLogic and Apache Airflow, you will be responsible for designing, developing, and maintaining enterprise-level data integration solutions. Your expertise in ETL development, workflow orchestration, and cloud technologies will be crucial for automating data workflows, optimizing performance, and ensuring the reliability and scalability of data systems. Your key responsibilities will include designing, developing, and managing ETL pipelines using SnapLogic to ensure efficient data transformation and integration across various systems and applications. You will leverage Apache Airflow for workflow automation, job scheduling, and task dependencies to ensu...

The Digital Success Engineering team at our organization is comprised of a diverse group of distributed software engineers, architects, and engineering managers who collaborate to develop Unified Experiences for our Trailblazers. As the pioneers of our products, we leverage Salesforce technology to create an intuitive and expert self-service platform that drives Trailblazer Success. As a Marketing Cloud Engineer (SMTS) in our Customer Engagement Engineering team, you will play a crucial role in providing technical solutions and support to our customers and business users. We are looking for someone with a strong background in the Salesforce Marketing Cloud platform, exceptional technical ski...

As a Data Engineer at our company, you will play a crucial role in designing, developing, and optimizing data pipelines and workflows in a cloud-based environment. Your expertise in PySpark, Snowflake, and AWS will be key as you leverage these technologies for data processing and analytics. Your responsibilities will include designing and implementing scalable ETL pipelines using PySpark on AWS, developing and optimizing data workflows for Snowflake integration, and managing and configuring various AWS services such as S3, Lambda, Glue, EMR, and Redshift. Collaboration with data analysts and business teams to understand requirements and deliver solutions will be essential, along with ensurin...

You are a Data Engineer with 3+ years of experience, proficient in SQL and Python development. You will be responsible for designing, developing, and maintaining scalable data pipelines to support ETL processes using tools like Apache Airflow, AWS Glue, or similar. Your role involves optimizing and managing relational and NoSQL databases such as MySQL, PostgreSQL, MongoDB, or Cassandra for high performance and scalability. You will write advanced SQL queries, stored procedures, and functions to efficiently extract, transform, and analyze large datasets. Additionally, you will implement and manage data solutions on cloud platforms like AWS, Azure, or Google Cloud, utilizing services such as R...

You should have 3 to 10 years of experience in AI development and be located in Coimbatore. Immediate joiners are preferred. A minimum of 2 years of experience in core Gen AI is required. As an AI Developer, your responsibilities will include designing, developing, and fine-tuning Large Language Models (LLMs) for various in-house applications. You will implement and optimize Retrieval-Augmented Generation (RAG) techniques to enhance AI response quality. Additionally, you will develop and deploy Agentic AI systems capable of autonomous decision-making and task execution. Building and managing data pipelines for processing, transforming, and feeding structured/unstructured data into AI models ...

As a Cloud Developer specializing in Google Cloud Platform (GCP), Python, and Apache Airflow, you will be responsible for designing and developing cloud-native applications and APIs. Your expertise in deploying microservices on GCP Cloud Run, utilizing Docker for containerization, and managing data workflows with Apache Airflow will be crucial in optimizing scalable cloud services. Collaborating with cross-functional teams in an Agile environment, you will ensure the development, maintenance, and security of cloud infrastructure using various GCP services. Your key responsibilities will include designing and developing cloud-native applications and APIs using Python and frameworks like Djang...

As a Data Analyst in our dynamic fintech environment, your primary responsibility will be to extract, clean, and analyze large datasets to identify patterns and trends. You will play a crucial role in developing and maintaining dashboards and reports to monitor business performance. Collaborating with cross-functional teams, you will work towards improving data accuracy and accessibility. Conducting deep-dive analysis on customer behavior, transactions, and engagement will enable you to enhance retention and acquisition strategies. Additionally, you will be expected to identify potential risks, anomalies, and growth opportunities using data. To excel in this role, you should hold a Bachelor'...

We are seeking an experienced Apache Airflow Subject Matter Expert (SME), (Contract , Remote - India ) to join our Data Engineering team . You will be responsible for optimizing Airflow environments, building scalable orchestration frameworks, and supporting enterprise-scale data pipelines, while collaborating with cross-functional teams. Skills: Optimize and fine-tune existing Apache Airflow environments, addressing performance and reliability. Design and develop scalable, modular, and reusable Airflow DAGs for complex data workflows. Integrate Airflow with cloud-native services such as data factories, compute platforms, storage, and analytics . Develop and maintain CI/CD pipelines for DAG ...

At American Express, our culture is built on a 175-year history of innovation, shared values, and Leadership Behaviors, with an unwavering commitment to supporting our customers, communities, and colleagues. As a member of Team Amex, you will receive comprehensive support for your holistic well-being and numerous opportunities to enhance your skills, develop leadership qualities, and advance your career. Your voice and ideas hold significance here, making a tangible impact as we collectively shape the future of American Express. Enterprise Architecture, situated within the Chief Technology Office at American Express, plays a crucial role as a key enabler of the company's technology strategy....