974 Apache Airflow Jobs - Page 3

As a Sr/Lead Data Engineer at Ccube in Indore, MP (Hybrid Model), your role involves gathering and assembling large, complex sets of data to meet non-functional and functional business requirements. You will be using skills such as SQL, Python, R, Data Modeling, Data Warehousing, and AWS (S3, Athena). Additionally, you will create new data pipelines or enhance existing pipelines to accommodate non-standard data formats from customers, utilizing ETL Tools (e.g., Apache NiFi, Talend), Python (Pandas, PySpark), AWS Glue, JSON, XML, and YAML. Furthermore, you will identify, design, and implement internal process improvements, including re-designing infrastructure for greater scalability, optimiz...

As a Data Engineer at our company, your role will involve a dual focus on infrastructure maintenance and ensuring seamless onboarding of data views. You will play a critical part in maintaining stable data platform operations and facilitating efficient data integration, especially amidst complex upstream changes and fiscal year transitions. Key Responsibilities: - Perform infrastructure maintenance tasks such as Azure subscription management, Azure Infrastructure and Platform Operations, ETL pipeline monitoring, source path validation and updates, as well as proactive issue identification and resolution. - Manage data onboarding activities by integrating new data sources and views, making ad...

Role Overview: As a Senior Manager, Integrated Test Lead in Data Product Engineering & Delivery at McDonalds, your primary responsibility will be to lead comprehensive testing strategy and execution for complex data engineering pipelines and product delivery initiatives. You will drive quality assurance across integrated systems, data workflows, and customer-facing applications while coordinating cross-functional testing efforts. Key Responsibilities: - Design and implement end-to-end testing frameworks for data pipelines, ETL / ELT processes, and analytics platforms - Ensure test coverage across ETL / ELT, data transformation, lineage and consumption layers - Develop integrated testing stra...

Job Description: At EY, you will have the opportunity to build a career unique to you, with global scale, support, inclusive culture, and technology to help you become the best version of yourself. Your unique voice and perspective are essential in contributing to making EY even better. Join the team to create an exceptional experience for yourself and a better working world for all. Role Overview: - Writes ETL (Extract / Transform / Load) processes, designs database systems, and develops tools for real-time and offline analytic processing. - Troubleshoots software and processes for data consistency and integrity. Integrates data from various sources to generate insights for business partner...

As a Cloud Solution Delivery Sr Advisor at NTT DATA, you will play a crucial role in leading and directing a team of engineers to engineer end-to-end data solutions using AWS services. Your responsibilities will include cataloguing data, collaborating with cross-functional teams to translate business requirements into technical solutions, and providing documentation for downstream teams. You will also be involved in testing tooling, delivering CI/CD and IaC, defining re-usable pipelines using DBT projects, and ensuring best practices in version control and Agile methodologies. Key Responsibilities: - Engineer end-to-end data solutions using AWS services such as Lambda, S3, Snowflake, DBT, an...

Role Overview: At Techjays, as a Data Analytics Engineer, you will be responsible for building scalable, data-driven solutions to support real-time decision-making and provide deep business insights. Your role will involve designing and delivering analytics systems using tools such as Power BI, Snowflake, and SQL. You will collaborate with various teams across the organization to ensure data-informed decisions are made confidently. This position offers the opportunity to work on exciting projects that redefine industries and contribute to solutions with a real-world impact. Key Responsibilities: - Develop and maintain scalable, robust ETL/ELT data pipelines across structured and semi-structu...

As an AI Automation Engineer, you will be responsible for designing, developing, and implementing intelligent automation solutions using AI, machine learning, and automation tools. Your role will involve streamlining workflows, optimizing business processes, and enhancing operational efficiency by integrating AI-powered automation into enterprise systems. Key Responsibilities: - Design and develop AI-driven automation solutions to optimize workflows and business operations. - Implement machine learning models and AI algorithms to enhance process automation. - Develop and maintain automation scripts, bots, and AI-based workflows. - Integrate AI automation with existing systems, APIs, and data...

You will be joining Bridgenext, a global digital consultancy that focuses on helping clients innovate and achieve their digital goals by developing digital products, experiences, and solutions tailored to real people's needs. **Key Responsibilities:** - Work on internal and customer-based projects for Bridgenext - Ensure high quality code and provide optimal solutions to meet client needs - Anticipate future client needs based on market understanding **Qualifications Required:** - 3-4 years of overall experience - Strong programming skills in Python/Java - 3+ years of experience with distributed systems and frameworks, particularly Spark - Proficiency in batch data processing; experience wit...

As a Test Automation Engineering Lead at Barclays, you will be at the forefront of driving innovation and excellence in the digital landscape. Your role will involve revolutionizing digital offerings to ensure unmatched customer experiences. **Key Responsibilities:** - Design, develop, and execute testing strategies to validate software functionality, performance, and user experience. - Collaborate with cross-functional teams to identify and resolve defects, continuously improve testing processes and methodologies. - Develop and implement comprehensive test plans and strategies to ensure compliance with quality standards. - Create and execute automated test scripts using testing frameworks a...

As a Machine Learning Engineer at our company, your role will involve designing, developing, deploying, and maintaining scalable ML solutions in a cloud-native environment. You will work with data scientists, engineers, and product teams to translate business requirements into ML solutions. Key responsibilities include: - Designing and implementing machine learning models and pipelines using AWS SageMaker and related services. - Developing and maintaining robust data pipelines for training and inference workflows. - Collaborating with teams to implement MLOps best practices such as CI/CD for ML, model versioning, monitoring, and retraining strategies. - Optimizing model performance and ensur...

As a Data Engineer at our company, you will play a crucial role in designing, developing, and maintaining data pipeline architecture to ensure the efficient flow of data across the organization. Your responsibilities will include: - Designing robust and scalable data pipelines using AWS Glue, Apache Airflow, and other relevant technologies. - Integrating various data sources like SAP HANA, Kafka, and SQL databases for seamless data flow and processing. - Optimizing data pipelines for enhanced performance and reliability. You will also be responsible for: - Designing and implementing data transformation processes to clean, enrich, and structure data for analytical purposes. - Utilizing SQL an...

As an experienced professional in software development, you will be responsible for leading and contributing to the design and development of cloud-native solutions using technologies such as C#, .Net, Azure, AWS, Google Cloud Platform, Terraform, Pulumi, Ansible, Chef, Docker, Kubernetes, Databricks, Apache Airflow, and more. Your role will involve working on infrastructure as code, configuration management, containerization, orchestration, and data security while adhering to compliance standards such as HIPPA and SOC 2. Key Responsibilities: - Lead software development projects with progressively increasing responsibilities - Design and build cloud-native solutions using C#, .Net, Azure, A...

As a Data Engineering Lead at Metyis, you will play a crucial role in driving impactful change for our clients by independently leading and managing data engineering projects. Your responsibilities will include engineering complete technical solutions to address business challenges in digital marketing, eCommerce, Business Intelligence, and self-service analytics. You will collect functional and non-functional requirements, taking into account technical environments, business constraints, and enterprise organizations. Additionally, you will support clients in executing their Big Data strategies by designing and building operational data platforms, such as ETL pipelines, data anonymization pi...

As a Data Engineer at McDonald's in Hyderabad, you will play a crucial role in building and maintaining reliable Menu data products to support menu and marketing Analytics. Your responsibilities will include: - Developing and implementing new technology solutions to enhance data reliability and observability. - Leading data engineering initiatives supporting Product Mix Analytics for timely and accurate delivery of marketing and menu-related products. - Collaborating with the Product owner to define business rules that ensure the quality of Menu datasets. - Driving best practices for pipeline development, data governance, data security, and quality across marketing and menu-related datasets....

About The Role Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : PySpark Good to have skills : NA Minimum 3 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating data pipelines, ensuring data quality, and implementing...

Role Overview: You will be part of the Enterprise Data organization at Lilly, contributing to the management, monitoring, and optimization of high-quality data flow to support data sharing and analytics initiatives. Your crucial role will involve ensuring smooth data pipelines, maintaining data integrity, communicating with stakeholders, conducting root cause analysis, optimizing data pipeline performance, and implementing measures for data accuracy and reliability. Key Responsibilities: - Monitor data pipelines to ensure smooth data flow - Manage incidents to maintain data integrity - Communicate effectively with stakeholders regarding data issues - Conduct root cause analysis to enhance pr...

You are in search of a talented and driven Data Pipeline Engineer to become a part of Atomicwork's dynamic team. As a Data Pipeline Engineer, you will play a pivotal role in designing, constructing, and managing scalable data pipelines that support enterprise search capabilities. Your main responsibility will involve ensuring that data from diverse sources is effectively ingested, processed, and indexed to facilitate seamless and secure search experiences throughout the organization. **Responsibilities:** - Design, develop, and maintain data pipelines for enterprise search applications. - Implement data ingestion processes from various sources like databases, file systems, and APIs. - Develo...

As a Software Engineer, you will be responsible for designing, developing, and maintaining scalable backend services and workflow orchestration components using Python and GoLang. You will collaborate with the Airflow and Temporal team to build and optimize data pipelines and asynchronous job execution frameworks. Your role will involve implementing and managing complex workflow logic using Apache Airflow and Temporal to ensure high code quality through unit testing, integration testing, and code reviews. Additionally, you will work closely with cross-functional teams, including Data Engineering, DevOps, and Platform Engineering. Your contributions will include participating in architectural...

As a visionary backend developer, you have the opportunity to join the Data Cloud team at Zeta Global and lead the evolution with Generative AI. By leveraging cutting-edge technology, you will be responsible for developing next-generation data products that offer innovative solutions to client challenges. This role presents an exciting opportunity to immerse yourself in the marketing tech landscape, work on advanced AI projects, and pioneer their application in marketing. **Key Responsibilities:** - Analyze data from various sources to generate insights for Gen AI and non-Gen AI tasks. - Collaborate effectively with cross-functional teams to ensure alignment and understanding. - Utilize AI a...

Job Purpose and Impact The Analytics Engineer will prepare business relevant information and reduce time to insight by leading technical activities that enable information capture, business intelligence and analytics competency. In this role, you will ensure teams have the data needed to make timely, accurate and actionable insights by investing in the core capabilities of data management, data engineering and key decision support applications that consume data. You will be a key partner between business needs and data engineering and will deliver the final information product including the dashboards that enable business value. Key Accountabilities Design and implement internal process impr...

Role Overview: You will be responsible for building enterprise-level data engineering solutions on AWS to support large-scale data initiatives. Your role will involve using Python and data orchestration tools such as Apache Airflow, AWS Step Functions, or equivalent. Additionally, you will work with distributed data frameworks like Apache Spark, Kafka, and Hive, as well as various AWS services like S3, Lambda, Kinesis, Redshift, EMR, Glue, and SQS. Your expertise in data modeling and designing ETL/ELT pipelines for batch and real-time ingestion will be crucial in this role. You will also need to be familiar with CI/CD pipelines, version control, and infrastructure-as-code tools like CloudFor...

As an experienced Senior dbt Engineer with a strong background in Snowflake and Azure cloud platforms, your primary responsibility will be to lead the design and development of scalable, governed, and efficient data transformation pipelines using dbt. You will collaborate across functions to deliver business-ready data solutions. **Key Responsibilities:** - Lead the design, development, and maintenance of dbt models and transformation layers. - Define and enforce data modeling standards, best practices, and development guidelines. - Drive the end-to-end ELT process to ensure reliability and data quality across all layers. - Collaborate with data product owners, analysts, and stakeholders to ...

As a Data Engineer with a focus on Google Cloud Platform (GCP), your main role will involve hands-on experience with GCP services such as Dataflow, Pub/Sub, Cloud Storage, and Cloud Composer. You will be responsible for workflow orchestration using Apache Airflow and have experience with cloud SQL or other relational databases, preferably PostgreSQL. Your solid understanding of data modeling, ETL processes, and streaming/batch data pipelines will be essential. Additionally, familiarity with DevOps practices including CI/CD and containerization using Docker and Kubernetes will be beneficial. Key Responsibilities: - Build Data Pipelines on GCP: Utilize core GCP services for data processing. - ...

Your future role Take on a new challenge and apply your data engineering expertise in a cutting-edge field. Youll work alongside collaborative and innovative teammates. You'll play a key role in enabling data-driven decision-making across the organization by ensuring data availability, quality, and accessibility. Day-to-day, youll work closely with teams across the business (e.g., Data Scientists, Analysts, and ML Engineers), mentor junior engineers, and contribute to the architecture and design of our data platforms and solutions. Youll specifically take care of designing and developing scalable data pipelines, but also managing and optimizing object storage systems. Well look to you for: D...

Senior Software Engineer (Data Platform | Bigdata, NiFi, Spark, Airflow, Kubernetes, MinIO,CEPH) Overview The Enterprise Data Solutions team is looking for a Big Data Engineer to drive our mission to unlock potential of data assets by consistently innovating, eliminating friction in how users access data from its Big Data repositories and enforce standards and principles in the Big Data space The candidate will be part of an exciting, fast paced environment developing Data Engineering solutions in the data and analytics domain About the Role As a Senior Software Engineer on the enterprise-wide Data Platform team, you will play a critical role in building, enhancing, and scaling distributed d...