10863 Airflow Jobs - Page 31

In-depth expertise in Azure Data Factory and Databricks , including building scalable data pipelines, orchestrating complex workflows, implementing dynamic and parameterized pipelines, and optimizing Spark-based data transformations for large-scale integrations. Hands-on experience with Databricks Unity Catalog for centralized data governance, fine-grained access control, auditing, and managing data assets securely across multiple workspaces. Expert-level knowledge in RDBMS (SQL Server) with clear understanding of SQL query writing, object creation and management and performance and optimisation of DB/DWH operations. Good understanding of Transactional and Dimensional Data Modelling, Star Sc...

Role Overview: You will be responsible for designing, implementing, and optimizing Airflow Directed Acyclic Graphs (DAGs) for scalable workflows. Additionally, you will manage Google Cloud Platform (GCP) Composer environments, troubleshoot orchestration failures, and collaborate with data engineering teams to ensure seamless pipeline execution. Key Responsibilities: - Design, implement, and optimize Airflow DAGs for scalable workflows. - Manage GCP Composer environments including upgrades, performance tuning, and security. - Troubleshoot orchestration failures and optimize scheduling efficiency. - Collaborate with data engineering teams to ensure seamless pipeline execution. Qualifications R...

As a Platform Software Engineer at Macquarie, you will bring your software engineering and cloud skills to the team to foster a platform engineering mindset within Engineering. You will be involved in various development activities, delivering common platforms and solutions to enable a broader engineering community in the global asset management business. **Key Responsibilities:** - Work with a variety of modern technologies built on cloud-based infrastructure - Simplify the development experience and provide core capabilities in shared platforms - Emphasize on platform engineering, platform support, infrastructure and security integration, API development, and collaboration **Qualifications...

As a Full-Time (FTE) team member in Pune or Chennai, you will be part of a dynamic team focused on GCP technologies. Your role will involve working with a range of tools including Airflow, GCP Composer, DAGs, BigQuery, Dataflow, Pub/Sub, GCS, Terraform, CI/CD, IAM, Networking, Governance, and Security. **Key Responsibilities:** - Work closely with Platform SMEs, Architects, Business Workspaces (Product/BA & Tech), and Platform Engineers (Specialist & Sr. Specialist). - Utilize your expertise in GCP technologies to contribute to the team's success. **Qualifications Required:** - Minimum of 5 years of experience in the field. - Proficiency in Airflow, GCP Composer, DAGs, BigQuery, Dataflow, Pu...

Role Overview: You will be responsible for designing and implementing scalable data solutions using Big Data technologies and Google Cloud Platform services. Your role will involve optimizing data pipelines, collaborating with cross-functional teams, ensuring data quality, and staying updated with emerging trends in GCP data analytics. Key Responsibilities: - Design, develop, and optimize data pipelines using GCP BigQuery, Dataflow, and Apache Airflow - Utilize the Big Data Hadoop ecosystem to manage and process large datasets efficiently - Collaborate with cross-functional teams to gather requirements and deliver reliable data solutions - Ensure data quality, consistency, and integrity acro...

At Aristocrat, we take pride in crafting outstanding gaming content that brings happiness to players worldwide. Join us as a Sr Tech Lead - Manual Testing to ensure data accuracy and quality in a dynamic team. What You'll Do Lead the design and execution of test strategies for Extract Transform Load (ETL) pipelines, data transformations, and BI dashboards. Analyze large datasets to identify and resolve data quality issues. Conduct both manual and automated testing for data ingestion pipelines, data transformations, and business reports. Validate Power BI dashboards and reports for data correctness, layout integrity, and performance. Develop and implement Quality Assurance Frameworks to ensur...

Role Overview: As a Senior Production Support Lead at Deutsche Bank in Pune, India, you will take full ownership of L1 and L2 support operations across strategic enterprise platforms including API Management Platform, Kafka-as-a-Service (KaaS), and Data Platform. Your role as a senior leader will involve ensuring 24X7 operational excellence, efficient incident resolution, and strong collaboration with engineering and business stakeholders. You will be responsible for driving process optimization and operational reliability, focusing on leadership and shaping the future support model with best-in-class practices such as automation, shift-left, and service reliability engineering. Key Responsi...

About The Job The Data & AI team is a highly focused effort to lead digital-first execution and transformation at Red Hat leveraging data & AI strategically for our customers, partners, and associates. What will you do? Evolve existing source data pipelines to an ELT model of data ingestion Cleanly separate source-aligned data products from aggregate data products Richly decorate your data products with metadata to support knowledge transfer, adoption, and the application of Machine Learning Tag and classify your data assets to ensure they are used responsibly throughout the organization, applying masking or restricting access where appropriate Apply software engineering best practices to yo...

Join us as a Test Automation Engineer at Barclays, where you'll spearhead the evolution of our digital landscape, driving innovation and excellence. You'll harness cutting-edge technology to revolutionise our digital offerings, ensuring unparalleled customer experiences. As a part of team of developers, you will deliver technology stack, using strong analytical and problem solving skills to understand the business requirements and deliver quality solutions. To be successful as a Test Automation Engineer you should have experience with: ETL Tools: Proficiency with enterprise ETL platforms like Abinitio primarily, Informatica PowerCenter, IBM DataStage, Talend, SSIS, or Matillion Scripting/Pro...

Job Summary At Aristocrat, we prioritize offering top-tier gaming content and groundbreaking technology solutions. Our goal is to bring happiness to life through the power of play, and we are looking for an outstanding Team Lead in Data Engineering to join our dynamic team. This position provides an exciting chance to lead and encourage a team of dedicated engineers, guide architectural choices, and implement innovative data solutions that will greatly influence our business. Data Engineering Team Lead builds data infrastructure, operates with cloud-based stacks, collaborates multi-functionally. What You'll Do Lead and manage a team of data engineers, driving delivery across multiple project...

At Aristocrat, we aim to bring happiness through play. Seeking a dedicated Technical Lead - Database QA to join our Global Data team . Critical role in maintaining data system performance. As a Technical Lead - Database QA, you will have a meaningful role in our mission by spearheading the quality assurance efforts for our data management processes. You will collaborate closely with a team of data stewards, architects, and analysts to ensure the integrity and accuracy of our data. This is an exciting and groundbreaking opportunity to be at the forefront of data quality assurance, using brand new technologies to deliver flawless solutions. What You'll Do:- Conduct comprehensive analysis and p...

At Aristocrat, we are driven by our mission to bring happiness through play and innovation. As a global leader in gaming content and technology, we pride ourselves on delivering outstanding gaming experiences across our diverse business units. We are seeking an ambitious and skilled Data Engineer to join our dynamic team. This role is crucial in crafting the future of our data infrastructure and ensuring that our data solutions are outstanding and reliable. If you are passionate about data engineering and looking to make a significant impact in a world-class gaming company, this opportunity is for you! What You'll Do Design, develop, and maintain batch and streaming data pipelines. Implement...

Role Overview: You will be responsible for designing, building, and maintaining data solutions including data infrastructure, pipelines, etc. for collecting, storing, processing, and analyzing large volumes of data efficiently and accurately. Collaborating with business and technology stakeholders to understand current and future data requirements will be a key aspect of your role. Key Responsibilities: - Collaborate with business and technology stakeholders to understand current and future data requirements - Design, build, and maintain reliable, efficient, and scalable data infrastructure for data collection, storage, transformation, and analysis - Plan, design, build, and maintain scalabl...

As a Data Engineer, you will be responsible for designing, building, and maintaining ETL/ELT data pipelines to facilitate finance data migration from SAP R/3 to S/4HANA. Your role will involve working with PySpark and Hadoop-based environments for managing large-scale data processing. You will utilize Apache Airflow to orchestrate and monitor workflows, ensuring the reliability of pipelines. Collaboration with data analysts is crucial to understand transformation logic and implement data mapping specifications. Additionally, you will coordinate with the BI reporting team to guarantee downstream data readiness and support reporting needs. Integrating data from SAP and non-SAP sources into AWS...

As an experienced AI professional with 5+ years of hands-on experience in building and deploying AI solutions, ideally in fintech or financial services, you will be responsible for the following key responsibilities: - Strong coding skills in Python and familiarity with libraries like TensorFlow, PyTorch, scikit-learn, XGBoost, etc. - Experience with NLP, time series forecasting, anomaly detection, or graph ML relevant to fintech applications. - Solid understanding of MLOps concepts including model versioning, deployment, and monitoring. - Experience working with cloud platforms such as AWS/GCP/Azure and ML services like SageMaker, Vertex AI. - Familiarity with Docker, Kubernetes, or other c...

Experience : 4.00 + years Salary : INR 1500000-3000000 / year (based on experience) Expected Notice Period : 15 Days Shift : (GMT+05:30) Asia/Kolkata (IST) Opportunity Type : Remote Placement Type : Full Time Permanent position(Payroll and Compliance to be managed by: NuStudio.AI) (*Note: This is a requirement for one of Uplers' client - AI-first, API-powered Data Platform) What do you need for this opportunity? Must have skills required: Databricks, dbt, Delta Lake, Spark, Unity catalog, AI, Airflow, Cloud Function, Cloud Storage, Databricks Workflows, Dataflow, ETL/ELT, Functions), GCP (BigQuery, Pub/Sub, PySpark, AWS, Hadoop AI-first, API-powered Data Platform is Looking for: We’re scalin...

Experience : 5.00 + years Salary : INR 3000000.00 / year (based on experience) Expected Notice Period : 15 Days Shift : (GMT+05:30) Asia/Kolkata (IST) Opportunity Type : Remote Placement Type : Full Time Permanent position(Payroll and Compliance to be managed by: Zero to 1) (*Note: This is a requirement for one of Uplers' client - Zero to 1) What do you need for this opportunity? Must have skills required: Advanced python, SQL Mastery, 5+ years in Data Engineering with AWS and Python, Linux Proficiency:, Strong AWS Experience Zero to 1 is Looking for: Responsibilities: ✅ Design, develop, and maintain scalable data pipelines using a variety of AWS services ✅ Work with relational databases suc...

Experience : 4.00 + years Salary : Confidential (based on experience) Shift : (GMT+05:30) Asia/Kolkata (IST) Opportunity Type : Remote Placement Type : Full time Permanent Position (*Note: This is a requirement for one of Uplers' client - Forbes Advisor) What do you need for this opportunity? Must have skills required: Airflow, dbt, Google Cloud Forbes Advisor is Looking for: Designation: Senior Data Engineer Forbes Advisor is a high-growth digital media and technology company dedicated to helping consumers make confident, informed decisions about their money, health, careers, and everyday life. We do this by combining data-driven content, rigorous product comparisons, and user-first design ...

Experience : 8.00 + years Salary : INR 8000000-9000000 / year (based on experience) Expected Notice Period : 15 Days Shift : (GMT+05:30) Asia/Kolkata (IST) Opportunity Type : Remote Placement Type : Full Time Permanent position(Payroll and Compliance to be managed by: PvX Partners) (*Note: This is a requirement for one of Uplers' client - PvX Partners) What do you need for this opportunity? Must have skills required: Effective Communication, entrepreneurial mindset, Problem-Solving Excellence, Forecast Revenue, Data Science, Machine Learning PvX Partners is Looking for: The Role We are seeking an exceptional Lead Data Scientist to spearhead our data science and AI initiatives in the global g...

Experience : 5.00 + years Salary : Confidential (based on experience) Expected Notice Period : 15 Days Shift : (GMT+05:30) Asia/Kolkata (IST) Opportunity Type : Remote Placement Type : Full Time Contract for 3 Months(40 hrs a week/160 hrs a month) (*Note: This is a requirement for one of Uplers' client - A Leading Loan Software Company in USA) What do you need for this opportunity? Must have skills required: Experience in mortgage, Experience with AWS CICD pipelines., Exposure to Power BI or similar BI tools connected to Snowflake., Familiarity with orchestration tools (Airflow, Loan Origination, MWAA)., or financial services data is a plus., schema versioning/migration tools, Servicing, Sno...

Experience : 5.00 + years Salary : INR 4500000-5500000 / year (based on experience) Expected Notice Period : 30 Days Shift : (GMT+05:30) Asia/Kolkata (IST) Opportunity Type : Remote Placement Type : Full Time Permanent position(Payroll and Compliance to be managed by: Medblocks) (*Note: This is a requirement for one of Uplers' client - Medblocks) What do you need for this opportunity? Must have skills required: Kafka, Snowflake, CI/CD Pipeline, ETL/ELT pipelines, PostgreSQL Medblocks is Looking for: As a Senior Data Platform Engineer at Medblocks, you will be the primary architect of our data infrastructure, designing and implementing the foundation that powers our entire healthcare platform...

Experience : 3.00 + years Salary : INR 2500000.00 / year (based on experience) Expected Notice Period : 30 Days Shift : (GMT+05:30) Asia/Kolkata (IST) Opportunity Type : Remote Placement Type : Full Time Permanent position(Payroll and Compliance to be managed by: NA) (*Note: This is a requirement for one of Uplers' client - Nomupay) What do you need for this opportunity? Must have skills required: Apache Hudi, Flink, Iceberg, AWS, Python, PySpark, Batch Processing, Apache Spark Nomupay is Looking for: 📈 Opportunity in a company with a solid track record of performance 🤝 Opportunity to work with diverse, global teams 🚀 Rapid career advancement with opportunities to learn 💰 Competitive salary ...

Experience : 4.00 + years Salary : INR 1500000-3000000 / year (based on experience) Expected Notice Period : 15 Days Shift : (GMT+05:30) Asia/Kolkata (IST) Opportunity Type : Remote Placement Type : Full Time Permanent position(Payroll and Compliance to be managed by: NuStudio.AI) (*Note: This is a requirement for one of Uplers' client - AI-first, API-powered Data Platform) What do you need for this opportunity? Must have skills required: Databricks, dbt, Delta Lake, Spark, Unity catalog, AI, Airflow, Cloud Function, Cloud Storage, Databricks Workflows, Dataflow, ETL/ELT, Functions), GCP (BigQuery, Pub/Sub, PySpark, AWS, Hadoop AI-first, API-powered Data Platform is Looking for: We’re scalin...

Experience : 5.00 + years Salary : Confidential (based on experience) Expected Notice Period : 15 Days Shift : (GMT+05:30) Asia/Kolkata (IST) Opportunity Type : Remote Placement Type : Full Time Contract for 3 Months(40 hrs a week/160 hrs a month) (*Note: This is a requirement for one of Uplers' client - A Leading Loan Software Company in USA) What do you need for this opportunity? Must have skills required: Experience in mortgage, Experience with AWS CICD pipelines., Exposure to Power BI or similar BI tools connected to Snowflake., Familiarity with orchestration tools (Airflow, Loan Origination, MWAA)., or financial services data is a plus., schema versioning/migration tools, Servicing, Sno...

Experience : 5.00 + years Salary : INR 4500000-5500000 / year (based on experience) Expected Notice Period : 30 Days Shift : (GMT+05:30) Asia/Kolkata (IST) Opportunity Type : Remote Placement Type : Full Time Permanent position(Payroll and Compliance to be managed by: Medblocks) (*Note: This is a requirement for one of Uplers' client - Medblocks) What do you need for this opportunity? Must have skills required: Kafka, Snowflake, CI/CD Pipeline, ETL/ELT pipelines, PostgreSQL Medblocks is Looking for: As a Senior Data Platform Engineer at Medblocks, you will be the primary architect of our data infrastructure, designing and implementing the foundation that powers our entire healthcare platform...