683 Apache Airflow Jobs - Page 23

As a Data Specialist, you will be responsible for utilizing your expertise in ETL Fundamentals, SQL, BigQuery, Dataproc, Python, Data Catalog, Data Warehousing, and various other tools to contribute to the successful implementation of data projects. Your role will involve working with technologies such as Cloud Trace, Cloud Logging, Cloud Storage, and Datafusion to build and maintain a modern data platform. To excel in this position, you should possess a minimum of 5 years of experience in the data engineering field, with a focus on GCP cloud data implementation suite including BigQuery, Pub Sub, Data Flow/Apache Beam, Airflow/Composer, and Cloud Storage. Your strong understanding of very la...

You are an experienced Data Engineer with expertise in PySpark, Snowflake, and AWS, and you will be responsible for designing, developing, and optimizing data pipelines and workflows in a cloud-based environment. Your main focus will be leveraging AWS services, PySpark, and Snowflake for data processing and analytics. Your key responsibilities will include designing and implementing scalable ETL pipelines using PySpark on AWS, developing and optimizing data workflows for Snowflake integration, managing and configuring AWS services such as S3, Lambda, Glue, EMR, and Redshift, collaborating with data analysts and business teams to understand requirements and deliver solutions, ensuring data se...

You will be responsible for fetching and transforming data from various systems, conducting in-depth analyses to identify gaps, opportunities, and insights, and providing recommendations that support strategic business decisions. Your key responsibilities will include data extraction and transformation, data analysis and insight generation, visualization and reporting, collaboration with cross-functional teams, and building strong working relationships with external stakeholders. You will report to the VP Business Growth and work closely with clients. To excel in this role, you should have proficiency in SQL for data querying and Python for data manipulation and transformation. Experience wi...

The ideal candidate ready to join immediately can share their details via email for quick processing at nitin.patil@ust.com. Act swiftly for immediate attention! With over 5 years of experience, the successful candidate will have the following roles and responsibilities: - Designing, developing, and maintaining scalable data pipelines using Spark (PySpark or Spark with Scala). - Constructing data ingestion and transformation frameworks for both structured and unstructured data sources. - Collaborating with data analysts, data scientists, and business stakeholders to comprehend requirements and deliver reliable data solutions. - Handling large volumes of data while ensuring quality, integrity...

As a Data Engineer II in our team, you will be responsible for managing the deprecation of migrated workflows and ensuring the seamless migration of workflows into new systems. Your expertise in building and maintaining scalable data pipelines, both on-premises and on the cloud, will be crucial. You should have a deep understanding of input and output data sources, upstream downstream dependencies, and data quality assurance. Proficiency in tools like Git, Apache Airflow, Apache Spark, SQL, data migration, and data validation is essential for this role. Your key responsibilities will include: Workflow Deprecation: - Evaluate current workflows" dependencies and consumption for deprecation. - ...

You will be responsible for designing and building scalable and efficient data warehouses to support analytics and reporting needs. Additionally, you will develop and optimize ETL pipelines by writing complex SQL queries and automating data pipelines using tools like Apache Airflow. Your role will also involve query optimization, performance tuning, and database management with MySQL, PostgreSQL, and Spark for structured and semi-structured data. Ensuring data quality and governance will be a key part of your responsibilities, where you will validate, monitor, and enforce practices to maintain data accuracy, consistency, and completeness. You will implement data governance best practices, de...

The Digital Success Engineering team is looking for an experienced Marketing Cloud Engineer (SMTS) to join the Customer Engagement Engineering team. In this role, you will be responsible for providing product support to customers and business users by leveraging your strong Salesforce Marketing Cloud platform knowledge, technical expertise, and exceptional business-facing skills. You will work closely with various teams to understand, troubleshoot, and coordinate operational issues within the Customer Engagement Ecosystem. Your responsibilities will include conducting technical requirements gathering, designing and implementing robust solutions for Salesforce Marketing Cloud projects, and en...

You will be responsible for designing and building data warehouses to support analytics and reporting needs. This includes architecting scalable and efficient data warehouses. Additionally, you will be developing and optimizing ETL pipelines by writing complex SQL queries and utilizing tools like Apache Airflow for automation. Query optimization and performance tuning will be a key aspect of your role, where you will focus on writing efficient SQL queries for both ETL jobs and dashboards in BI tools. Database management is another crucial responsibility, involving working with MySQL, PostgreSQL, and Spark to manage structured and semi-structured data. You will also ensure data quality and go...

Founded in 1976, CGI is among the largest independent IT and business consulting services firms in the world. With 94,000 consultants and professionals across the globe, CGI delivers an end-to-end portfolio of capabilities, from strategic IT and business consulting to systems integration, managed IT and business process services and intellectual property solutions. CGI works with clients through a local relationship model complemented by a global delivery network that helps clients digitally transform their organizations and accelerate results. CGI Fiscal 2024 reported revenue is CA$14.68 billion and CGI shares are listed on the TSX (GIB.A) and the NYSE (GIB). Learn more at cgi.com. Position...

Whether you're at the start of your career or looking to discover your next adventure, your story begins here. At Citi, you'll have the opportunity to expand your skills and make a difference at one of the world's most global banks. We're fully committed to supporting your growth and development from the start with extensive on-the-job training and exposure to senior leaders, as well as more traditional learning. You'll also have the chance to give back and make a positive impact where we live and work through volunteerism. The Product Developer is a strategic professional who stays abreast of developments within their field and contributes to directional strategy by considering their applic...

As a Developer contracted by Luxoft for supporting customer initiatives, your main task will involve developing solutions based on client requirements within the Telecom/network work environment. You will be responsible for utilizing technologies such as Databricks and Azure, Apache Spark, Python, SQL, and Apache Airflow to create and manage Databricks clusters for ETL processes. Integration with ADLS, Blob Storage, and efficient data ingestion from various sources including on-premises databases, cloud storage, APIs, and streaming data will also be part of your role. Moreover, you will work on handling secrets using Azure Key Vault, interacting with APIs, and gaining hands-on experience wit...

You are a highly skilled and experienced Senior Backend Developer with a focus on Python and backend development. In this role, you will be responsible for designing, developing, and maintaining backend applications using Python. Collaborating with cross-functional teams, you will implement RESTful APIs and web services to ensure high-performance and scalable backend systems. Your key responsibilities will include optimizing database performance, working with relational databases such as MySQL and PostgreSQL, and GraphDB like Neo4j. You will also develop and manage orchestration workflows using tools like Apache Airflow, as well as implementing and maintaining CI/CD pipelines for smooth depl...

Zeta Global is looking for a visionary backend developer to join the Data Cloud team and lead the evolution with Generative AI. By leveraging this cutting-edge technology, you will be responsible for developing next-generation data products that provide innovative solutions to client challenges. This role presents an exciting opportunity to immerse yourself in the marketing tech landscape, work on advanced AI projects, and pioneer their application in marketing. In this role, you will be expected to fulfill two main responsibilities. Firstly, as a Backend Developer, you will conduct data analysis and generate outputs for Gen AI tasks while also supporting standard data analysis functions. Se...

At Lilly, the focus is on uniting caring with discovery to enhance the lives of people worldwide. As a global healthcare leader headquartered in Indianapolis, Indiana, we are committed to developing and providing life-changing medicines, advancing disease management, and contributing to communities through philanthropy and volunteerism. Our dedicated team of 35,000 employees collaborates to prioritize people and strive towards making a positive impact globally. The Enterprise Data organization at Lilly has pioneered an integrated data and analytics platform designed to facilitate the efficient processing and analysis of data sets across various environments. As part of this team, you will pl...

As a Data Analyst with 1+ years of experience in AdTech, you will be an integral part of our analytics team. Your primary role will involve analyzing large-scale advertising and digital media datasets to support business decisions. You will work with various AdTech data such as ads.txt, programmatic delivery, campaign performance, and revenue metrics. Your responsibilities will include designing, developing, and maintaining scalable data pipelines using GCP-native tools like Cloud Functions, Dataflow, and Composer. You will be required to write and optimize complex SQL queries in BigQuery for data extraction and transformation. Additionally, you will build and maintain dashboards and reports...

As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines using Spark, specifically PySpark or Spark with Scala. Your role will involve building data ingestion and transformation frameworks for various structured and unstructured data sources. Collaboration with data analysts, data scientists, and business stakeholders is essential to understand requirements and deliver reliable data solutions. Working with large volumes of data, you will ensure quality, integrity, and consistency while optimizing data workflows for performance, scalability, and cost efficiency on cloud platforms such as AWS, Azure, or GCP. Implementation of data quality c...

Imagine what you could do here. At Apple, new ideas have a way of becoming phenomenal products, services, and customer experiences very quickly. Every single day, people do amazing things at Apple. Do you want to impact the future of Manufacturing here at Apple through cutting edge ML techniques This position involves a wide variety of skills, innovation, and is a rare opportunity to be working on ground breaking, new applications of machine-learning, research and implementation. Ultimately, your work would have a huge impact on billions of users across the globe. You can help inspire change, by using your skills to influence globally recognized products" supply chain. The goal of Apple's Ma...

Candidates who are ready to join immediately can share their details via email for quick processing to nitin.patil@ust.com. Act fast for immediate attention! With over 5 years of experience, the ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines using Spark, either PySpark or Spark with Scala. They will also be tasked with building data ingestion and transformation frameworks for structured and unstructured data sources. Collaboration with data analysts, data scientists, and business stakeholders to understand requirements and deliver reliable data solutions is a key aspect of the role. The candidate will work with large volumes of data to ...

As a Data Engineer, you will be responsible for designing and building efficient data pipelines using Azure Databricks (PySpark). You will implement business logic for data transformation and enrichment at scale, as well as manage and optimize Delta Lake storage solutions. Additionally, you will develop REST APIs using FastAPI to expose processed data and deploy them on Azure Functions for scalable and serverless data access. You will play a key role in data orchestration by developing and managing Airflow DAGs to orchestrate ETL processes. This includes ingesting and processing data from various internal and external sources on a scheduled basis. Database management will also be part of you...

The ideal candidate ready to join immediately can share their details via email for quick processing at nitin.patil@ust.com. Act fast for immediate attention! With over 5 years of experience, you will be responsible for designing, developing, and maintaining scalable data pipelines using Spark (PySpark or Spark with Scala). You will also build data ingestion and transformation frameworks for structured and unstructured data sources. Collaboration with data analysts, data scientists, and business stakeholders to understand requirements and deliver reliable data solutions will be a key aspect of the role. Working with large volumes of data, ensuring quality, integrity, and consistency, and opt...

The ideal candidate for this position should have at least 5 years of experience and must be ready to join immediately. In this role, you will be responsible for designing, developing, and maintaining scalable data pipelines using Spark, specifically PySpark or Spark with Scala. You will also be tasked with building data ingestion and transformation frameworks for structured and unstructured data sources. Collaboration with data analysts, data scientists, and business stakeholders to understand requirements and deliver reliable data solutions is a key aspect of this role. Working with large volumes of data to ensure quality, integrity, and consistency is crucial. Additionally, optimizing dat...

As an AI and Machine Learning Engineer at Dailoqa, you will play a crucial role in shaping the future of Financial Services clients and the company as a whole. Working directly with the founding team, you will have the opportunity to apply the latest AI techniques to address real-world problems encountered by Financial Services clients. Your responsibilities will include designing, constructing, and enhancing datasets to assess and continually enhance our solutions, as well as engaging in strategy and product ideation sessions to influence our product and solution roadmap. Key Responsibilities: - Agentic AI Development: Build scalable multi-modal Large Language Model (LLM) based AI agents us...

The role of Staff Engineer - Data in SonyLIV's Digital Business is to lead the data engineering strategy, architect scalable data infrastructure, drive innovation in data processing, ensure operational excellence, and build a high-performance team to enable data-driven insights for OTT content and user engagement. This position is based in Mumbai and requires a minimum of 8 years of experience in the field. Responsibilities include defining the technical vision for scalable data infrastructure using modern technologies like Spark, Kafka, Snowflake, and cloud services, leading innovation in data processing and architecture through real-time data processing and streaming analytics, ensuring op...

As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines using Spark (PySpark or Spark with Scala). Your role will involve building data ingestion and transformation frameworks for structured and unstructured data sources. Collaborating with data analysts, data scientists, and business stakeholders to understand requirements and deliver reliable data solutions will be a key aspect of your responsibilities. Additionally, you will work with large volumes of data to ensure quality, integrity, and consistency, optimizing data workflows for performance, scalability, and cost efficiency on cloud platforms such as AWS, Azure, or GCP. Implementin...

Key Skills: Data engineering, Apache Airflow, GCP, BigQuery, GCS, SQL, ETL/ELT, Docker, Kubernetes, data governance, Agile, CI/CD, DevOps, pipeline orchestration, technical leadership. Roles & Responsibilities: Evaluate and provide scalable technical solutions to address complex and interdependent data processes. Ensure data quality and accuracy by implementing data quality checks, data contracts, and governance processes. Collaborate with software development teams and business analysts to understand data requirements and deliver fit-for-purpose data solutions. Lead the team in delivering end-to-end data engineering solutions. Design, develop, and maintain complex applications to support da...