682 Apache Airflow Jobs - Page 6

We are seeking a skilled Databricks Developer to join our team on a 4-month contract basis. The ideal candidate will have strong expertise in modern data engineering technologies and the ability to work in a fast-paced, remote environment. Key Responsibilities : - Develop, optimize, and manage large-scale data pipelines using Databricks, PySpark, DBT, and AWS S3/Glue/Redshift. - Work in multi-cloud environments including AWS, Azure, and GCP. - Implement workflow orchestration using Airflow or similar frameworks. - Design, implement, and manage data warehouse solutions, schema evolution, and data versioning. - Collaborate with cross-functional teams to deliver high-quality data solutions. Req...

We are looking for an experienced Databricks Tech Lead to join our team on a 4-month extendable contract. The ideal candidate will bring deep expertise in data engineering, big data platforms, and cloud-based data warehouse solutions, with the ability to work in a fast-paced remote environment. Key Responsibilities : - Lead the design, optimization, and management of large-scale data pipelines using Databricks, Spark (PySpark), and AWS data services. - Productionize and deploy Big Data platforms and applications across multi-cloud environments (AWS, Azure, GCP). - Build and manage data warehouse solutions, schema evolution, and data versioning. - Implement and manage workflow orchestration u...

We are looking for an experienced Databricks Tech Lead to join our team on a 4-month extendable contract. The ideal candidate will bring deep expertise in data engineering, big data platforms, and cloud-based data warehouse solutions, with the ability to work in a fast-paced remote environment. Key Responsibilities : - Lead the design, optimization, and management of large-scale data pipelines using Databricks, Spark (PySpark), and AWS data services. - Productionize and deploy Big Data platforms and applications across multi-cloud environments (AWS, Azure, GCP). - Build and manage data warehouse solutions, schema evolution, and data versioning. - Implement and manage workflow orchestration u...

We are seeking a skilled Databricks Developer to join our team on a 4-month contract basis. The ideal candidate will have strong expertise in modern data engineering technologies and the ability to work in a fast-paced, remote environment. Key Responsibilities : - Develop, optimize, and manage large-scale data pipelines using Databricks, PySpark, DBT, and AWS S3/Glue/Redshift. - Work in multi-cloud environments including AWS, Azure, and GCP. - Implement workflow orchestration using Airflow or similar frameworks. - Design, implement, and manage data warehouse solutions, schema evolution, and data versioning. - Collaborate with cross-functional teams to deliver high-quality data solutions. Req...

We are looking for an experienced Databricks Tech Lead to join our team on a 4-month extendable contract. The ideal candidate will bring deep expertise in data engineering, big data platforms, and cloud-based data warehouse solutions, with the ability to work in a fast-paced remote environment. Key Responsibilities : - Lead the design, optimization, and management of large-scale data pipelines using Databricks, Spark (PySpark), and AWS data services. - Productionize and deploy Big Data platforms and applications across multi-cloud environments (AWS, Azure, GCP). - Build and manage data warehouse solutions, schema evolution, and data versioning. - Implement and manage workflow orchestration u...

We are seeking a skilled Databricks Developer to join our team on a 4-month contract basis. The ideal candidate will have strong expertise in modern data engineering technologies and the ability to work in a fast-paced, remote environment. Key Responsibilities : - Develop, optimize, and manage large-scale data pipelines using Databricks, PySpark, DBT, and AWS S3/Glue/Redshift. - Work in multi-cloud environments including AWS, Azure, and GCP. - Implement workflow orchestration using Airflow or similar frameworks. - Design, implement, and manage data warehouse solutions, schema evolution, and data versioning. - Collaborate with cross-functional teams to deliver high-quality data solutions. Req...

As a skilled Data Engineer at BigThinkCode Technologies, your role involves designing, building, and maintaining robust data pipelines and infrastructure to optimize data flow and ensure scalability. Your technical expertise in Python, SQL, ETL/ELT frameworks, and cloud data warehouses, along with strong collaboration skills, will be crucial in partnering with cross-functional teams to enable seamless access to structured and unstructured data across the organization. **Key Responsibilities:** - Design, develop, and maintain scalable ETL/ELT pipelines for structured and unstructured data processing. - Optimize and manage SQL queries for performance and efficiency in handling large-scale data...

As a Senior Analytical Data Engineer at Springer Nature in Pune, India, you will be responsible for designing and building scalable, efficient data solutions. Your key responsibilities will include: - Designing, implementing, testing, and maintaining ETL/ELT pipelines for ingesting and transforming large datasets from diverse sources. - Architecting, building, testing, and optimizing data warehouses and data lakes with a focus on data consistency, performance, and analytics best practices. - Collaborating with cross-functional teams to understand data needs, translate requirements into data models, and develop reliable data sources. - Implementing data quality checks, validation rules, and m...

As a Staff Software Engineer in the Data Quality & Audience team at Warner Bros. Discovery, you will play a crucial role in maintaining data quality and facilitating data-driven decisions. Your technical expertise will be vital in developing systems that adhere to quality standards and in creating datasets using top-notch software engineering and data management practices. Your responsibilities will include: - Applying comprehensive knowledge of technology options within the Data Engineering ecosystem - Constructing systems and datasets while adhering to best practices and operational excellence - Grasping data warehousing concepts, ETL processes, and tools such as Apache Airflow and Talend ...

As a Data Engineer at 1buy.ai, an innovative procurement and pricing intelligence platform in the electronics industry, you will play a crucial role in the dynamic tech team. Your responsibilities will include managing, transforming, and delivering data-driven insights that are essential for empowering the platform and decision-making processes. Key Responsibilities: - Develop and optimize complex SQL queries for data extraction and analytics. - Build and maintain robust ETL/ELT pipelines for data ingestion and transformation. - Collaborate closely with stakeholders to support data visualization initiatives. - Ensure data accuracy, integrity, and availability across various databases (MongoD...

You will be responsible for designing advanced, industry-relevant Python training modules customized for data engineering and cloud environments. Your role will involve delivering engaging sessions covering: - Core and Advanced Python Programming - Functional & Object-Oriented Programming - Data Manipulation using pandas, numpy, etc. - Scripting for automation and data pipelines - Python for API handling, file processing, and real-time data ingestion Additionally, you will need to integrate Python with tools like SQL, PySpark, Apache Airflow, and cloud services (Azure/AWS/GCP). Your tasks will include creating real-time projects, case studies, and hands-on labs simulating data engineering sc...

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-...

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-...

As a DevOps Engineer specializing in data, you will be responsible for implementing and managing our cloud-based data infrastructure using AWS and Snowflake. You will collaborate with data engineers, data scientists, and other stakeholders to design, deploy, and maintain a robust data ecosystem that supports our analytics and business intelligence initiatives. Your expertise in modern data tech stacks, MLOps methodologies, automation, and information security will be crucial in enhancing our data pipelines and ensuring data integrity and availability. - **Infrastructure Management:** Design, deploy, and manage AWS cloud infrastructure for data storage, processing, and analytics, ensuring hig...

As a Data Scientist/Engineer at Arcadis, you will play a crucial role in extracting data from various sources such as PDFs, images, databases, and APIs. You will be responsible for driving optical character recognition (OCR) processes, applying advanced natural language processing (NLP) techniques, and developing highly accurate statistical models and data engineering pipelines. Additionally, you will design and manage scalable cloud-based data architectures using Azure, AWS, or GCP services, collaborating closely with business domain experts to drive key business value drivers. - Lead data extraction from multiple sources, including PDFs, images, databases, and APIs. - Drive optical charact...

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-...

Role Overview: As a Lead Software Test Engineer (Automation Tester) at Mastercard, you will be joining the PVS Identity Solutions team to play a pivotal role in ensuring the quality, performance, and automation of various data-driven pipelines. Your responsibilities will include managing testing infrastructure for Functional test, Automation, and coordinating testing activities across multiple programs and projects. The ideal candidate will have experience in working with large-scale data and automation testing of Java, Cloud Native application/services. Your role will involve leading the development and maintenance of automated testing frameworks, providing technical leadership for new majo...

As a Data Engineer at UST, your role involves designing, developing, and optimizing ETL pipelines using PySpark on Google Cloud Platform (GCP). You will work with various GCP services such as BigQuery, Cloud Dataflow, Cloud Composer (Apache Airflow), and Cloud Storage for data transformation and orchestration. Your responsibilities will include: - Developing and optimizing Spark-based ETL processes for large-scale data processing. - Implementing best practices for data governance, security, and monitoring in a cloud environment. - Collaborating with data engineers, analysts, and business stakeholders to understand data requirements. - Troubleshooting performance bottlenecks and optimizing Sp...

As a Head of Analytics or Data Architect with over 15 years of experience, you will play a crucial role in designing and implementing an organizational data strategy aligned with business processes. Your responsibilities will include developing data models, managing data warehouses, implementing data governance standards, and collaborating with business leaders to define actionable metrics and data visualization dashboards. - Interact closely with Business Leaders and Product Team to design, develop, and implement an overall organizational data strategy - Identify data sources and build complex data ingestion, data integration, and ETL strategy - Create and maintain an end-to-end vision for ...

Role Overview: As a Back-End Developer at SuperZoom, CBRE's data quality platform, your main responsibility will be to build new capabilities by focusing on developing a backend pipeline. This pipeline will be utilized to process client asset files, run data profiling and quality checks, and integrate LLM-based assessments via API calls such as ChatGPT. Your expertise in data processing, API integrations, and scalable backend architecture will play a crucial role in achieving success in this position. Key Responsibilities: - Develop a backend pipeline to ingest, profile, and assess client asset data files. - Implement data profiling and quality checks using appropriate libraries and framewor...

As a Business Intelligence Analyst (R Shiny Developer) at MCGI Private Limited in Kochi, India, your primary responsibility will be to collaborate with the U.S.-based business analytics team. You will focus on maintaining and enhancing the in-house Shiny dashboard, developing new functionalities and automation, and supporting business intelligence and data engineering needs. Key Responsibilities: - Maintain, optimize, and enhance the Shiny dashboard to ensure reliability and performance. - Develop SQL data pipelines to support merchants and the business analytics team. - Create new features and automation to streamline merchandising processes. - Collaborate with merchants, sales, and stakeho...

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-...

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-...

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-...

As an experienced Data Engineer with expertise in Python, AWS, DBT, Airflow, Fivetran, and data streaming technologies, you will play a crucial role in building scalable, reliable, and secure data solutions. Key Responsibilities: - Develop and deploy data pipelines, both batch and streaming, into production with at least 5 years of experience. - Focus on ingesting, cleansing, and delivering high-quality data products using Google Cloud Platform (GCP). - Work with various relational SQL and NoSQL databases. - Utilize cloud-native data services, preferably Google Cloud Platform, including BigQuery, Vertex AI, Pub/Sub, Cloud Functions, etc. - Have deep expertise in one of the popular data wareh...