974 Apache Airflow Jobs

Role & Responsibilities : - Design, build, and maintain scalable batch and streaming ETL pipelines on Databricks using Delta Lake and Delta Live Tables (DLT). - Develop and optimize Spark/PySpark jobs for performance, cost-efficiency, and reliability; tune cluster sizing and autoscaling policies. - Implement data quality, observability, lineage and monitoring (alerts, dashboards, job health) to ensure production SLAs. - Collaborate with Data Engineers, Data Scientists and Architects to define schemas, partitioning strategies and data models that support analytics and ML use cases. - Build CI/CD and release automation for Databricks assets (notebooks, jobs, Delta tables); manage Git-based sou...

Role & Responsibilities : - Design, build, and maintain scalable batch and streaming ETL pipelines on Databricks using Delta Lake and Delta Live Tables (DLT). - Develop and optimize Spark/PySpark jobs for performance, cost-efficiency, and reliability; tune cluster sizing and autoscaling policies. - Implement data quality, observability, lineage and monitoring (alerts, dashboards, job health) to ensure production SLAs. - Collaborate with Data Engineers, Data Scientists and Architects to define schemas, partitioning strategies and data models that support analytics and ML use cases. - Build CI/CD and release automation for Databricks assets (notebooks, jobs, Delta tables); manage Git-based sou...

Role & Responsibilities : - Design, build, and maintain scalable batch and streaming ETL pipelines on Databricks using Delta Lake and Delta Live Tables (DLT). - Develop and optimize Spark/PySpark jobs for performance, cost-efficiency, and reliability; tune cluster sizing and autoscaling policies. - Implement data quality, observability, lineage and monitoring (alerts, dashboards, job health) to ensure production SLAs. - Collaborate with Data Engineers, Data Scientists and Architects to define schemas, partitioning strategies and data models that support analytics and ML use cases. - Build CI/CD and release automation for Databricks assets (notebooks, jobs, Delta tables); manage Git-based sou...

Role & Responsibilities : - Design, build, and maintain scalable batch and streaming ETL pipelines on Databricks using Delta Lake and Delta Live Tables (DLT). - Develop and optimize Spark/PySpark jobs for performance, cost-efficiency, and reliability; tune cluster sizing and autoscaling policies. - Implement data quality, observability, lineage and monitoring (alerts, dashboards, job health) to ensure production SLAs. - Collaborate with Data Engineers, Data Scientists and Architects to define schemas, partitioning strategies and data models that support analytics and ML use cases. - Build CI/CD and release automation for Databricks assets (notebooks, jobs, Delta tables); manage Git-based sou...

Role & Responsibilities : - Design, build, and maintain scalable batch and streaming ETL pipelines on Databricks using Delta Lake and Delta Live Tables (DLT). - Develop and optimize Spark/PySpark jobs for performance, cost-efficiency, and reliability; tune cluster sizing and autoscaling policies. - Implement data quality, observability, lineage and monitoring (alerts, dashboards, job health) to ensure production SLAs. - Collaborate with Data Engineers, Data Scientists and Architects to define schemas, partitioning strategies and data models that support analytics and ML use cases. - Build CI/CD and release automation for Databricks assets (notebooks, jobs, Delta tables); manage Git-based sou...

Role & Responsibilities : - Design, build, and maintain scalable batch and streaming ETL pipelines on Databricks using Delta Lake and Delta Live Tables (DLT). - Develop and optimize Spark/PySpark jobs for performance, cost-efficiency, and reliability; tune cluster sizing and autoscaling policies. - Implement data quality, observability, lineage and monitoring (alerts, dashboards, job health) to ensure production SLAs. - Collaborate with Data Engineers, Data Scientists and Architects to define schemas, partitioning strategies and data models that support analytics and ML use cases. - Build CI/CD and release automation for Databricks assets (notebooks, jobs, Delta tables); manage Git-based sou...

Role & Responsibilities : - Design, build, and maintain scalable batch and streaming ETL pipelines on Databricks using Delta Lake and Delta Live Tables (DLT). - Develop and optimize Spark/PySpark jobs for performance, cost-efficiency, and reliability; tune cluster sizing and autoscaling policies. - Implement data quality, observability, lineage and monitoring (alerts, dashboards, job health) to ensure production SLAs. - Collaborate with Data Engineers, Data Scientists and Architects to define schemas, partitioning strategies and data models that support analytics and ML use cases. - Build CI/CD and release automation for Databricks assets (notebooks, jobs, Delta tables); manage Git-based sou...

Role & Responsibilities : - Design, build, and maintain scalable batch and streaming ETL pipelines on Databricks using Delta Lake and Delta Live Tables (DLT). - Develop and optimize Spark/PySpark jobs for performance, cost-efficiency, and reliability; tune cluster sizing and autoscaling policies. - Implement data quality, observability, lineage and monitoring (alerts, dashboards, job health) to ensure production SLAs. - Collaborate with Data Engineers, Data Scientists and Architects to define schemas, partitioning strategies and data models that support analytics and ML use cases. - Build CI/CD and release automation for Databricks assets (notebooks, jobs, Delta tables); manage Git-based sou...

Role & Responsibilities : - Design, build, and maintain scalable batch and streaming ETL pipelines on Databricks using Delta Lake and Delta Live Tables (DLT). - Develop and optimize Spark/PySpark jobs for performance, cost-efficiency, and reliability; tune cluster sizing and autoscaling policies. - Implement data quality, observability, lineage and monitoring (alerts, dashboards, job health) to ensure production SLAs. - Collaborate with Data Engineers, Data Scientists and Architects to define schemas, partitioning strategies and data models that support analytics and ML use cases. - Build CI/CD and release automation for Databricks assets (notebooks, jobs, Delta tables); manage Git-based sou...

Description : Industry & Sector : Technology consulting firm operating in the blockchain, fintech and distributed ledger technology (DLT) space. We deliver enterprise-grade DLT integrations, ledger analytics and data engineering solutions for global clients, transforming on-chain and off-chain data into actionable business insights. Role : Senior Data Engineer (DLT) Remote (India). Role & Responsibilities : - Design and build scalable ETL/ELT pipelines to ingest, process and transform on-chain and off-chain ledger data using streaming and batch frameworks. - Integrate with DLT nodes, RPC endpoints and indexers to capture transactions, blocks and smart-contract events into analytical stores. ...

Description : Industry & Sector : Technology consulting firm operating in the blockchain, fintech and distributed ledger technology (DLT) space. We deliver enterprise-grade DLT integrations, ledger analytics and data engineering solutions for global clients, transforming on-chain and off-chain data into actionable business insights. Role : Senior Data Engineer (DLT) Remote (India). Role & Responsibilities : - Design and build scalable ETL/ELT pipelines to ingest, process and transform on-chain and off-chain ledger data using streaming and batch frameworks. - Integrate with DLT nodes, RPC endpoints and indexers to capture transactions, blocks and smart-contract events into analytical stores. ...

Description : Industry & Sector : Technology consulting firm operating in the blockchain, fintech and distributed ledger technology (DLT) space. We deliver enterprise-grade DLT integrations, ledger analytics and data engineering solutions for global clients, transforming on-chain and off-chain data into actionable business insights. Role : Senior Data Engineer (DLT) Remote (India). Role & Responsibilities : - Design and build scalable ETL/ELT pipelines to ingest, process and transform on-chain and off-chain ledger data using streaming and batch frameworks. - Integrate with DLT nodes, RPC endpoints and indexers to capture transactions, blocks and smart-contract events into analytical stores. ...

Description : Industry & Sector : Technology consulting firm operating in the blockchain, fintech and distributed ledger technology (DLT) space. We deliver enterprise-grade DLT integrations, ledger analytics and data engineering solutions for global clients, transforming on-chain and off-chain data into actionable business insights. Role : Senior Data Engineer (DLT) Remote (India). Role & Responsibilities : - Design and build scalable ETL/ELT pipelines to ingest, process and transform on-chain and off-chain ledger data using streaming and batch frameworks. - Integrate with DLT nodes, RPC endpoints and indexers to capture transactions, blocks and smart-contract events into analytical stores. ...

Description : Industry & Sector : Technology consulting firm operating in the blockchain, fintech and distributed ledger technology (DLT) space. We deliver enterprise-grade DLT integrations, ledger analytics and data engineering solutions for global clients, transforming on-chain and off-chain data into actionable business insights. Role : Senior Data Engineer (DLT) Remote (India). Role & Responsibilities : - Design and build scalable ETL/ELT pipelines to ingest, process and transform on-chain and off-chain ledger data using streaming and batch frameworks. - Integrate with DLT nodes, RPC endpoints and indexers to capture transactions, blocks and smart-contract events into analytical stores. ...

Description : Industry & Sector : Technology consulting firm operating in the blockchain, fintech and distributed ledger technology (DLT) space. We deliver enterprise-grade DLT integrations, ledger analytics and data engineering solutions for global clients, transforming on-chain and off-chain data into actionable business insights. Role : Senior Data Engineer (DLT) Remote (India). Role & Responsibilities : - Design and build scalable ETL/ELT pipelines to ingest, process and transform on-chain and off-chain ledger data using streaming and batch frameworks. - Integrate with DLT nodes, RPC endpoints and indexers to capture transactions, blocks and smart-contract events into analytical stores. ...

Description : Industry & Sector : Technology consulting firm operating in the blockchain, fintech and distributed ledger technology (DLT) space. We deliver enterprise-grade DLT integrations, ledger analytics and data engineering solutions for global clients, transforming on-chain and off-chain data into actionable business insights. Role : Senior Data Engineer (DLT) Remote (India). Role & Responsibilities : - Design and build scalable ETL/ELT pipelines to ingest, process and transform on-chain and off-chain ledger data using streaming and batch frameworks. - Integrate with DLT nodes, RPC endpoints and indexers to capture transactions, blocks and smart-contract events into analytical stores. ...

Description : Industry & Sector : Technology consulting firm operating in the blockchain, fintech and distributed ledger technology (DLT) space. We deliver enterprise-grade DLT integrations, ledger analytics and data engineering solutions for global clients, transforming on-chain and off-chain data into actionable business insights. Role : Senior Data Engineer (DLT) Remote (India). Role & Responsibilities : - Design and build scalable ETL/ELT pipelines to ingest, process and transform on-chain and off-chain ledger data using streaming and batch frameworks. - Integrate with DLT nodes, RPC endpoints and indexers to capture transactions, blocks and smart-contract events into analytical stores. ...

Description : Industry & Sector : Technology consulting firm operating in the blockchain, fintech and distributed ledger technology (DLT) space. We deliver enterprise-grade DLT integrations, ledger analytics and data engineering solutions for global clients, transforming on-chain and off-chain data into actionable business insights. Role : Senior Data Engineer (DLT) Remote (India). Role & Responsibilities : - Design and build scalable ETL/ELT pipelines to ingest, process and transform on-chain and off-chain ledger data using streaming and batch frameworks. - Integrate with DLT nodes, RPC endpoints and indexers to capture transactions, blocks and smart-contract events into analytical stores. ...

Description : Industry & Sector : Technology consulting firm operating in the blockchain, fintech and distributed ledger technology (DLT) space. We deliver enterprise-grade DLT integrations, ledger analytics and data engineering solutions for global clients, transforming on-chain and off-chain data into actionable business insights. Role : Senior Data Engineer (DLT) Remote (India). Role & Responsibilities : - Design and build scalable ETL/ELT pipelines to ingest, process and transform on-chain and off-chain ledger data using streaming and batch frameworks. - Integrate with DLT nodes, RPC endpoints and indexers to capture transactions, blocks and smart-contract events into analytical stores. ...

You are a SQL Expert with around 5 years of hands-on experience in designing, developing, and optimizing complex SQL queries, stored procedures, and database solutions. Your key role will involve supporting data-driven applications, ensuring efficient data processing, performance tuning, and robust database design. **Responsibilities:** - Design and develop complex SQL queries, stored procedures, views, and functions. - Optimize query performance, indexing strategies, and database tuning. - Develop and maintain ETL/ELT pipelines for data processing and transformation. - Collaborate with developers to design scalable and normalized database schemas. - Analyze and troubleshoot database perform...

Role Overview: At Markovate, we are looking for a highly experienced and innovative Senior Data Engineer who can design, build, and optimize robust, scalable, and production-ready data pipelines across both AWS and Azure platforms. As a Senior Data Engineer, you will be responsible for developing hybrid ETL/ELT pipelines, processing files from AWS S3 and Azure Data Lake Gen2, implementing event-based orchestration, creating scalable ingestion workflows, integrating with metadata and lineage tools, and collaborating with cross-functional teams to align solutions with AI/ML use cases. Key Responsibilities: - Design and develop hybrid ETL/ELT pipelines using AWS Glue and Azure Data Factory (ADF...

Role Overview: As a Data Architect at Inspire Brands, you will play a crucial role in collaborating with the business, Product, and Development teams to qualify, identify, and build Data & Analytics solutions. Your responsibilities include defining technical architecture roadmaps, designing data architectures, creating data flow diagrams, and ensuring the development of efficient data systems. You will also be responsible for becoming a thought leader by sharing best practices, educating technical leaders, and contributing to publications and conferences. Key Responsibilities: - Work closely with product and business teams to understand their strategic, business, and technical needs - Define...

As a GCP Data Engineer specializing in Data Migration & Transformation, your role will involve designing and building robust, scalable data pipelines and architectures on GCP, with a focus on BigQuery. Your responsibilities will include migrating and transforming large-scale data systems to GCP while ensuring performance, scalability, and reliability. You will automate data lineage extraction, maintain data integrity across platforms, and collaborate with stakeholders to implement GCP-native and 3rd-party tools for data ingestion and integration. Additionally, you will optimize complex SQL queries in BigQuery for data analysis, operationalize data pipelines using tools like Apache Airflow, D...

As a skilled Technology Stack Administrator, your role will involve managing and optimizing the on-premises infrastructure which includes RedHat OpenShift, Apache Spark, Apache Airflow, Delta Lake, Apache Kafka, and Debezium. Your responsibilities will encompass a variety of tasks such as platform administration, monitoring & optimization, security & compliance, automation & CI/CD, troubleshooting & support. Key Responsibilities: - Platform Administration: - Install, configure, and maintain RedHat OpenShift clusters in an on-premises environment. - Build and orchestrate data pipelines using Apache Airflow. - Implement and manage Delta Lake for ACID-compliant data lake architecture. - Manage ...

Roles and Responsibility Collaborate with cross-functional teams to identify and prioritize project requirements. Implement data quality checks and validation processes to ensure data integrity. Optimize database performance and troubleshoot issues as needed. Work closely with stakeholders to understand business needs and provide technical solutions. Job Requirements Strong knowledge of Google BigQuery, SQL, Python, Apache Airflow, and Oracle to BigQuery DWH migration and modernization. Experience with data processing technologies such as PySpark and GCS. Familiarity with database management systems like Oracle DB and PL/SQL. Excellent problem-solving skills and attention to detail. Ability ...