15216 Airflow Jobs - Page 7

About The Role Data engineers are responsible for building reliable and scalable data infrastructure that enables organizations to derive meaningful insights, make data-driven decisions, and unlock the value of their data assets. About The Role - Grade Specific The role support the team in building and maintaining data infrastructure and systems within an organization. Skills (competencies) Ab Initio Agile (Software Development Framework) Apache Hadoop AWS Airflow AWS Athena AWS Code Pipeline AWS EFS AWS EMR AWS Redshift AWS S3 Azure ADLS Gen2 Azure Data Factory Azure Data Lake Storage Azure Databricks Azure Event Hub Azure Stream Analytics Azure Sunapse Bitbucket Change Management Client Ce...

Myridius x Aethereus is looking for Data Engineer to join our dynamic team and embark on a rewarding career journey Liaising with coworkers and clients to elucidate the requirements for each task. Conceptualizing and generating infrastructure that allows big data to be accessed and analyzed. Reformulating existing frameworks to optimize their functioning. Testing such structures to ensure that they are fit for use. Preparing raw data for manipulation by data scientists. Detecting and correcting errors in your work. Ensuring that your work remains backed up and readily accessible to relevant coworkers. Remaining up-to-date with industry standards and technological advancements that will impro...

hackajob is collaborating with Comcast to connect them with exceptional tech professionals for this role. Comcast brings together the best in media and technology. We drive innovation to create the world's best entertainment and online experiences. As a Fortune 50 leader, we set the pace in a variety of innovative and fascinating businesses and create career opportunities across a wide range of locations and disciplines. We are at the forefront of change and move at an amazing pace, thanks to our remarkable people, who bring cutting-edge products and services to life for millions of customers every day. If you share in our passion for teamwork, our vision to revolutionize industries and our ...

We are seeking a passionate and skilled Data Engineer to join our dynamic US OBU Pelican report engineering team In this role, you will be responsible for designing, building, and maintaining scalable and efficient data pipelines and workflows that support essential business reporting and analytics functions This is a collaborative and impactful role, where your contributions will directly influence data-driven decision-making across the organization As a Data Engineer, you will design, develop, and optimize data pipelines using SQL and Python, ensuring they are robust and efficient for handling large volumes of data You will leverage platforms such as Databricks to implement complex data tr...

7+ years of experience designing and building data pipelines using Apache Spark, Databricks or equivalent bigdata frameworks. Handson expertise with streaming and messaging systems such as Apache Kafka (publish subscribe architecture), Confluent Cloud, RabbitMQ or Azure Event Hub. Experience creating producers, consumers and topics and integrating them into downstream processing. Deep understanding of relational databases and CDC. Proficiency in SQL Server, Oracle or other RDBMSs; experience capturing change events using Debezium or native CDC tools and transforming them for downstream consumption. Proficiency in programming languages such as Python, Scala or Java and solid knowledge of SQL ...

Bachelors degree or higher in Computer Science, Engineering, Mathematics, Statistics, or a related discipline. Substantial experience designing and managing ML workflows in Databricks, utilizing both provisioned clusters and serverless compute resources on AWS. Demonstrated advanced programming skills in Python and SQL for data engineering and pipeline development. Expertise using MLflow for experiment tracking, model management, and deployment within Databricks. Working knowledge of ML frameworks, libraries, and technologies (e.g. TensorFlow, PyTorch, scikit-learn, XGBoost, MLflow). Hands-on experience provisioning and optimizing compute resources for ML applications. Experience with workfl...

Strong expertise in SQL and PL/SQL. Hands-on experience with DBT (not just theoretical knowledge). Ability to reverse engineer Oracle code and convert to DBT models. Familiarity with high-volume data processing. Optional Experience with Java (plus, not mandatory) & Knowledge of Airflow (plus, not mandatory). Suggest performance improvements (e.g., clustering, merge optimization). Work independently without hand-holding. Collaborate with existing frameworks and standards.

Key Responsibilities: Design and implement scalable and efficient full-stack solutions using Java and cloud technologies Develop and maintain cloud-based solutions on Google Cloud Platform (GCP), utilizing services like BigQuery, Astronomer, Terraform, Airflow, and Dataflow Architect and implement complex data engineering solutions using GCP services Collaborate with cross-functional teams to develop, deploy, and optimize cloud-based applications Utilize Python for data engineering and automation tasks within the cloud environment Ensure alignment with GCP architecture best practices and contribute to the design of high-performance systems Lead and mentor junior developers, fostering a cultu...

Key Responsibilities: Develop and maintain scalable full-stack applications using Java, Spring Boot, and Angular for building rich UI screens and custom/reusable components. Design and implement cloud-based solutions leveraging Google Cloud Platform (GCP) services such as BigQuery, Google Cloud Storage, Cloud Run, and PubSub. Manage and optimize CI/CD pipelines using Tekton to ensure smooth and efficient development workflows. Deploy and manage Google Cloud services using Terraform, ensuring infrastructure as code principles. Mentor and guide junior software engineers, fostering professional development and promoting systemic change across the development team. Collaborate with cross-functio...

Job Title: Foreman & Chargehand – HVAC & Ducting Location: Dubai, UAE Industry: Construction / MEP Company: Reputed MEP Contracting Company Job Type: Full-Time Job Summary The Foreman & Chargehand – HVAC & Ducting will oversee, coordinate, and execute all HVAC installation and ducting works at the project site. The role requires strong technical skills, manpower supervision, quality control, and the ability to deliver work as per approved drawings, project schedules, and safety compliance. Key Responsibilities HVAC & Ducting Installation Supervise and execute installation of HVAC systems, including ducting, GI/PI ducts, chilled water piping, FCUs, AHUs, and exhaust systems. Ensure duct fabri...

Location Gurugram - Haryana, India FORVIA HELLA is a listed international automotive supplier. As a company of the FORVIA Group, FORVIA HELLA stands for high-performance lighting technology and vehicle electronics and, with the Lifecycle Solutions Business Group, also covers a broad service and product portfolio for the spare parts and workshop business as well as for manufacturers of special vehicles. With currently around 36,500 employees at over 125 locations, the Company is active worldwide and generated adjusted sales of €8.1 billion in fiscal year 2024. YOUR TASKS We’re hiring a Lead Data Engineer to join our Data Engineering team as a high-impact individual contributor. This team powe...

Job Title: Foreman & Chargehand – HVAC & Ducting Location: Dubai, UAE Industry: Construction / MEP Company: Reputed MEP Contracting Company Job Type: Full-Time Job Summary The Foreman & Chargehand – HVAC & Ducting will oversee, coordinate, and execute all HVAC installation and ducting works at the project site. The role requires strong technical skills, manpower supervision, quality control, and the ability to deliver work as per approved drawings, project schedules, and safety compliance. Key Responsibilities HVAC & Ducting Installation Supervise and execute installation of HVAC systems, including ducting, GI/PI ducts, chilled water piping, FCUs, AHUs, and exhaust systems. Ensure duct fabri...

Job Purpose and Impact The Professional, Trading Engineering job maintains and partners to design and develop software applications and systems that drive the trading operations. With limited supervision, this job participates in co-creation and maintenance of sophisticated software applications vital to our trading activities, collaboration with cross functional teams to conceptualize, design and implement new software features that meet user needs and advance our business objectives, including upgrading existing software by writing clean, efficient and scalable code to improve performance and functionality, and identifying and resolving issues promptly to ensure the seamless operation of t...

Job Title: Foreman & Chargehand – HVAC & Ducting Location: Dubai, UAE Industry: Construction / MEP Company: Reputed MEP Contracting Company Job Type: Full-Time Job Summary The Foreman & Chargehand – HVAC & Ducting will oversee, coordinate, and execute all HVAC installation and ducting works at the project site. The role requires strong technical skills, manpower supervision, quality control, and the ability to deliver work as per approved drawings, project schedules, and safety compliance. Key Responsibilities HVAC & Ducting Installation Supervise and execute installation of HVAC systems, including ducting, GI/PI ducts, chilled water piping, FCUs, AHUs, and exhaust systems. Ensure duct fabri...

We are seeking a highly skilled and passionate Senior Solution Architect – Generative AI to join our team and lead the design and implementation of cutting-edge AI solutions. This role is ideal for a seasoned professional with extensive experience in AI/ML development and architecture, a deep understanding of generative AI technologies, and a strategic mindset to align innovations with business requirements. Responsibilities Develop and oversee architectural designs for generative AI models, frameworks, and solutions tailored to business needs Design scalable pipelines for integrating and deploying generative AI solutions in alignment with enterprise architecture Perform in-depth research to...

3+ years experienced engineer who has worked on GCP environment and its relevant tools/services. (Big Query, Data Proc, Data flow, Cloud Storage, Terraform, Tekton , Cloudrun , Cloud scheduler, Astronomer/Airflow, Pub/sub, Kafka, Cloud Spanner streaming etc) 1 or 2 + years of strong experience in Python development (Object oriented/Functional Programming, Pandas, Pyspark etc) 1 or 2 + years of strong experience in SQL Language (CTEs, Window functions, Aggregate functions etc) Mandatory Key Skills terraform,oops,cloud spanner,hadoop,java,hive,spark,mapreduce,big data,gcp,aws, javascript, mysql, postgresql,sql server,oracle,bigtable,software development,sql,python development,python,bigquery,p...

We are seeking a detail-oriented and analytical Data Analyst to join our team. The ideal candidate should have strong hands-on experience in SQL/PL-SQL, data modeling, and data extraction from large datasets. The role requires strong analytical thinking, the ability to work with cross-functional teams, and a quick learning mindset. Key Responsibilities Extract, clean, and analyze data from multiple sources to support business and operational requirements. Write complex SQL / PL-SQL queries to perform data extraction, transformation, and analysis. Work on data modeling for operational, transactional, and analytical use cases. Develop dashboards and reports using Tableau / Power BI / Advanced ...

We are looking for a Full stack UI Developer to join our HIDA Technology team in Bangalore. This is an amazing opportunity to work on HIDA, a platform supporting analytics and insights in the healthcare distribution space. The team consists of 8 members and reports to the Senior Manager. We have a great skill set in Angular, Python-based backend services, data modeling, and Snowflake data pipelines, and we would love to speak with you if you have skills in building modern UI applications, Snowflake integration with Python-based applications, and developing scalable full stack solutions. About You experience, education, skills, and accomplishments Bachelors Degree (or equivalent) in Computer ...

Job Title: Foreman & Chargehand – HVAC & Ducting Location: Dubai, UAE Industry: Construction / MEP Company: Reputed MEP Contracting Company Job Type: Full-Time Job Summary The Foreman & Chargehand – HVAC & Ducting will oversee, coordinate, and execute all HVAC installation and ducting works at the project site. The role requires strong technical skills, manpower supervision, quality control, and the ability to deliver work as per approved drawings, project schedules, and safety compliance. Key Responsibilities HVAC & Ducting Installation Supervise and execute installation of HVAC systems, including ducting, GI/PI ducts, chilled water piping, FCUs, AHUs, and exhaust systems. Ensure duct fabri...

Job Title: Foreman & Chargehand – HVAC & Ducting Location: Dubai, UAE Industry: Construction / MEP Company: Reputed MEP Contracting Company Job Type: Full-Time Job Summary The Foreman & Chargehand – HVAC & Ducting will oversee, coordinate, and execute all HVAC installation and ducting works at the project site. The role requires strong technical skills, manpower supervision, quality control, and the ability to deliver work as per approved drawings, project schedules, and safety compliance. Key Responsibilities HVAC & Ducting Installation Supervise and execute installation of HVAC systems, including ducting, GI/PI ducts, chilled water piping, FCUs, AHUs, and exhaust systems. Ensure duct fabri...

We are seeking a highly skilled and passionate Senior Solution Architect – Generative AI to join our team and lead the design and implementation of cutting-edge AI solutions. This role is ideal for a seasoned professional with extensive experience in AI/ML development and architecture, a deep understanding of generative AI technologies, and a strategic mindset to align innovations with business requirements. Responsibilities Develop and oversee architectural designs for generative AI models, frameworks, and solutions tailored to business needs Design scalable pipelines for integrating and deploying generative AI solutions in alignment with enterprise architecture Perform in-depth research to...

Required Skills: 3- 6 years of relevant experience in data engineering, data analytics, or BI development. Proven hands-on experience with Google BigQuery and GCP data services (e.g., Cloud Storage, Cloud Functions, Dataflow optional). Strong expertise in SQL , including analytical functions, joins, CTEs, and window functions. Basic to intermediate knowledge of Python for scripting and automation. Understanding of data warehouse concepts , data modeling, and normalization. Familiarity with version control systems like Git.

Duties & Responsibilities Design, develop, and maintain scalable ETL/ELT data pipelines to support business and analytics needs Write, tune, and optimize complex SQL queries for data transformation, aggregation, and analysis Translate business requirements into well-designed, documented, and reusable data solutions Partner with analysts, data scientists, and stakeholders to deliver accurate, timely, and trusted datasets Automate data workflows using orchestration/scheduling tools (Airflow, ADF, Luigi, etc.) Develop unit tests, integration tests, and validation checks to ensure data accuracy and pipeline reliability Document pipelines, workflows, and design decisions for knowledge sharing and...

Key Responsibilities Data Integration & Extraction Develop and maintain ETL/ELT pipelines using Airbyte and Airflow. Extract data from multiple sources including APIs, direct database connections, and flat files. Ensure data quality, consistency, and reliability across all ingestion processes. Data Modeling & Warehousing Design and implement Kimball-style dimensional models in Snowflake. Build and optimize fact tables and dimension tables to support analytical workloads. Collaborate with business teams to define and maintain the BUS Matrix for subject areas . Transformation & Orchestration Use DBT to develop modular, testable, and version-controlled transformations. Implement data quality ch...

We are seeking a highly skilled and passionate Senior Solution Architect – Generative AI to join our team and lead the design and implementation of cutting-edge AI solutions. This role is ideal for a seasoned professional with extensive experience in AI/ML development and architecture, a deep understanding of generative AI technologies, and a strategic mindset to align innovations with business requirements. Responsibilities Develop and oversee architectural designs for generative AI models, frameworks, and solutions tailored to business needs Design scalable pipelines for integrating and deploying generative AI solutions in alignment with enterprise architecture Perform in-depth research to...