15216 Airflow Jobs - Page 2

• Strong knowledge on Python • Airflow Administration - Includes installation/upgrade of Airflow versions (CI/CD), Exposure to executors & POD operators, Optimization of Airflow components to support complex large volume batches • Airflow BAU - Create / Debug / Deploy complex DAGs; Exposure to Airflow APIs and how to integrate them with other systems • Experience on managing Airflow components as containers on Kubernetes • Exposure to Redis cacheing & streams and event driven architecture

JD: Hands-on expertise in GCP data stack: BigQuery, Dataflow (Apache Beam), Dataproc, Cloud Storage, Pub/Sub, Cloud Composer (Airflow) . Strong Spark (PySpark or Scala) for batch processing on Dataproc. Solid Airflow DAG design (idempotent tasks, backfills, retries, SLAs). Advanced SQL and data modeling (star/snowflake, slowly changing dimensions, partition strategies). Proficiency in Python (preferred) or Scala/Java for data engineering. Experience with Git and CI/CD (Cloud Build/GitHub Actions/GitLab CI). Familiarity with security & governance on GCP (IAM, service accounts, secrets management, VPC-SC basics). Strong debugging skills and ownership mindset; clear communication with technical...

JD Senior Data Engineer Responsibilities Design and Develop Scalable Data Pipelines: Build and maintain robust data pipelines using Python to process, transform, and integrate large-scale data from diverse sources. Orchestration and Automation: Implement and manage workflows using orchestration tools such as Apache Airflow to ensure reliable and efficient data operations. Data Warehouse Management: Work extensively with Snowflake to design and optimize data models, schemas, and queries for analytics and reporting. Queueing Systems: Leverage message queues like Kafka, SQS, or similar tools to enable real-time or batch data processing in distributed environments. Collaboration: Partner with Da...

Condé Nast is a global media company producing the highest quality content with a footprint of more than 1 billion consumers in 32 territories through print, digital, video and social platforms. The company’s portfolio includes many of the world’s most respected and influential media properties including Vogue, Vanity Fair, Glamour, Self, GQ, The New Yorker, Condé Nast Traveler/Traveller, Allure, AD, Bon Appétit and Wired, among others. Job Description Location: Chennai, TN About The Role : The Data Solutions Engineering team has a wide range of responsibilities and plays a critical role in shaping how Condé Nast enables its business using data. The team is responsible for building data pipe...

Key Responsibilities: Design, develop, and maintain scalable data pipelines using Apache Spark on Databricks Write efficient and production-ready PySpark or Scala code for data transformation and ETL processes Integrate data from various structured and unstructured sources into a unified platform Implement Delta Lake and manage data versioning, updates, and schema evolution Optimize data processing workflows for performance, scalability, and cost efficiency Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality datasets Implement data quality checks, validation routines, and logging mechanisms Monitor and debug production jobs using Databricks jobs, not...

Job title: Web Scraping Engineer Location: Gurgaon About the Role We are looking for a Web Scraping Engineer with proven experience in building, maintaining, and scaling data extraction systems - especially from e-commerce platforms such as Amazon, Flipkart, Carrefour, etc . The ideal candidate will design and implement robust scrapers to collect, clean, and normalize product data (pricing, availability, reviews, images, etc.) that power our analytics and competitive pricing systems. What you'll do : Develop and maintain scalable scraping pipelines to extract structured data from complex, dynamic e-commerce sites (Amazon, etc.). Design and manage distributed scraping infrastructure (e.g., ro...

Job Summary We are looking for an experienced PostgreSQL Developer with deep expertise in SQL development, schema design, query optimization, stored procedures, and application integration. The ideal candidate will have a strong background in building scalable database solutions, writing complex queries, and collaborating with backend/frontend teams to ensure efficient and reliable data access. This role is focused on development and engineering within PostgreSQL. Roles and Responsibilities Database Design & Development Design normalized and denormalized database schemas for high-performance applications. Develop complex SQL queries, functions, views, triggers, and stored procedures to suppo...

Important Note (Please Read Before Applying) 🚫 Do NOT apply if: • You have less than 10 years of total IT experience • You lack GCP-based data engineering project exposure • You do not have strong Python programming expertise • You are not comfortable with hybrid / client-interfacing roles • Your notice period exceeds 30 days ✅ Apply ONLY if you meet ALL the mandatory skills and experience criteria. Random or unrelated applications will not be processed. Job Title: Lead Data Engineer – Python & GCP Location: Hyderabad Experience: 10+ Years (STRICTLY) Employment Type: Permanent / Full-Time Notice Period: Immediate Joiners only About the Company: Our client is a trusted global innovator of IT ...

Job Title: Data Engineer Location: Hyderabad (Hybrid) Experience: 6+ Years (STRICTLY) Employment Type: Permanent Notice Period: Immediate Joiners Only Mode of Interview: Face to Face at client location, Hyderabad About the Company Our client is a trusted global innovator of IT and business services, present in 50+ countries. They specialize in digital & IT modernization, consulting, managed services, and industry-specific solutions. With a commitment to long-term success, they empower clients and society to move confidently into the digital future. About the Role We are seeking a highly skilled Senior Data Engineer with 6+ years of experience to design, build, and optimize large-scale data p...

Responsibilities:  Design the data pipelines and engineering infrastructure to support our clients’ enterprise machine learning systems at scale  Take offline models data scientists build and turn them into a real machine learning production system  Develop and deploy scalable tools and services for our clients to handle machine learning training and inference  Identify and evaluate new technologies to improve performance, maintainability, and reliability of our clients’ machine learning systems  Apply software engineering rigor and best practices to machine learning, including CI/CD, automation, etc.  Support model development, with an emphasis on auditability, versioning, and data se...

Data Engineering concepts (Data Warehouse (DW), Data Lake, OLTP vs OLAP, etc.); API Development (Python); Docker & Kubernetes; Airflow; AWS project;Strong communication skills

The Skills that will make you successful in this role Strong Java/J2EE, Object Oriented design and development skills. Core Java, Spring Framework (IOC, AOP, MVC), Spring Boot, REST Web Services, Multithreading Hands on Cloud Computing (AWS), Containers (Docker), Orchestration (Kubernetes) Test Automation tool (Karate, Cucumber, Selenium, JMeter etc.,) Knowledge on Performance Testing is a plus. Development tools (GitHub, SonarQube, Jenkins Core etc.,) Experienced in Agile SCRUM practices Excellent presentation, documentation and communication skills The Skills that are good to have for this role SQL proficiency (Oracle experience preferred), ORM Tools (Spring Data JPA) Messaging platforms (...

Experience: 6 to 10 years of hands-on experience in the data analytics space as a data engineer. Familiar with ETL, DQ, DM, and reject and recycling concepts. A significant portion of this experience should involve building data analytics solutions in a big data environment using Hadoop clusters or cloud environment. Technical Skills: Extensive experience in building data pipelines using Spark, particularly PySpark. Candidates must have a minimum of 3 years of hands-on experience in coding with PySpark applications using RRDs, DataFrames & datasets and NOT Spark SQLs. Candide should has developed numerous spark application for various use cases of processing large volumes of data, used perfo...

About Position: We are looking for Lead Azure Data engineer who have experience in Azure data bricks, Data factory, to join our development team. Role: Azure Data Lead Engineer Location: Pune/ Bangalore Experience: 8 to 10 years Job Type: Full Time Employment What You'll Do: Design and implement complex, scalable data pipelines for ingestion, processing, and transformation using Azure Databricks technologies. Collaborate with Architects, Data Analysts, and Business Analysts to understand data requirements and develop efficient workflows. Develop and manage data storage solutions including Azure SQL Database, Data Lake, and Blob Storage. Leverage Azure Data Factory and other cloud-native tool...

About Position: We are looking for Sr Azure Data engineer who have experience in Azure data bricks, Data factory, to join our development team. Role: Sr. Azure Data Engineer Location: Pune/Bangalore Experience: 5 to 8 years Job Type: Full Time Employment What You'll Do: Design and implement complex, scalable data pipelines for ingestion, processing, and transformation using Azure Databricks technologies. Collaborate with Architects, Data Analysts, and Business Analysts to understand data requirements and develop efficient workflows. Develop and manage data storage solutions including Azure SQL Database, Data Lake, and Blob Storage. Leverage Azure Data Factory and other cloud-native tools to ...

About the Role We are seeking a highly skilled Data Engineer with deep expertise in PySpark and the Cloudera Data Platform (CDP) to join our data engineering team. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines that ensure high data quality and availability across the organization. This role requires a strong background in big data ecosystems, cloud-native tools, and advanced data processing techniques. The ideal candidate has hands-on experience with data ingestion, transformation, and optimization on the Cloudera Data Platform, along with a proven track record of implementing data engineering best practices. You will work clos...

Responsibilities: Design & Modeling: Design and develop mechanical components and assemblies for new and existing genset models, with a focus on canopies (enclosures), baseframes, and overall genset layout creation. 3D/2D CAD: Utilize Creo as the primary CAD software to create precise 3D models, detailed engineering drawings, bills of materials (BOMs), and assembly instructions. Component Selection: Lead the technical selection and specification of key genset components, including but not limited to radiators, fuel tanks, exhaust systems, vibration isolators, and electrical control panel mounting. Design Analysis: Perform necessary engineering calculations (e.g., structural analysis, airflow...

Position: Data Scientist Location: Hyderabad Experience: 7 Years About the Role: We’re looking for a Data Scientist with strong analytical and programming skills to develop and deploy predictive models, perform statistical analysis, and generate actionable insights from large datasets. The ideal candidate will have hands-on experience with Python, ML algorithms, visualization tools, and cloud-based AI platforms . Key Responsibilities: Build and deploy predictive models using machine learning algorithms and statistical techniques . Perform data preprocessing, feature engineering, and model evaluation. Visualize trends and insights using Power BI, Tableau, or Python libraries (Matplotlib, Seab...

Experience : 2.00 + years Salary : USD 30000-42000 / year (based on experience) Expected Notice Period : 15 Days Shift : (GMT+05:30) Asia/Kolkata (IST) Opportunity Type : Remote Placement Type : Full Time Indefinite Contract(40 hrs a week/160 hrs a month) (*Note: This is a requirement for one of Uplers' client - Talia Healthcare) What do you need for this opportunity? Must have skills required: Athena, EHR Integrations, Healthie, Healthcare, n8n, Zapier, AWS Talia Healthcare is Looking for: Workflow Automation & Integrations Engineer (Healthcare / AWS / n8n) Location: Remote (India) Engagement: Full-time (Contractor) Compensation: Up to USD 3,500/month Timezone: Flexible, with a few overlapp...

Experience : 2.00 + years Salary : USD 30000-42000 / year (based on experience) Expected Notice Period : 15 Days Shift : (GMT+05:30) Asia/Kolkata (IST) Opportunity Type : Remote Placement Type : Full Time Indefinite Contract(40 hrs a week/160 hrs a month) (*Note: This is a requirement for one of Uplers' client - Talia Healthcare) What do you need for this opportunity? Must have skills required: Athena, EHR Integrations, Healthie, Healthcare, n8n, Zapier, AWS Talia Healthcare is Looking for: Workflow Automation & Integrations Engineer (Healthcare / AWS / n8n) Location: Remote (India) Engagement: Full-time (Contractor) Compensation: Up to USD 3,500/month Timezone: Flexible, with a few overlapp...

Experience : 2.00 + years Salary : USD 30000-42000 / year (based on experience) Expected Notice Period : 15 Days Shift : (GMT+05:30) Asia/Kolkata (IST) Opportunity Type : Remote Placement Type : Full Time Indefinite Contract(40 hrs a week/160 hrs a month) (*Note: This is a requirement for one of Uplers' client - Talia Healthcare) What do you need for this opportunity? Must have skills required: Athena, EHR Integrations, Healthie, Healthcare, n8n, Zapier, AWS Talia Healthcare is Looking for: Workflow Automation & Integrations Engineer (Healthcare / AWS / n8n) Location: Remote (India) Engagement: Full-time (Contractor) Compensation: Up to USD 3,500/month Timezone: Flexible, with a few overlapp...

Experience : 2.00 + years Salary : USD 30000-42000 / year (based on experience) Expected Notice Period : 15 Days Shift : (GMT+05:30) Asia/Kolkata (IST) Opportunity Type : Remote Placement Type : Full Time Indefinite Contract(40 hrs a week/160 hrs a month) (*Note: This is a requirement for one of Uplers' client - Talia Healthcare) What do you need for this opportunity? Must have skills required: Athena, EHR Integrations, Healthie, Healthcare, n8n, Zapier, AWS Talia Healthcare is Looking for: Workflow Automation & Integrations Engineer (Healthcare / AWS / n8n) Location: Remote (India) Engagement: Full-time (Contractor) Compensation: Up to USD 3,500/month Timezone: Flexible, with a few overlapp...

Experience : 2.00 + years Salary : USD 30000-42000 / year (based on experience) Expected Notice Period : 15 Days Shift : (GMT+05:30) Asia/Kolkata (IST) Opportunity Type : Remote Placement Type : Full Time Indefinite Contract(40 hrs a week/160 hrs a month) (*Note: This is a requirement for one of Uplers' client - Talia Healthcare) What do you need for this opportunity? Must have skills required: Athena, EHR Integrations, Healthie, Healthcare, n8n, Zapier, AWS Talia Healthcare is Looking for: Workflow Automation & Integrations Engineer (Healthcare / AWS / n8n) Location: Remote (India) Engagement: Full-time (Contractor) Compensation: Up to USD 3,500/month Timezone: Flexible, with a few overlapp...

Experience : 2.00 + years Salary : USD 30000-42000 / year (based on experience) Expected Notice Period : 15 Days Shift : (GMT+05:30) Asia/Kolkata (IST) Opportunity Type : Remote Placement Type : Full Time Indefinite Contract(40 hrs a week/160 hrs a month) (*Note: This is a requirement for one of Uplers' client - Talia Healthcare) What do you need for this opportunity? Must have skills required: Athena, EHR Integrations, Healthie, Healthcare, n8n, Zapier, AWS Talia Healthcare is Looking for: Workflow Automation & Integrations Engineer (Healthcare / AWS / n8n) Location: Remote (India) Engagement: Full-time (Contractor) Compensation: Up to USD 3,500/month Timezone: Flexible, with a few overlapp...

Experience : 2.00 + years Salary : USD 30000-42000 / year (based on experience) Expected Notice Period : 15 Days Shift : (GMT+05:30) Asia/Kolkata (IST) Opportunity Type : Remote Placement Type : Full Time Indefinite Contract(40 hrs a week/160 hrs a month) (*Note: This is a requirement for one of Uplers' client - Talia Healthcare) What do you need for this opportunity? Must have skills required: Athena, EHR Integrations, Healthie, Healthcare, n8n, Zapier, AWS Talia Healthcare is Looking for: Workflow Automation & Integrations Engineer (Healthcare / AWS / n8n) Location: Remote (India) Engagement: Full-time (Contractor) Compensation: Up to USD 3,500/month Timezone: Flexible, with a few overlapp...