974 Apache Airflow Jobs - Page 4

5-9 years In Data Engineering, software development such as ELT/ETL, data extraction and manipulation in Data Lake/Data Warehouse environment Expert level Hands to the following: Python, SQL PySpark DBT and Apache Airflow DevOps, Jenkins, CI/CD Data Governance and Data Quality frameworks Data Lakes, Data Warehouse AWS services including S3, SNS, SQS, Lambda, EMR, Glue, Athena, EC2, VPC etc. Source code control - GitHub, VSTS etc. Mandatory Competencies Python - Python Database - SQL Data on Cloud - AWS S3 DevOps - CI/CD DevOps - Github Beh - Communication Big Data - Big Data - Pyspark

Project description Luxoft has been asked to contract a Developer in support of a number of customer initiatives. The primary objective is to develop based on client requirements in the Telecom/network work environment Responsibilities A Data Engineer with experience in the following techologies: Databricks and Azure Apache Spark-based, hands on Python, SQL, Apache Airflow. Databricks clusters for ETL processes. Integration with ADLS, Blob Storage. Efficiently ingest data from various sources, including on-premises databases, cloud storage, APIs, and streaming data. Use Azure Key Vault for managing secrets. Hands on experience working with API's Kafka/Azure EventHub streaming hands on experi...

Project description We need a Senior Python and Pyspark Developer to work for a leading investment bank client. Responsibilities Develop software applications based on business requirements Maintain software applications and make enhancements according to project specifications Participate in requirement analysis, design, development, testing, and implementation activities Propose new techniques and technologies for software development. Perform unit testing and user acceptance testing to evaluate application functionality Ensure to complete the assigned development tasks within the deadlines Work in compliance with coding standards and best practices Provide assistance to Junior Developers ...

We are seeking a skilled and motivated Backend Developer with expertise in FastAPI , Pub/Sub messaging systems , and Apache Airflow to build and maintain scalable, cloud-native applications on AWS . The ideal candidate has strong experience in modern Python development and is having strong hands-on experience with event-driven architectures and data workflow orchestration in AWS cloud environments. Required Qualifications: Bachelors degree in computer science, data science, or a related technical discipline. 7+ years of hands-on experience in data engineering, include developing ETL/ElT data pipeline, API Integration (Fast API Preferred), data platform/products and or data warehouse. 3+ year...

Mandatory skills : AWS,Pyspark, Apache airflow. Location: Bangalore and chennai. Experience : 6-12 yrs. DE Job Description We are seeking a skilled and motivated Backend Developer with expertise in FastAPI , Pub/Sub messaging systems , and Apache Airflow to build and maintain scalable, cloud-native applications on AWS . The ideal candidate has strong experience in modern Python development and is having strong hands-on experience with event-driven architectures and data workflow orchestration in AWS cloud environments. Required Qualifications: Bachelors degree in computer science, data science, or a related technical discipline. 7+ years of hands-on experience in data engineering, include de...

As an Applications Development Senior Programmer Analyst at our company, you will play a crucial role in establishing and implementing new or revised application systems and programs in coordination with the Technology team. Your main responsibilities will involve conducting feasibility studies, providing IT planning, developing new applications, and offering user support. You will need to leverage your specialty knowledge to analyze complex problems, recommend security measures, and consult with users on advanced programming solutions. Additionally, you will be responsible for ensuring operating standards are followed and acting as an advisor to junior analysts. **Key Responsibilities:** - ...

You are expected to have proficiency in Python with experience in Databricks (PySpark). It is mandatory for you to possess these skills. Additionally, it would be beneficial if you have hands-on experience with Apache Airflow and working knowledge of PostgreSQL and MongoDB. Basic experience on cloud technologies such as Azure, AWS, and Google is considered a plus.,

As a Data Engineer at Synram Software Services Pvt. Ltd., a subsidiary of FG International GmbH, you will play a crucial role in delivering innovative IT solutions across various industries. You will work on designing, building, and maintaining scalable data pipelines and infrastructure to facilitate data-driven decision-making throughout the organization. Your key responsibilities in this role will include: - Developing, testing, and maintaining data architectures - Designing and implementing ETL processes - Optimizing data systems - Collaborating with cross-functional teams to understand data requirements - Ensuring data quality, integrity, and security - Automating repetitive data tasks -...

As a Senior Python Integration Developer at HCL Software, you will be responsible for designing, developing, and implementing integration solutions for HCL's Marketing Platform (Unica) and Customer Data Platform (CDP) using Python, Microservices, REST APIs, and API Gateways. Your role will involve translating conceptual system and business requirements into technical data and integration requirements, developing efficient, scalable, and reliable APIs and Connectors, and creating automated Unit Tests and monitoring scripts to ensure the reliability and high quality of the integrations. You will also be expected to participate in research, make recommendations on integration products and servi...

As a Data Pipeline Engineer at Atomicwork, your main responsibility will be to design, construct, and maintain scalable data pipelines that support our enterprise search capabilities. Your efforts will ensure that data from diverse sources is efficiently ingested, processed, and indexed, facilitating seamless and secure search experiences across the organization. **Key Responsibilities:** - Designing, developing, and maintaining data pipelines for enterprise search applications - Implementing data ingestion processes from various sources - Developing data transformation and enrichment processes - Integrating with search platforms - Ensuring data quality and integrity - Monitoring pipeline pe...

Role Overview: You will be responsible for fetching and transforming data from various systems, conducting in-depth analyses to identify gaps, opportunities, and insights, and providing recommendations that support strategic business decisions. Key Responsibilities: - Data Extraction and Transformation: - Fetch data from multiple systems, transform it to create a unified dataset, and ensure data accuracy. - Data Analysis and Insight Generation: - Analyze large datasets to identify patterns, gaps, and trends. - Develop actionable insights and recommendations that support business objectives. - Visualization and Reporting: - Create impactful visualizations and reports to effectively communicat...

At EY, youll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we're counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. **Role Overview:** As a member of our team, your role will involve investigating, researching, reviewing, evaluating, and eliciting requirements. You will conduct risks assessment and provide mitigation recommendations. Additionally, you will be responsible for performing functional data or back-end test executions by writing SQL and automatio...

As a Data Engineer, you will be responsible for designing and building efficient data pipelines using Azure Databricks (PySpark). You will implement business logic for data transformation and enrichment at scale, as well as manage and optimize Delta Lake storage solutions. Additionally, you will develop REST APIs using FastAPI to expose processed data and deploy them on Azure Functions for scalable and serverless data access. Key Responsibilities: - Design and build efficient data pipelines using Azure Databricks (PySpark) - Implement business logic for data transformation and enrichment at scale - Manage and optimize Delta Lake storage solutions - Develop REST APIs using FastAPI to expose p...

As part of our expanding AI-driven teams, we are looking for developers, engineers, and researchers who are enthusiastic about shaping the future of artificial intelligence systems. Your strength may lie in various areas such as building user interfaces, developing back-end logic, training AI models, or designing infrastructure. Regardless of your expertise, we have a suitable position for you! Read on to identify which area aligns best with your skills and interests. If you are a Front-End Developer, your role will involve designing and implementing user-facing applications and dashboards. Key Skills required include React, JavaScript/TypeScript, TailwindCSS, Figma, and knowledge of UX prin...

In your role at Sundew, a leading digital transformation firm, you will be responsible for designing and optimizing data pipelines, supporting data-driven initiatives, and contributing to the growth of the organization. As a skilled and experienced Data Engineer, you will play a crucial role in the development and maintenance of scalable data pipelines. You will work with tools like Apache Airflow, AWS Glue, or similar technologies to support ETL processes and ensure efficient data processing. Key Responsibilities: - Design, develop, and maintain scalable data pipelines for ETL processes using tools like Apache Airflow, AWS Glue, etc. - Design, optimize, and manage relational and NoSQL datab...

General Summary: As a leading technology innovator, Qualcomm pushes the boundaries of what's possible to enable next-generation experiences and drives digital transformation to help create a smarter, connected future for all. As a Qualcomm Hardware Engineer, you will plan, design, optimize, verify, and test electronic systems, bring-up yield, circuits, mechanical systems, Digital/Analog/RF/optical systems, equipment and packaging, test systems, FPGA, and/or DSP systems that launch cutting-edge, world class products. Qualcomm Hardware Engineers collaborate with cross-functional teams to develop solutions and meet performance requirements. Minimum Qualifications: Bachelor's degree in Computer ...

As a Data Engineer, you will play a crucial role in designing, developing, and maintaining robust ELT pipelines using modern data engineering tools and practices. Your responsibilities will include: - Working with distributed, column-oriented databases such as Vertica and Snowflake to build high-performance data solutions. - Implementing and managing data workflows using orchestration tools like Apache Airflow. - Performing root cause analysis and troubleshooting data issues to ensure data integrity and availability. - Collaborating with cross-functional teams to understand data requirements and deliver scalable solutions. - Documenting data processes, workflows, and architecture for transpa...

We are looking for a skilled Data Engineer with expertise in PySpark, AWS, and SQL to support data processing and analytical initiatives. This role involves working closely with data engineering and data science teams to build, maintain, and optimize large-scale data pipelines and integrations on AWS. The ideal candidate will be proficient in ETL processes using PySpark and SQL, with a deep understanding of cloud data infrastructure, specifically within the AWS ecosystem. Key Responsibilities : Data Pipeline Development : - Design, build, and optimize ETL pipelines using PySpark for data ingestion, transformation, and storage on AWS. - Collaborate with stakeholders to understand data require...

About the Role Were seeking a highly skilled Data Engineer with a strong development background and a passion for transforming data into valuable insights. The ideal candidate will play a key role in designing, building, and maintaining scalable data pipelines and analytics solutions that support critical business decisions. What youll be doing Design, build, and optimize robust data pipelines for large-scale data processing Write complex SQL queries for data extraction, transformation, and reporting Collaborate with analytics and reporting teams to deliver data-driven insights Develop scalable solutions using programming languages such as Java, Python, and Node. js Integrate APIs and third-...

Job Responsibilities : Airflow Administration and Data Analytics : - Administer and manage Airflow for scheduling and monitoring complex data workflows. - Utilize open -source technologies like Docker, Kubernetes, and container -based microservices to develop scalable solutions. - Provide installation and support for Business Intelligence systems such as SAP BODS, Collibra, HVR (Fivetran), Qlik Replicate, SAP SLT, and SAP BOXI (BOBI).Database Administration and Development : - Install, configure, and maintain databases like PostgreSQL, MongoDB, and RabbitMQ across cloud platforms such as Azure and AWS. - Develop ETL processes to extract, transform, and load data from various sources into dat...

Contract Duration : 3 months Skills : - PySpark or Scala with Spark, Spark Architecture, Hadoop, SQL - Streaming Technologies like Kafka etc.- Proficiency in Advanced SQL (Window functions)- Airflow, S3, and Stream Sets or similar ETL tools.- Basic Knowledge on AWS IAM, AWS EMR and Snowflake. Responsibilities : - Data Pipeline Development: Design, implement, and maintain scalable and efficient data pipelines to collect, process, and store large volumes of structured and unstructured data. - Data Modeling: Develop and maintain data models, schemas, and metadata to support the organization's data initiatives. Ensure data integrity and optimize data storage and retrieval processes. - Data Integ...

Job Responsibilities : MDM Data Vault Design and Development : - Efficiently design and develop DBT-based data models and entities by understanding project requirements. - Implement Data Vault 2.0 architecture principles, including the creation of Hubs, Links, and Satellites for master data management (MDM). Data Warehousing : - Design, develop, and maintain robust data structures to facilitate efficient data processing, storage, and retrieval within an enterprise data warehouse (Snowflake). - Collaborate with data architects to ensure data warehousing solutions align with business needs and scalability requirements. SQL Development : - Create, debug, and optimize stored procedures, function...

Role Overview: As a Retail Specialized Data Scientist at Level 9 in the SnC GN Data & AI team, you will be responsible for utilizing advanced analytics, machine learning, and statistical modeling techniques to support data-driven decision-making in the retail sector. Your role will involve collaborating with various teams such as marketing, product management, supply chain, and customer insights to drive business strategies and innovations. The ideal candidate should possess experience in retail analytics and demonstrate the ability to transform data into actionable insights. Key Responsibilities: - Leverage Retail Knowledge: Utilize your in-depth understanding of the retail industry, includ...

You will be part of the PricewaterhouseCoopers (PwC) team, a global consulting firm dedicated to building trust in society and solving important problems for clients and communities worldwide. As a member of the US Advisory Bangalore Acceleration Center, you will play a crucial role in supporting various practice teams and contributing to PwC's success in the marketplace. **Key Responsibilities:** - Supervise, develop, and coach teams to deliver complex analytics solutions - Analyze complex problems and provide actionable insights - Automate repetitive data preparation, cleansing, and reporting tasks using AI-assisted pipelines and low-code/no-code tools - Leverage data visualization tools a...

Project Role :Data Engineer Project Role Description :Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills :Apache Airflow Good to have skills :NA Minimum 5 year(s) of experience is required Educational Qualification :15 years full time education Summary:As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating data pipelines, ensuring data quality, and implementing ETL processes...