683 Apache Airflow Jobs - Page 21

You are looking for a GCP Cloud Engineer for a position based in Pune. As a GCP Data Engineer, you will be responsible for designing, implementing, and optimizing data solutions on Google Cloud Platform. Your expertise in GCP services, solution design, and programming skills will be crucial for developing scalable and efficient cloud solutions. Your key responsibilities will include designing and implementing GCP-based data solutions following best practices, developing workflows and pipelines using Cloud Composer and Apache Airflow, building and managing data processing clusters using Dataproc, working with GCP services like Cloud Functions, Cloud Run, and Cloud Storage, and integrating mul...

As a Data Engineer at our company, you will be responsible for designing scalable and robust AI/ML systems in production, focusing on high-performance and cost-effective solutions. Your expertise in various technologies, including GCP services like BigQuery, Cloud Dataflow, Pub/Sub, Dataproc, and Cloud Storage, along with programming languages such as Python, Java/Scala, and SQL, will be crucial for the success of our projects. Additionally, your experience with data processing tools like Apache Beam, Apache Kafka, and Cloud Dataprep, as well as orchestration tools like Apache Airflow and Terraform, will play a significant role in implementing efficient data pipelines. Knowledge of security ...

You will be joining McDonald's Corporation, one of the world's largest employers with a presence in over 100 countries, at their corporate opportunity in Hyderabad. The global offices in Hyderabad serve as innovation and operations hubs, fostering McDonald's global talent pool and in-house expertise. In this role, you will play a pivotal part in developing impactful solutions for the business and customers worldwide, focusing on business, technology, analytics, and AI. As a Data Engineer at the G4 level, you will be responsible for creating scalable and efficient data solutions to support the Brand Marketing and Menu function, with a specific emphasis on the Menu Data product and associated ...

The ideal candidate should be highly interested and available immediately. Please submit your resume along with your total experience, current CTC, notice period, and current location details to Nitin.patil@ust.com. You will be responsible for designing, developing, and optimizing data pipelines and ETL workflows. Your work will involve collaborating with Apache Hadoop, Airflow, Kubernetes, and Containers to streamline data processing. Additionally, you will implement data analytics and mining techniques to derive valuable business insights. Managing cloud-based big data solutions on GCP and Azure will also be part of your job. Lastly, you will troubleshoot Hadoop log files and utilize multi...

You should have at least 5 years of experience working as a Data Engineer. Your expertise should include a strong background in Azure Cloud services and proficiency in tools such as Azure Databricks, PySpark, and Delta Lake. It is essential to have solid experience in Python and FastAPI for API development, as well as familiarity with Azure Functions for serverless API deployments. Experience in managing ETL pipelines using Apache Airflow is also required. Hands-on experience with databases like PostgreSQL and MongoDB is necessary. Strong SQL skills and the ability to work with large datasets are key for this role.,

As a skilled developer with 5-8 years of experience, you will be responsible for developing, updating, and maintaining applications to meet specified requirements, scale efficiently, and ensure high performance. Your role will involve analyzing project requirements, designing effective solutions within the broader product architecture, and deploying APIs and web services with reusable, testable, and efficient code. You will implement low-latency, scalable applications with optimized performance and create Docker files for containerization, deploying applications within a Kubernetes environment. Your ability to adapt quickly to a dynamic, start-up style environment, demonstrate strong problem...

You will be responsible for developing, deploying, monitoring, and maintaining ETL Jobs as well as all data engineering and pipeline activities. Your role will involve having a good understanding of DB activities and providing support in DB solutions. Additionally, you must possess proven expertise in SQL queries. Your key responsibilities will include designing and constructing various enterprise procedure constructs using any ETL tool, preferably PentahoDI. You will be expected to provide accurate work estimates, manage efforts across multiple lines of work, design and develop exception handling and data cleansing/standardization procedures, gather requirements from various stakeholders re...

As a Project Leader at BCN Labs, you will be an integral part of a Center of Excellence (CoE) within Bain & Company, working on delivering innovative data-driven solutions across various sectors and industries. Your role will involve collaborating with other CoEs and Practices at Bain to provide end-to-end analytical solutions that drive high-impact results for clients globally. Your primary responsibilities will include designing and implementing scalable data pipelines using modern data engineering tools, leading project teams in framing business problems and delivering strategic solutions, mentoring a team of engineers and analysts, and engaging with clients and stakeholders to communicat...

As a Data Engineer, you will be responsible for designing and building efficient data pipelines using Azure Databricks (PySpark). Your role will involve implementing business logic for data transformation and enrichment at scale and managing and optimizing Delta Lake storage solutions. You will also be involved in developing REST APIs using FastAPI to expose processed data and deploying APIs on Azure Functions for scalable and serverless data access. Additionally, you will develop and manage Airflow DAGs to orchestrate ETL processes, ingest and process data from various internal and external sources on a scheduled basis. Database management will be a key part of your responsibilities, includ...

As a Data Engineer (Snowflake and Python) Vice President in the Technology department, you will need to have 8-12 years of experience in designing and building highly resilient data platforms. Your strong knowledge of data engineering, architecture, and data modeling will be crucial for this role. You should possess strong analytical and problem-solving skills and have experience working with platforms like Snowflake, DBT, and Apache AirFlow. Experience in building applications on the cloud, specifically Azure, is required. Proficiency in programming skills such as Python, and optionally Java or Scala, along with a willingness to learn other languages is essential. Prior experience in workin...

As a Lead Data Engineer at Rearc, you will play a crucial role in establishing and maintaining technical excellence within our data engineering team. Your extensive experience in data architecture, ETL processes, and data modeling will be key in optimizing data workflows for efficiency, scalability, and reliability. Collaborating closely with cross-functional teams, you will design and implement robust data solutions that align with business objectives and adhere to best practices in data management. Building strong partnerships with technical teams and stakeholders is essential as you drive data-driven initiatives and ensure their successful implementation. With over 10 years of experience ...

As the Lead Python Software Engineer at Citi, you will have the exciting opportunity to build the foundational backend systems for "Citi Assist", a Generative AI assistant that is set to reach every Citi employee globally. Joining at the ground floor, you will play a pivotal role in developing a high-visibility platform that establishes the standard for the safe and scalable adoption of this technology by a global bank. In this hands-on role, your primary responsibility will be to build, code, and deliver high-quality, tested Python backend features efficiently. Taking ownership of the features you develop, you will drive them from conception to completion, ensuring security and compliance a...

As an MLOps Engineer at Barclays, you will be responsible for operationalizing cutting-edge machine learning and generative AI solutions. Your main focus will be on ensuring scalable, secure, and efficient deployment across the infrastructure. Collaborating closely with data scientists, ML engineers, and business stakeholders, you will build and maintain robust MLOps pipelines. These pipelines will enable rapid experimentation and reliable production implementation of AI models, including LLMs and real-time analytics systems. To excel in this role, you should have strong programming skills in Python and hands-on experience with ML libraries such as scikit-learn, TensorFlow, and PyTorch. Addi...

As a member of the Dun & Bradstreet team, you will play a crucial role in unlocking the power of data through analytics to create a better tomorrow. Our global community of over 6,000 team members is dedicated to accelerating creativity, innovation, and growth in order to help clients turn uncertainty into confidence, risk into opportunity, and potential into prosperity. We welcome bold and diverse thinkers who are passionate about making a positive impact. You will be responsible for designing and developing data pipelines within our Big Data ecosystem using technologies such as Apache Spark and Apache Airflow. Your role will involve architecting, building, and deploying scalable and effici...

You are an experienced Senior Data Engineer with a strong background in Databricks, Apache Airflow, SQL, and Python. Your main responsibility will involve designing and optimizing data pipelines, ETL workflows, and cloud-based data solutions. Your expertise in big data processing, cloud-based analytics, and automation will be crucial in developing efficient and scalable data engineering solutions. You will be responsible for tasks such as designing, developing, and optimizing data pipelines using Databricks and Apache Airflow. Implementing PySpark-based transformations and processing in Databricks to handle large-scale data, and maintaining SQL-based data pipelines with a focus on performanc...

You will work as a Machine Learning Ops Engineer at Netsmart, contributing to the design, implementation, and maintenance of ML pipelines for model training, validation, and deployment. Your role involves automating model deployment processes using CI/CD pipelines and containerization technologies. You will be responsible for monitoring model performance, data drift, and system health in production environments, collaborating with data scientists to operationalize machine learning models and algorithms. Additionally, you will implement version control for models, datasets, and ML experiments using MLOps tools, optimize ML infrastructure for scalability, reliability, and cost-effectiveness, a...

As a Senior/Lead Data Engineer at Srijan Technologies PVT LTD, you will play a crucial role in designing, developing, and maintaining data pipelines for our clients. Your expertise in AWS Glue, Apache Airflow, Kafka, SQL, Python, and DataOps tools will be key in ensuring the efficient and reliable flow of data across the organization. You will collaborate with cross-functional teams to understand data requirements, implement DataOps practices, and optimize data pipeline performance. Monitoring data pipeline performance, troubleshooting data-related issues, and staying up-to-date with the latest industry trends in data engineering and DataOps will also be part of your responsibilities. Key Re...

One of our prestigious clients, a TOP MNC Giant with a global presence, is currently seeking a Lead Enterprise Architect to join their team in Pune, Mumbai, or Bangalore. **Qualifications and Certifications:** **Education:** - Bachelors or masters degree in Computer Science, Information Technology, Engineering, or a related field. **Experience:** - A minimum of 10+ years of experience in data engineering, with at least 4 years of hands-on experience with GCP cloud platforms. - Proven track record in designing and implementing data workflows using GCP services such as BigQuery, Dataform Cloud Dataflow, Cloud Pub/Sub, and Cloud Composer. **Certifications:** - Google Cloud Professional Data Eng...

You have over 8 years of experience and are located in Balewadi, Pune. You possess a strong understanding of Data Architecture and have led data-driven projects. Your expertise includes knowledge of Data Modelling paradigms like Kimball, Inmon, Data Marts, Data Vault, Medallion, etc. Experience with Cloud Based data strategies, particularly AWS, is preferred. Designing data pipelines for ETL with expert knowledge on ingestion, transformation, and data quality is a must, along with hands-on experience in SQL. In-depth understanding of PostGreSQL development, query optimization, and designing indexes is a key requirement. Proficiency in Postgres PL/SQL for complex warehouse workflows is necess...

As a Python Machine Learning Engineer at our organization, you will be responsible for designing, developing, and deploying scalable recommender systems using various machine learning algorithms. Your primary focus will be on leveraging machine learning techniques to personalize user experiences, enhance engagement, and drive business outcomes. You will collaborate closely with cross-functional teams to understand business requirements and translate them into actionable machine learning solutions. Conducting thorough exploratory data analysis to identify relevant features and patterns in large-scale datasets will be a crucial part of your role. Additionally, you will implement and optimize m...

The ideal candidate for this role should have proficiency in Python along with practical experience in working with Databricks (PySpark). It is mandatory for the candidate to possess these skills in order to effectively carry out the responsibilities associated with the position. In addition to the mandatory requirements, it would be advantageous for the candidate to have hands-on experience with Apache Airflow. Furthermore, a working knowledge of databases such as PostgreSQL and MongoDB would be beneficial for the role. Basic experience with cloud technologies like Azure, AWS, and Google Cloud Platform is also considered to be a positive attribute. Overall, the candidate should be well-vers...

As a technically proficient Product Owner with expertise in data and analytics platforms, you will play a crucial role in leading the development of scalable and insight-driven data products. Your responsibilities will involve collaborating closely with data engineers, architects, analysts, and business stakeholders to convert raw data into impactful tools and solutions that drive business intelligence, advanced analytics, and operational efficiency. You will be responsible for defining and managing the product roadmap for data-centric platforms, services, and analytics tools. This will include translating business and analytical goals into detailed technical product requirements, user stori...

You are an experienced Data Engineer with expertise in PySpark, Snowflake, and AWS. Your role involves designing, developing, and optimizing data pipelines and workflows in a cloud-based environment, utilizing AWS services, PySpark, and Snowflake for data processing and analytics. Your key responsibilities include designing and implementing scalable ETL pipelines using PySpark on AWS, developing and optimizing data workflows for Snowflake integration, managing and configuring AWS services like S3, Lambda, Glue, EMR, and Redshift, collaborating with data analysts and business teams to understand requirements and deliver solutions, ensuring data security and compliance with best practices in A...

You will be part of a team responsible for developing a next-generation Data Analytics Engine that converts raw market and historical data into actionable insights for the electronics supply chain industry. This platform processes high-volume data from suppliers, parts, and trends to provide real-time insights and ML-driven applications. We are seeking an experienced Lead or Staff Data Engineer to assist in shaping and expanding our core data infrastructure. The ideal candidate should have a strong background in designing and implementing scalable ETL pipelines and real-time data systems in AWS and open-source environments such as Airflow, Spark, and Kafka. This role involves taking technica...

As a Lead Data Engineer with 7-12 years of experience, you will be an integral part of our team, contributing significantly to the design, development, and maintenance of our data infrastructure. Your primary responsibilities will revolve around creating and managing robust data architectures, ETL processes, data warehouses, and utilizing big data and cloud technologies to support our business intelligence and analytics needs. You will lead the design and implementation of data architectures that facilitate data warehousing, integration, and analytics platforms. Developing and optimizing ETL pipelines will be a key aspect of your role, ensuring efficient processing of large datasets and impl...