144 Apache Airflow Jobs

You are a talented and driven Machine Learning Engineer with 2-5 years of experience, looking to join a dynamic team in Chennai. Your expertise lies in machine learning principles and hands-on experience in building, deploying, and managing ML models in production environments. In this role, you will focus on MLOps practices and orchestration to ensure robust, scalable, and automated ML pipelines. Your responsibilities will include designing, developing, and implementing end-to-end MLOps pipelines for deploying, monitoring, and managing machine learning models in production. You will use orchestration tools such as Apache Airflow, Kubeflow, AWS Step Functions, or Azure Data Factory to automate ML workflows. Implementing CI/CD practices for ML code, models, and infrastructure will be crucial for ensuring rapid and reliable releases. You will also establish monitoring and alerting systems for deployed ML models, optimize performance, troubleshoot and debug issues across the ML lifecycle, and create and maintain technical documentation. To qualify for this role, you should have a Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related quantitative field, along with 2-5 years of professional experience as a Machine Learning Engineer or MLOps Engineer. Your skills should include proficiency in Python and its ML ecosystem, hands-on experience with major cloud platforms and their ML/MLOps services, knowledge of orchestration tools, containerization technologies, CI/CD pipelines, and database systems. Strong problem-solving, analytical, and communication skills are essential for collaborating effectively with Data Scientists, Data Engineers, and Software Developers in an Agile environment.,

At Rhythm, our values serve as the cornerstone of our organization. We are deeply committed to customer success, fostering innovation, and nurturing our employees. These values shape our decisions, actions, and interactions, ensuring that we consistently create a positive impact on the world around us. Rhythm Innovations is currently looking for a skilled and enthusiastic Machine Learning (ML) Developer to conceptualize, create, and implement machine learning models that enhance our supply chain risk management and other cutting-edge solutions. As an ML Developer, you will collaborate closely with our AI Architect and diverse teams to construct intelligent systems that tackle intricate business challenges and further our goal of providing unparalleled customer satisfaction. Key Responsibilities Model Development: Devise, execute, and train machine learning models utilizing cutting-edge algorithms and frameworks like TensorFlow, PyTorch, and scikit-learn. Data Preparation: Process, refine, and convert extensive datasets for the training and assessment of ML models. Feature Engineering: Identify and engineer pertinent features to enhance model performance and precision. Algorithm Optimization: Explore and implement advanced algorithms to cater to specific use cases such as classification, regression, clustering, and anomaly detection. Integration: Coordinate with software developers to integrate ML models into operational systems and guarantee smooth functionality. Performance Evaluation: Assess model performance using suitable metrics and consistently refine for accuracy, efficacy, and scalability. MLOps: Aid in establishing and overseeing CI/CD pipelines for model deployment and monitoring in production environments. Research and Development: Stay abreast of the latest breakthroughs in Gen AI AI/ML technologies and propose inventive solutions. Collaboration: Engage closely with data engineers, product teams, and stakeholders to grasp requirements and deliver customized ML solutions. Requirements Educational Background: Bachelor's in Engineering in Computer Science, Data Science, Artificial Intelligence, or a related field. Experience: 3 to 6 years of practical experience in developing and deploying machine learning models. Technical Skills Proficiency in Python and ML libraries/frameworks (e.g., scikit-learn, TensorFlow, PyTorch). Experience with data manipulation tools like Pandas, NumPy, and visualization libraries such as Matplotlib or Seaborn. Familiarity with big data frameworks (Hadoop, Spark) is advantageous. Knowledge of SQL/NoSQL databases and data pipeline tools (e.g., Apache Airflow). Hands-on experience with cloud platforms (AWS, Azure, Google Cloud) and their Gen AI AI/ML services. Thorough understanding of supervised and unsupervised learning, deep learning, and reinforcement learning. Exposure to MLOps practices and model deployment pipelines. Soft Skills Strong problem-solving and analytical abilities. Effective communication and teamwork skills. Capability to thrive in a dynamic, collaborative environment.,

DifferentByte Technologies Pvt Ltd, a GENAI consultancy firm located in Cochin, Kerala, is in immediate need of a talented Python Developer/Backend Developer to join their team. The primary focus of this position is to contribute to the development of an AI-based high-frequency trading platform. This role presents a unique opportunity for individuals looking to establish a rewarding career in a technology-oriented trading organization. As a Python Developer at DifferentByte Technologies Pvt Ltd, your responsibilities will include writing effective and scalable code, enhancing back-end components to optimize responsiveness and performance, integrating user-facing features into applications, as well as testing and debugging programs. You will also be tasked with improving the functionality of existing systems, implementing security measures, evaluating and prioritizing feature requests, and collaborating with internal teams to understand user needs and offer technical solutions. The ideal candidate should possess a minimum of 2 years of demonstrable experience as a Python Developer within a product-based company. Proficiency in Python frameworks such as Fast API and Flask, along with knowledge of PostgreSQL and familiarity with Supabase, are essential requirements for this role. Additionally, expertise in queuing systems like Apache Kafka and Airflow, as well as strong problem-solving skills, are highly valued. Candidates with experience in artificial intelligence, machine learning, deep learning, and technologies like TensorFlow and PyTorch will be preferred. A degree in B.Tech/B.E. in Computer Science, Engineering, or a related field is a prerequisite. This position is open to candidates currently residing in Kerala or those willing to relocate to Cochin. Apart from the technical qualifications, DifferentByte Technologies Pvt Ltd offers a collaborative and informal work environment with a casual dress code and a flat structure. Joining the highly driven and Agile team at DifferentByte presents an excellent opportunity for career advancement, competitive salary, and a comprehensive benefits package. If you are looking to be a part of an innovative team that values code quality, testing, and making a significant impact on the future of AI-based trading platforms, apply now by contacting hr@differentbyte.in.,

We empower our people to stay resilient and relevant in a constantly changing world. We are looking for individuals who are always seeking creative ways to grow and learn, individuals who aspire to make a real impact, both now and in the future. If this resonates with you, then you would be a valuable addition to our dynamic international team. We are currently seeking a Senior Software Engineer - Data Engineer (AI Solutions). In this role, you will have the opportunity to: - Design, build, and maintain data pipelines to cater to the requirements of various stakeholders, including software developers, data scientists, analysts, and business teams. - Ensure that the data pipelines are modular, resilient, and optimized for performance and low maintenance. - Collaborate with AI/ML teams to support training, inference, and monitoring needs through structured data delivery. - Implement ETL/ELT workflows for structured, semi-structured, and unstructured data using cloud-native tools. - Work with large-scale data lakes, streaming platforms, and batch processing systems to ingest and transform data. - Establish robust data validation, logging, and monitoring strategies to uphold data quality and lineage. - Optimize data infrastructure for scalability, cost-efficiency, and observability in cloud-based environments. - Ensure adherence to governance policies and data access controls across projects. To excel in this role, you should possess the following qualifications and skills: - A Bachelor's degree in Computer Science, Information Systems, or a related field. - Minimum of 4 years of experience in designing and deploying scalable data pipelines in cloud environments. - Proficiency in Python, SQL, and data manipulation tools and frameworks such as Apache Airflow, Spark, dbt, and Pandas. - Practical experience with data lakes, data warehouses (e.g., Redshift, Snowflake, BigQuery), and streaming platforms (e.g., Kafka, Kinesis). - Strong understanding of data modeling, schema design, and data transformation patterns. - Experience with AWS (Glue, S3, Redshift, Sagemaker) or Azure (Data Factory, Azure ML Studio, Azure Storage). - Familiarity with CI/CD for data pipelines and infrastructure-as-code (e.g., Terraform, CloudFormation). - Exposure to building data solutions that support AI/ML pipelines, including feature stores and real-time data ingestion. - Understanding of observability, data versioning, and pipeline testing tools. - Previous engagement with diverse stakeholders, data requirement gathering, and support for iterative development cycles. - Background or familiarity with the Power, Energy, or Electrification sector is advantageous. - Knowledge of security best practices and data compliance policies for enterprise-grade systems. This position is based in Bangalore, offering you the opportunity to collaborate with teams that impact entire cities, countries, and shape the future. Siemens is a global organization comprising over 312,000 individuals across more than 200 countries. We are committed to equality and encourage applications from diverse backgrounds that mirror the communities we serve. Employment decisions at Siemens are made based on qualifications, merit, and business requirements. Join us with your curiosity and creativity to help shape a better tomorrow. Learn more about Siemens careers at: www.siemens.com/careers Discover the Digital world of Siemens here: www.siemens.com/careers/digitalminds,

Golden Eagle IT Technologies Pvt. Ltd. is looking for a skilled Data Engineer with 2 to 4 years of experience to join the team in Indore. The ideal candidate should have a solid background in data engineering, big data technologies, and cloud platforms. As a Data Engineer, you will be responsible for designing, building, and maintaining efficient, scalable, and reliable data pipelines. You will be expected to develop and maintain ETL pipelines using tools like Apache Airflow, Spark, and Hadoop. Additionally, you will design and implement data solutions on AWS, leveraging services such as DynamoDB, Athena, Glue Data Catalog, and SageMaker. Working with messaging systems like Kafka for managing data streaming and real-time data processing will also be part of your responsibilities. Proficiency in Python and Scala for data processing, transformation, and automation is essential. Ensuring data quality and integrity across multiple sources and formats will be a key aspect of your role. Collaboration with data scientists, analysts, and other stakeholders to understand data needs and deliver solutions is crucial. Optimizing and tuning data systems for performance and scalability, as well as implementing best practices for data security and compliance, are also expected. Preferred skills include experience with infrastructure as code tools like Pulumi, familiarity with GraphQL for API development, and exposure to machine learning and data science workflows, particularly using SageMaker. Qualifications for this position include a Bachelor's degree in Computer Science, Information Technology, or a related field, along with 2-4 years of experience in data engineering or a similar role. Proficiency in AWS cloud services and big data technologies, strong programming skills in Python and Scala, knowledge of data warehousing concepts and tools, as well as excellent problem-solving and communication skills are required.,

We are looking for a Data Engineer to join our data team. You will be responsible for managing our master data set, developing reports, and troubleshooting data issues. To excel in this role, attention to detail, experience as a data analyst, and a deep understanding of popular data analysis tools and databases are essential. Your responsibilities include: - Building, maintaining, and managing data pipelines for efficient data flow between systems. - Collaborating with stakeholders to design and manage customized data pipelines. - Testing various ETL (Extract, Transform, Load) tools for data ingestion and processing. - Assisting in scaling the data infrastructure to meet the organization's growing data demands. - Monitoring data pipeline performance and troubleshooting data issues. - Documenting pipeline architectures and workflows for future reference and scaling. - Evaluating data formats, sources, and transformation techniques. - Working closely with data scientists to ensure data availability and reliability for analytics. We require the following skill sets/experience: - Proficiency in Python, PySpark, and Big Data concepts such as Data Lakes and Data Warehouses. - Strong background in SQL. - Familiarity with cloud computing platforms like AWS, Azure, or Google Cloud. - Basic knowledge of containerization technologies like Docker. - Exposure to data orchestration tools like Apache Airflow or Luigi. Pedigree: - Bachelor's degree in Computer Science, Electrical Engineering, or IT.,

Job Description 15 to 18 years. with at-least 3 to 4 years as expertise in ETL, data engineering and Cloud Technologies, with a proven ability to orchestrate cutting-edge technology to connect various applications within the cloud environment in a large development project. Primary Technical Skills: ETL, Apache Spark, AWS EMR, EKS, Serverless, Data Engineering, Distributed Computing, Data Lineage, Apache Airflow, Java 17+, Springboot/ Quarkus, Hibernate ORM, REST, Postgres or any RDBMS, Microservices, Cloud-native development, Secondary Technical Skills Devops : Docker, Kubernetes, CI / CD stack Jenkins or Gitlab CI, Maven, GIT, SonarQube, Nexus, AWS, expertise in at least one Data Engineering tools (e.g. Informatica, Data stage) Apache Airflow, Redis, No-SQL (any Document DB), Kafka / Rabbit MQ, OAUTH2, ARGO, Swagger, OAS Experience / Application of skills Experience in ETL implementation using Cloud technologies, Distributed computing and Big data processing. Orchestrate the integration of Cloud-native principles, Kubernetes, Micro profile specs, Spark framework. Hands-on Java Lead. Strong in OOPs concepts, Java design patterns, Reactive programming, writing High level solutions & Clean architecture. Very strong advocate of coding best practices (SOLID, DRY, Clean Code, Exception handling, TDD, Unit testing, Integration testing). Have implemented common framework for an application/platform (like Exception Library, Security Authentication/Authorization, Auditing, Idempotency, Connectors etc) Experience in implementing HLD, Microservices architecture, design patterns like Resiliency, Service Orchestration, DB per service, CQRS etc. Preferred personal qualities Proactive, Self-starter , Willing to learn new technology. Develop rapid prototypes/PoC/MVP in data integration within cloud environments. Working with team members, mentoring, and guiding them in their career track Excellent problem-solving skills and ability to work in a fast-paced environment. Stay updated with the latest advancements in cloud and data technologies, as well as best practices. Strong leadership and communication skills. Role Engineering Lead Data Engineering Architect Shift : General shift Location : Chennai , Hyderabad and Pune. Those who are interested can drop their resumes at Krishna.Kumaravel@ltimindtree.com

As a Data Engineer specializing in ETL, you should possess a minimum of 7 to 8 years of relevant experience in the field. This position is open across Pan India, and immediate joiners are highly preferred. You will be expected to demonstrate expertise in a range of mandatory skills, including ETL Developer, Synapse, Pyspark, ADF, SSIS, Databricks, SQL, Apache Airflow, and proficiency in Azure & AWS. It is important to note that proficiency in all the mentioned skills is a prerequisite for this role. The selection process for this position involves a total of three rounds - L1 with the External Panel, L2 with the Internal Panel, and L3 with the Client Round. Your responsibilities will include working as an ETL Developer for at least 7+ years, demonstrating proficiency in Pyspark for 5+ years, SSIS for 3 to 4+ years, Databricks for 4 to 4+ years, SQL for 6+ years, Apache Airflow for 4+ years, and experience in Azure and AWS for 3 to 4 years. Additionally, familiarity with Synapse for 3 to 4 years is required to excel in this role.,

We are searching for a Senior Data Engineer with significant experience in developing ETL processes utilizing PySpark Notebooks and Microsoft Fabric, as well as supporting existing legacy SQL Server environments. The perfect candidate will have a solid foundation in Spark-based development, showcase advanced SQL skills, and feel at ease working autonomously, collaboratively within a team, or guiding other developers when necessary, all while possessing excellent communication abilities. The ideal candidate will also demonstrate expertise with Azure Data Services, such as Azure Data Factory, Azure Synapse, or similar tools, familiarity with creating DAG's, implementing activities, and running Apache Airflow, and knowledge of DevOps practices, CI/CD pipelines, and Azure DevOps. Key Responsibilities: - Design, develop, and manage ETL Notebook orchestration pipelines utilizing PySpark and Microsoft Fabric. - Collaborate with data scientists, analysts, and stakeholders to grasp data requirements and provide effective data solutions. - Migrate and integrate data from legacy SQL Server environments into modern data platforms. - Optimize data pipelines and workflows for scalability, efficiency, and reliability. - Provide technical leadership and mentorship to junior developers and team members. - Troubleshoot and resolve complex data engineering issues related to performance, data quality, and system scalability. - Develop, maintain, and uphold data engineering best practices, coding standards, and documentation. - Conduct code reviews and offer constructive feedback to enhance team productivity and code quality. - Support data-driven decision-making processes by ensuring data integrity, availability, and consistency across different platforms. Qualifications: - Bachelors or Masters degree in Computer Science, Data Science, Engineering, or a related field. - 10+ years of experience in data engineering, focusing on ETL development using PySpark or other Spark-based tools. - Proficiency in SQL with extensive experience in complex queries, performance tuning, and data modeling. - Experience with Microsoft Fabric or similar cloud-based data integration platforms is advantageous. - Strong understanding of data warehousing concepts, ETL frameworks, and big data processing. - Familiarity with other data processing technologies (e.g., Hadoop, Hive, Kafka) is a plus. - Experience dealing with both structured and unstructured data sources. - Excellent problem-solving skills and the ability to troubleshoot complex data engineering issues. - Experience with Azure Data Services, including Azure Data Factory, Azure Synapse, or similar tools. - Experience of creating DAG's, implementing activities, and running Apache Airflow. - Familiarity with DevOps practices, CI/CD pipelines, and Azure DevOps. In conclusion, Aspire Systems is a global technology services firm that acts as a trusted technology partner for over 275 clients worldwide. Aspire collaborates with leading enterprises in Banking, Insurance, Retail, and ISVs to help them leverage technology for business transformation in the current digital era. The company's dedication to Attention. Always. reflects its commitment to providing care and attention to both its customers and employees. With over 4900 employees globally and a CMMI Level 3 certification, Aspire Systems operates in North America, LATAM, Europe, Middle East, and Asia Pacific. Aspire Systems has been consistently recognized as one of the Top 100 Best Companies to Work For by the Great Place to Work Institute for the 12th consecutive time. For more information about Aspire Systems, please visit https://www.aspiresys.com/.,

We are looking for a skilled and motivated Data Engineer with at least 4 years of experience in GCP, Teradata, and Data Warehousing. The ideal candidate should have hands-on expertise in developing robust data engineering solutions on Google Cloud Platform (GCP) and working experience with Teradata. You must be proficient in designing and automating scalable data pipelines and possess excellent leadership, communication, and collaboration skills. Your responsibilities will include analyzing source systems, profiling data, and resolving data quality issues. You will be required to gather and comprehend business requirements for data transformation, design, develop, test, and deploy ETL/data pipelines using GCP services and Airflow. Additionally, writing complex SQL queries for data extraction, formatting, and analysis, creating and maintaining Source to Target Mapping, and designing documentation will be part of your role. You will also need to build metadata-driven frameworks for scalable data pipelines, perform unit testing, and document results, utilize DevOps tools for version control and deployment, provide production support, enhancements, and bug fixes, troubleshoot issues, and support ad-hoc business requests. Collaboration with stakeholders to resolve EDW incidents, manage expectations, apply ITIL concepts for incident and problem management, perform data cleaning, transformation, and validation, and stay updated on GCP advancements and industry best practices are also key responsibilities. Requirements: - Minimum 4 years of experience in ETL and Data Warehousing - Hands-on experience with GCP services such as BigQuery, Dataflow, Cloud Storage, etc. - Experience in Apache Airflow for workflow orchestration - Experience in automating ETL solutions - Experience in executing at least 2 GCP Cloud Data Warehousing projects - Exposure to Agile/SAFe methodologies in at least 2 projects - Mid-level proficiency in PySpark and Teradata - Strong SQL skills and experience working with semi-structured data formats like JSON, Parquet, XML - Experience with DevOps tools like GitHub, Jenkins, or similar - Deep understanding of Data Warehousing concepts, data profiling, quality, and mapping Preferred Qualifications: - B.Tech/B.E. in Computer Science or a related field - Google Cloud Professional Data Engineer Certification - Strong leadership and communication skills ,

You have the opportunity to apply for the position of Senior ETL and Feature Engineer at PrivaSapien, based in Bangalore. PrivaSapien is at the forefront of Privacy Enhancing & Responsible AI Technologies, where you will play a crucial role in setting up the big data ecosystem for the world's first privacy red teaming and blue teaming platform. As an individual contributor, you will work on cutting-edge privacy platform requirements with clients globally, spanning across various industry verticals. Joining as one of the early employees, you will receive a significant ESOP option and collaborate with brilliant minds from prestigious institutions such as IISc and IIMs. Your responsibilities will include developing and maintaining ETL pipelines for processing large-scale datasets, creating a Python connector for ETL applications, and demonstrating proficiency in AWS Glue. You will be involved in ETL pipeline development for AI/ML workloads, orchestrating scaling, and resource management. Additionally, you will work on managing unstructured data tasks, optimizing query performance in SQL databases, and integrating multiple databases into the ETL pipeline within a multi-cloud environment. To be eligible for this role, you should have a minimum of 4 years of hands-on experience in setting up ETL and feature engineering pipelines on cloud or big data ecosystems. Proficiency in Apache Spark, pyspark, Apache Airflow, and AWS Glue is essential, along with expertise in at least one ETL tool. Strong programming skills in Python, familiarity with data manipulation libraries, and experience in handling various data types are required. Furthermore, you should possess knowledge in SQL databases, networking, security, and cloud platforms. The interview process will consist of a technical round with the Director, an assessment, an assessment review round with the Senior Backend person, and an HR round. To apply for this opportunity, you need to register or login on the portal, fill out the application form, clear the video screening, and click on "Apply" to be shortlisted. Your profile will then be shared with the client for the interview round upon selection. At Uplers, our aim is to simplify and expedite the hiring process, assisting talents in finding and applying for relevant contractual onsite opportunities. We provide support for any challenges faced during the engagement and assign a dedicated Talent Success Coach to guide you throughout the process. If you are prepared for a new challenge, a conducive work environment, and an opportunity to elevate your career, seize this chance today. We look forward to welcoming you aboard!,

As a Data Engineer at our company, your primary responsibility will be the development and maintenance of scalable and efficient data pipelines and ETL processes using Python and related technologies. You will play a crucial role in optimizing the performance of these pipelines and queries to handle large volumes of data and improve processing times. Collaboration is key in this role, as you will closely work with our team of data scientists and engineers at Matrix Space. To excel in this position, you should have 2-5 years of experience in data engineering or a related field with a strong focus on Python. Proficiency in Python programming is a must, including knowledge of libraries such as Pandas, NumPy, and SQL Alchemy. Additionally, hands-on experience with data engineering tools and frameworks like Apache Airflow, Luigi, or similar is highly desirable. A solid grasp of SQL and experience with relational databases such as PostgreSQL and MySQL will be beneficial. In addition to technical skills, we value certain soft skills in our team members. Problem-solving abilities, the capacity to work both independently and collaboratively, and effective communication skills are essential. You should be able to articulate technical concepts to non-technical stakeholders and demonstrate a proven track record of completing tasks efficiently. If you are an immediate joiner and can commence within a week, we encourage you to apply for this position. Join our team and be part of an exciting journey in data engineering where your skills and expertise will be valued and put to good use.,

You are a Sr. Data Engineer with a strong background in building ELT pipelines and expertise in modern data engineering practices. You are experienced with Databricks and DBT, proficient in SQL and Python, and have a solid understanding of data warehousing methodologies such as Kimball or Data Vault. You are comfortable working with DevOps tools, particularly within AWS, Databricks, and GitLab. Your role involves collaborating with cross-functional teams to design, develop, and maintain scalable data infrastructure and pipelines using Databricks and DBT. Your responsibilities include designing, building, and maintaining scalable ELT pipelines for processing and transforming large datasets efficiently in Databricks. You will implement Kimball data warehousing methodologies or other multi-dimensional modeling approaches using DBT. Leveraging AWS, Databricks, and GitLab, you will implement CI/CD practices for data engineering workflows. Additionally, you will optimize SQL queries and database performance, monitor and fine-tune data pipelines and queries, and ensure compliance with data security, privacy, and governance standards. Key qualifications for this role include 6+ years of data engineering experience, hands-on experience with Databricks and DBT, proficiency in SQL and Python, experience with Kimball data warehousing or Data Vault methodologies, familiarity with DevOps tools and practices, strong problem-solving skills, and the ability to work in a fast-paced, agile environment. Preferred qualifications include experience with Apache Spark for large-scale data processing, familiarity with CI/CD pipelines for data engineering workflows, understanding of orchestration tools like Apache Airflow, and certifications in AWS, Databricks, or DBT. In return, you will receive benefits such as medical insurance for employees, spouse, and children, accidental life insurance, provident fund, paid vacation time, paid holidays, employee referral bonuses, reimbursement for high-speed internet at home, one-month free stay for employees moving from other cities, tax-free benefits, and other bonuses as determined by management.,

As a Data Engineer at Synoptek, you will be responsible for designing, developing, and maintaining robust and scalable data pipelines on the Google Cloud Platform (GCP). You will leverage your hands-on experience with GCP services such as BigQuery, Jitterbit, Cloud Dataflow, Cloud Pub/Sub, and Cloud Storage to build efficient data processing solutions. Collaborating with cross-functional teams, you will translate their data needs into technical requirements, ensuring data quality, integrity, and security throughout the data lifecycle. Your role will involve developing and optimizing ETL/ELT processes to extract, transform, and load data from various sources into data warehouses and data lakes. Additionally, you will build and maintain data models and schemas to support business intelligence and analytics, while troubleshooting data quality issues and performance bottlenecks. To excel in this position, you should have a Bachelor's degree in Computer Science, Engineering, or a related field, along with 3 to 4 years of experience as a Data Engineer focusing on GCP. Proficiency in Python, SQL, and BigQuery is essential, as well as hands-on experience with data ingestion, transformation, and loading tools like Jitterbit and Apache Beam. A strong understanding of data warehousing and data lake concepts, coupled with experience in data modeling and schema design, will be beneficial. The ideal candidate will exhibit excellent problem-solving and analytical skills, working both independently and collaboratively with internal and external teams. Familiarity with acquiring and managing data from various sources, as well as the ability to identify trends in complex datasets and propose business solutions, are key attributes for success in this role. At Synoptek, we value employees who embody our core DNA behaviors, including clarity, integrity, innovation, accountability, and a results-focused mindset. We encourage continuous learning, adaptation, and growth in a fast-paced environment, promoting a culture of teamwork, flexibility, respect, and collaboration. If you have a passion for data engineering, a drive for excellence, and a commitment to delivering impactful results, we invite you to join our dynamic team at Synoptek. Work hard, play hard, and let's achieve superior outcomes together.,

As a Data Engineer, you will be responsible for designing and building efficient data pipelines using Azure Databricks (PySpark). You will implement business logic for data transformation and enrichment at scale, as well as manage and optimize Delta Lake storage solutions. Additionally, you will develop REST APIs using FastAPI to expose processed data and deploy them on Azure Functions for scalable and serverless data access. Your role will also involve developing and managing Airflow DAGs to orchestrate ETL processes, ingesting and processing data from various internal and external sources on a scheduled basis. You will handle data storage and access using PostgreSQL and MongoDB, writing optimized SQL queries to support downstream applications and analytics. Collaboration is key in this role, as you will work cross-functionally with teams to deliver reliable, high-performance data solutions. It is essential to follow best practices in code quality, version control, and documentation to ensure the success of projects. To excel in this position, you should have at least 5 years of hands-on experience as a Data Engineer and strong expertise in Azure Cloud services. Proficiency in Azure Databricks, PySpark, Delta Lake, Python, and FastAPI for API development is required. Experience with Azure Functions for serverless API deployments, managing ETL pipelines using Apache Airflow, and hands-on experience with PostgreSQL and MongoDB are also essential. Strong SQL skills and experience in handling large datasets will be beneficial for this role.,

As a Senior Developer specializing in SnapLogic and Apache Airflow, you will be responsible for designing, developing, and maintaining enterprise-level data integration solutions. Your expertise in ETL development, workflow orchestration, and cloud technologies will be crucial for automating data workflows, optimizing performance, and ensuring the reliability and scalability of data systems. Your key responsibilities will include designing, developing, and managing ETL pipelines using SnapLogic to ensure efficient data transformation and integration across various systems and applications. You will leverage Apache Airflow for workflow automation, job scheduling, and task dependencies to ensure optimized execution and monitoring. Collaboration with cross-functional teams such as Data Engineering, DevOps, and Data Science will be essential to understand data requirements and deliver effective solutions. In this role, you will be involved in designing and implementing data pipeline architectures to support large-scale data processing in cloud environments like AWS, Azure, and GCP. Developing reusable SnapLogic pipelines, integrating with third-party applications and data sources, optimizing pipeline performance, and providing guidance to junior developers will be part of your responsibilities. Additionally, troubleshooting pipeline failures, implementing automated testing, continuous integration (CI), and continuous delivery (CD) practices for data pipelines will be crucial for maintaining high data quality and minimal downtime. The required skills and experience for this role include at least 6 years of hands-on experience in data engineering with a focus on SnapLogic and Apache Airflow. Proficiency in SnapLogic Designer, SnapLogic cloud environment, and Apache Airflow for building data integrations and ETL pipelines is essential. You should have a strong understanding of ETL concepts, data integration, cloud platforms like AWS, Azure, or Google Cloud, data storage systems such as S3, Azure Blob, and Google Cloud Storage, as well as experience with SQL, relational databases, NoSQL databases, REST APIs, and CI/CD pipelines. Your problem-solving skills, ability to work in an Agile development environment, and strong communication and collaboration skills will be valuable assets in this role. By staying current with new SnapLogic features, Airflow upgrades, and industry best practices, you will contribute to the continuous improvement of data integration solutions. Join our team at Virtusa, where teamwork, quality of life, and professional development are values we embody. Be part of a global team that cares about your growth and provides exciting projects, opportunities, and exposure to state-of-the-art technologies throughout your career with us. At Virtusa, great minds come together to nurture new ideas and foster excellence in a dynamic environment.,

The Digital Success Engineering team at our organization is comprised of a diverse group of distributed software engineers, architects, and engineering managers who collaborate to develop Unified Experiences for our Trailblazers. As the pioneers of our products, we leverage Salesforce technology to create an intuitive and expert self-service platform that drives Trailblazer Success. As a Marketing Cloud Engineer (SMTS) in our Customer Engagement Engineering team, you will play a crucial role in providing technical solutions and support to our customers and business users. We are looking for someone with a strong background in the Salesforce Marketing Cloud platform, exceptional technical skills, and the ability to effectively communicate with stakeholders. The ideal candidate should excel in troubleshooting and coordinating operational issues within our Customer Engagement Ecosystem, working closely with various cross-functional teams to develop solutions and workarounds. Your responsibilities will include collaborating with business users, engineering teams, operations teams, and vendor teams to enhance existing and new business processes and systems functionality. You will be responsible for analyzing problems, providing root cause analysis, resolving issues, and planning long-term support strategies. Key Responsibilities: - Adapting to the Digital Success Engineering team structures and culture - Conducting technical requirements gathering, review, and architecture sessions - Translating business requirements into data architecture, data pipelines, and Salesforce Marketing Cloud software design - Providing guidance and thought leadership in resolving complex problems - Designing and implementing robust, scalable solutions for Salesforce Marketing Cloud projects - Triaging, troubleshooting, and ensuring timely issue resolution - Developing and maintaining technical expertise in assigned areas of product functionality - Engaging in continuous feedback with engineering and data platform teams - Communicating effectively with development, leadership, and business users - Collaborating with cross-functional teams to drive successful program execution - Meeting deadlines, prioritizing requests, and managing tasks efficiently - Serving as the primary contact for application support in Pacific time zone Professional Experience/Skills Required: - Bachelor's degree in Computer Science, Software Engineering, or equivalent - 8+ years of hands-on experience in Salesforce Marketing Cloud and related Salesforce Core products - Strong technical proficiency in various technologies including AMPScript, HTML, CSS, JavaScript, SQL, Python, etc. - Experience in systems integrations, APIs, marketing compliance, and security protocols - Ability to work under pressure, adapt quickly, and drive data-driven decisions - Excellent verbal and written communication skills - Proficiency in technical communication and project management - Strong analytical and problem-solving abilities Join our team and be part of a dynamic environment where your technical expertise and problem-solving skills will contribute to the success of our customers and business users.,

As a Data Engineer at our company, you will play a crucial role in designing, developing, and optimizing data pipelines and workflows in a cloud-based environment. Your expertise in PySpark, Snowflake, and AWS will be key as you leverage these technologies for data processing and analytics. Your responsibilities will include designing and implementing scalable ETL pipelines using PySpark on AWS, developing and optimizing data workflows for Snowflake integration, and managing and configuring various AWS services such as S3, Lambda, Glue, EMR, and Redshift. Collaboration with data analysts and business teams to understand requirements and deliver solutions will be essential, along with ensuring data security and compliance with best practices in AWS and Snowflake environments. Monitoring and troubleshooting data pipelines and workflows for performance and reliability, as well as writing efficient, reusable, and maintainable code for data processing and transformation, will also be part of your role. To excel in this position, you should have strong experience with AWS services like S3, Lambda, Glue, and MSK, proficiency in PySpark for large-scale data processing, hands-on experience with Snowflake for data warehousing and analytics, and a solid understanding of SQL and database optimization techniques. Knowledge of data lake and data warehouse architectures, familiarity with CI/CD pipelines and version control systems like Git, as well as strong problem-solving and debugging skills are also required. Experience with Terraform or CloudFormation for infrastructure as code, knowledge of Python for scripting and automation, familiarity with Apache Airflow for workflow orchestration, and understanding of data governance and security best practices will be beneficial. Certification in AWS or Snowflake is a plus. You should hold a Bachelor's degree in Computer Science, Engineering, or a related field with 6 to 10 years of experience, including 5+ years of experience in AWS cloud engineering and 2+ years of experience with PySpark and Snowflake. Join us in our Technology team as a valuable member of the Digital Software Engineering job family, working full-time to contribute your most relevant skills while continuously growing and expanding your expertise.,

You are a Data Engineer with 3+ years of experience, proficient in SQL and Python development. You will be responsible for designing, developing, and maintaining scalable data pipelines to support ETL processes using tools like Apache Airflow, AWS Glue, or similar. Your role involves optimizing and managing relational and NoSQL databases such as MySQL, PostgreSQL, MongoDB, or Cassandra for high performance and scalability. You will write advanced SQL queries, stored procedures, and functions to efficiently extract, transform, and analyze large datasets. Additionally, you will implement and manage data solutions on cloud platforms like AWS, Azure, or Google Cloud, utilizing services such as Redshift, BigQuery, or Snowflake. Your contributions to designing and maintaining data warehouses and data lakes will support analytics and BI requirements. Automation of data processing tasks through script and application development in Python or other programming languages is also part of your responsibilities. As a Data Engineer, you will implement data quality checks, monitoring, and governance policies to ensure data accuracy, consistency, and security. Collaboration with data scientists, analysts, and business stakeholders to understand data needs and translate them into technical solutions is essential. Identifying and resolving performance bottlenecks in data systems, optimizing data storage, and retrieval are key aspects. Maintaining comprehensive documentation for data processes, pipelines, and infrastructure is crucial. Staying up-to-date with the latest trends in data engineering, big data technologies, and cloud services is expected from you. You should hold a Bachelors or Masters degree in Computer Science, Information Technology, Data Engineering, or a related field. Proficiency in SQL, relational databases, NoSQL databases, Python programming, and experience with data pipeline tools and cloud platforms is required. Knowledge of big data tools like Apache Spark, Hadoop, or Kafka is a plus. Strong analytical and problem-solving skills with a focus on performance optimization and scalability are essential. Excellent verbal and written communication skills are necessary to convey technical concepts to non-technical stakeholders. You should be able to work collaboratively in cross-functional teams. Preferred certifications include AWS Certified Data Analytics, Google Professional Data Engineer, or similar. An eagerness to learn new technologies and adapt quickly in a fast-paced environment is a mindset that will be valuable in this role.,

You should have 3 to 10 years of experience in AI development and be located in Coimbatore. Immediate joiners are preferred. A minimum of 2 years of experience in core Gen AI is required. As an AI Developer, your responsibilities will include designing, developing, and fine-tuning Large Language Models (LLMs) for various in-house applications. You will implement and optimize Retrieval-Augmented Generation (RAG) techniques to enhance AI response quality. Additionally, you will develop and deploy Agentic AI systems capable of autonomous decision-making and task execution. Building and managing data pipelines for processing, transforming, and feeding structured/unstructured data into AI models will be part of your role. It is essential to ensure scalability, performance, and security of AI-driven solutions in production environments. Collaboration with cross-functional teams, including data engineers, software developers, and product managers, is expected. You will conduct experiments and evaluations to improve AI system accuracy and efficiency while staying updated with the latest advancements in AI/ML research, open-source models, and industry best practices. You should have strong experience in LLM fine-tuning using frameworks like Hugging Face, DeepSpeed, or LoRA/PEFT. Hands-on experience with RAG architectures, including vector databases such as Pinecone, ChromaDB, Weaviate, OpenSearch, and FAISS, is required. Experience in building AI agents using LangChain, LangGraph, CrewAI, AutoGPT, or similar frameworks is preferred. Proficiency in Python and deep learning frameworks like PyTorch or TensorFlow is necessary. Experience in Python web frameworks such as FastAPI, Django, or Flask is expected. You should also have experience in designing and managing data pipelines using tools like Apache Airflow, Kafka, or Spark. Knowledge of cloud platforms (AWS/GCP/Azure) and containerization technologies (Docker, Kubernetes) is essential. Familiarity with LLM APIs (OpenAI, Anthropic, Mistral, Cohere, Llama, etc.) and their integration in applications is a plus. A strong understanding of vector search, embedding models, and hybrid retrieval techniques is required. Experience with optimizing inference and serving AI models in real-time production systems is beneficial. Experience with multi-modal AI (text, image, audio) and familiarity with privacy-preserving AI techniques and responsible AI frameworks are desirable. Understanding of MLOps best practices, including model versioning, monitoring, and deployment automation, is a plus. Skills required for this role include PyTorch, RAG architectures, OpenSearch, Weaviate, Docker, LLM fine-tuning, ChromaDB, Apache Airflow, LoRA, Python, hybrid retrieval techniques, Django, GCP, CrewAI, OpenAI, Hugging Face, Gen AI, Pinecone, FAISS, AWS, AutoGPT, embedding models, Flask, FastAPI, LLM APIs, DeepSpeed, vector search, PEFT, LangChain, Azure, Spark, Kubernetes, AI Gen, TensorFlow, real-time production systems, LangGraph, and Kafka.,

As a Cloud Developer specializing in Google Cloud Platform (GCP), Python, and Apache Airflow, you will be responsible for designing and developing cloud-native applications and APIs. Your expertise in deploying microservices on GCP Cloud Run, utilizing Docker for containerization, and managing data workflows with Apache Airflow will be crucial in optimizing scalable cloud services. Collaborating with cross-functional teams in an Agile environment, you will ensure the development, maintenance, and security of cloud infrastructure using various GCP services. Your key responsibilities will include designing and developing cloud-native applications and APIs using Python and frameworks like Django, FastAPI, or Flask. You will deploy microservices on GCP Cloud Run, implement data workflows with Apache Airflow, and secure cloud infrastructure using GCP services such as Compute Engine, BigQuery, and Cloud Storage. Additionally, you will apply best practices in GCP security, collaborate with cross-functional teams, and optimize data pipelines for reliability and observability. To excel in this role, you must possess strong experience with GCP core services, proficiency in Python and frameworks, hands-on experience with Docker and Cloud Run, and a good understanding of GCP IAM and security practices. Experience with SQL, API Gateway tools, Agile development practices, and cross-functional collaboration will be essential. Nice-to-have skills include experience with CI/CD pipelines, DevOps practices, serverless architecture in GCP, and exposure to Dataflow, Pub/Sub, and BigQuery in production environments. Your problem-solving mindset, willingness to learn emerging cloud technologies, strong communication skills, and passion for automation and scalable architecture will be valuable assets in this role. Key Qualifications: - Strong experience with GCP and its core services. - Proficiency in Python and frameworks like Django, FastAPI, or Flask. - Proven experience with Apache Airflow for workflow orchestration. - Hands-on experience with Docker and Cloud Run. - Good understanding of GCP IAM and security best practices. - Experience with SQL for data manipulation and analysis. - Basic working knowledge of API Gateway tools. - Familiarity with Agile development practices and cross-functional collaboration. If you are looking for a challenging role where you can leverage your expertise in GCP, Python, and Apache Airflow to contribute to the development of scalable cloud services, this position as a Cloud Developer may be the perfect fit for you.,

As a Data Analyst in our dynamic fintech environment, your primary responsibility will be to extract, clean, and analyze large datasets to identify patterns and trends. You will play a crucial role in developing and maintaining dashboards and reports to monitor business performance. Collaborating with cross-functional teams, you will work towards improving data accuracy and accessibility. Conducting deep-dive analysis on customer behavior, transactions, and engagement will enable you to enhance retention and acquisition strategies. Additionally, you will be expected to identify potential risks, anomalies, and growth opportunities using data. To excel in this role, you should hold a Bachelor's or Master's degree in Data Science, Statistics, Mathematics, Economics, Computer Science, or a related field. A minimum of 1 year of domain exposure in Banking, Lending, or Fintech verticals is required. Proficiency in Python for data analysis, including EDA, feature engineering, and predictive analytics, is essential. You should also possess expertise in SQL for data querying and transformation, as well as mandatory experience in Tableau for building executive dashboards and visual storytelling. While not mandatory, exposure to Apache Airflow for the orchestration of ETL workflows, cloud platforms such as AWS or GCP, and version control tools like Git or Bitbucket would be beneficial. This role offers the opportunity to work on real-time data pipelines, credit risk models, and customer lifecycle analytics.,

We are seeking an experienced Apache Airflow Subject Matter Expert (SME), (Contract , Remote - India ) to join our Data Engineering team . You will be responsible for optimizing Airflow environments, building scalable orchestration frameworks, and supporting enterprise-scale data pipelines, while collaborating with cross-functional teams. Skills: Optimize and fine-tune existing Apache Airflow environments, addressing performance and reliability. Design and develop scalable, modular, and reusable Airflow DAGs for complex data workflows. Integrate Airflow with cloud-native services such as data factories, compute platforms, storage, and analytics . Develop and maintain CI/CD pipelines for DAG deployment, testing, and release automation. Implement monitoring, alerting, and logging standards to ensure operational excellence. Provide architectural guidance and hands-on support for new data pipeline development. Document Airflow configurations, deployment processes, and operational procedures. Mentor engineers and lead knowledge-sharing on orchestration best practices. Expertise in Airflow internals , including schedulers, executors (Celery, Kubernetes), and plugins. Experience with autoscaling solutions (KEDA) and Celery for distributed task execution. Strong hands-on skills in Python programming and modular code development . Proficiency with cloud services (Azure, AWS, or GCP), including data pipelines, compute, and storage. Solid experience with CI/CD tools such as Azure DevOps, Jenkins, or GitHub Actions. Familiarity with Docker, Kubernetes, and related deployment technologies. Strong background in monitoring tools (Prometheus, Grafana) and log aggregation (ELK, Log Analytics). Excellent problem-solving, communication, and collaboration skills. Interested? Please send your updated CV to jobs.india@pixelcodetech.com and a member of our resource team will be in touch.

At American Express, our culture is built on a 175-year history of innovation, shared values, and Leadership Behaviors, with an unwavering commitment to supporting our customers, communities, and colleagues. As a member of Team Amex, you will receive comprehensive support for your holistic well-being and numerous opportunities to enhance your skills, develop leadership qualities, and advance your career. Your voice and ideas hold significance here, making a tangible impact as we collectively shape the future of American Express. Enterprise Architecture, situated within the Chief Technology Office at American Express, plays a crucial role as a key enabler of the company's technology strategy. This organization focuses on four primary pillars: - Architecture as Code: Responsible for managing foundational technologies utilized by engineering teams across the enterprise. - Architecture as Design: Involves solution and technical design for transformation programs and critical projects requiring architectural guidance. - Governance: Defines technical standards and develops innovative tools to automate controls for ensuring compliance. - Colleague Enablement: Concentrates on colleague development, recognition, training, and enterprise outreach. As part of the team, your responsibilities will include: - Designing, developing, and ensuring the scalability, security, and resilience of applications and data pipelines. - Providing architectural guidance and documentation to support regulatory audits when necessary. - Contributing to enterprise architecture initiatives, domain reviews, and solution architecture. - Promoting innovation by exploring new tools, frameworks, and design methodologies. To qualify for this role, we are seeking candidates with the following qualifications: - Ideally possess a BS or MS degree in computer science, computer engineering, or a related technical discipline. - Minimum of 6 years of software engineering experience with a strong proficiency in Java and Node.js. - Experience with Python and workflow orchestration tools like Apache Airflow is highly desirable. - Demonstrated expertise in designing and implementing distributed systems and APIs. - Familiarity with cloud platforms such as GCP, AWS, and modern CI/CD pipelines. - Ability to articulate clear architectural documentation and present ideas concisely. - Proven success working collaboratively in a cross-functional, matrixed environment. - Passion for innovation, problem-solving, and driving technology modernization. - Preferred experience with microservices architectures and event-driven architecture. American Express provides benefits that cater to your holistic well-being, ensuring you can perform at your best. These benefits include competitive base salaries, bonus incentives, support for financial well-being and retirement, comprehensive medical, dental, vision, life insurance, and disability benefits, flexible working models, generous paid parental leave policies, access to global on-site wellness centers, confidential counseling support through the Healthy Minds program, and career development and training opportunities. Please note that an offer of employment with American Express is subject to the successful completion of a background verification check, as per applicable laws and regulations.,

As a Data Specialist, you will be responsible for utilizing your expertise in ETL Fundamentals, SQL, BigQuery, Dataproc, Python, Data Catalog, Data Warehousing, and various other tools to contribute to the successful implementation of data projects. Your role will involve working with technologies such as Cloud Trace, Cloud Logging, Cloud Storage, and Datafusion to build and maintain a modern data platform. To excel in this position, you should possess a minimum of 5 years of experience in the data engineering field, with a focus on GCP cloud data implementation suite including BigQuery, Pub Sub, Data Flow/Apache Beam, Airflow/Composer, and Cloud Storage. Your strong understanding of very large-scale data architecture and hands-on experience in data warehouses, data lakes, and analytics platforms will be crucial for the success of our projects. Key Requirements: - Minimum 5 years of experience in data engineering - Hands-on experience in GCP cloud data implementation suite - Strong expertise in GBQ Query, Python, Apache Airflow, and SQL (BigQuery preferred) - Extensive hands-on experience with SQL and Python for working with data If you are passionate about data and have a proven track record of delivering results in a fast-paced environment, we invite you to apply for this exciting opportunity to be a part of our dynamic team.,