147 Apache Airflow Jobs - Page 4

As a Data Engineer II in the Deprecation Accelerator scope, you will be responsible for designing and maintaining scalable data pipelines both on-premises and on the cloud. Your role will involve understanding input and output data sources, managing upstream and downstream dependencies, and ensuring data quality. A crucial aspect of your job will be to focus on deprecating migrated workflows and migrating workflows to new systems when necessary. The ideal candidate will have expertise in tools like Git, Apache Airflow, Apache Spark, SQL, data migration, and data validation. Your key responsibilities will include: Workflow Deprecation: - Planning and executing the deprecation of migrated workflows by assessing their dependencies and consumption. - Utilizing tools and best practices to identify, mark, and communicate deprecated workflows to stakeholders. Data Migration: - Planning and executing data migration tasks to transfer data between different storage systems or formats. - Ensuring data accuracy and completeness during the migration processes. - Implementing strategies to accelerate data migration by backfilling, validating, and preparing new data assets for use. Data Validation: - Defining and implementing data validation rules to guarantee data accuracy, completeness, and reliability. - Using data validation solutions and anomaly detection methods to monitor data quality. Workflow Management: - Leveraging Apache Airflow to schedule, monitor, and automate data workflows. - Developing and managing Directed Acyclic Graphs (DAGs) in Airflow to orchestrate complex data processing tasks. Data Processing: - Creating and maintaining data processing scripts using SQL and Apache Spark. - Optimizing data processing for performance and efficiency. Version Control: - Utilizing Git for version control, collaborating with the team to manage the codebase and track changes. - Ensuring adherence to best practices in code quality and repository management. Continuous Improvement: - Staying updated with the latest advancements in data engineering and related technologies. - Continuously enhancing and refactoring data pipelines, tooling, and processes to improve performance and reliability. Skills and Qualifications: - Bachelor's degree in Computer Science, Engineering, or a related field. - Proficiency in Git for version control and collaborative development. - Strong knowledge of SQL and experience with database technologies. - Experience with data pipeline tools like Apache Airflow. - Proficiency in Apache Spark for data processing and transformation. - Familiarity with data migration and validation techniques. - Understanding of data governance and security practices. - Strong problem-solving skills and the ability to work both independently and in a team. - Excellent communication skills to collaborate with a global team. - Ability to thrive in a high-performing team environment.,

You should have a minimum of 6 years of experience in the technical field and possess the following skills: Python, Spark SQL, PySpark, Apache Airflow, DBT, Snowflake, CI/CD, Git, GitHub, and AWS. Your role will involve understanding the existing code base in AWS services and SQL, and converting it to a tech stack primarily using Airflow, Iceberg, Python, and SQL. Your responsibilities will include designing and building data models to support business requirements, developing and maintaining data ingestion and processing systems, implementing data storage solutions, ensuring data consistency and accuracy through validation and cleansing techniques, and collaborating with cross-functional teams to address data-related issues. Proficiency in Python, experience with big data Spark, orchestration experience with Airflow, and AWS knowledge are essential for this role. You should also have experience in security and governance practices such as role-based access control (RBAC) and data lineage tools, as well as knowledge of database management systems like MySQL. Strong problem-solving and analytical skills, along with excellent communication and collaboration abilities, are key attributes for this position. At NucleusTeq, we foster a positive and supportive culture that encourages our associates to perform at their best every day. We value and celebrate individual uniqueness, offering flexibility for making daily choices that contribute to overall well-being. Our well-being programs and continuous efforts to enhance our culture aim to create an environment where our people can thrive, lead healthy lives, and excel in their roles.,

As a Data Engineer at Blis, you will be part of a globally recognized and award-winning team that specializes in big data analytics and advertising. We collaborate with iconic brands like McDonald's, Samsung, and Mercedes Benz, providing precise audience insights to help them target their ideal customers effectively. Upholding ethical data practices and privacy rights is at the core of our operations, and we are committed to ensuring outstanding performance and reliability in all our systems. Working at Blis means being part of an international company with a diverse culture, spanning across four continents and comprising over 300 team members. Headquartered in the UK, we are financially successful and poised for continued growth, offering you an exciting opportunity to contribute to our journey. Your primary responsibility as a Data Engineer will involve designing and implementing high-performance data pipelines on Google Cloud Platform (GCP) to handle massive amounts of data efficiently. With a focus on scalability and automation, you will play a crucial role in building secure pipelines that can process over 350GB of data per hour and respond to 400,000 decision requests each second. Your expertise will be instrumental in driving improvements in data architecture, optimizing resource utilization, and delivering fast, accurate insights to stakeholders. Collaboration is key at Blis, and you will work closely with product and engineering teams to ensure that our data infrastructure evolves to support new initiatives seamlessly. Additionally, you will mentor and support team members, fostering a collaborative environment that encourages knowledge sharing, innovation, and professional growth. To excel in this role, you should have at least 5 years of hands-on experience with large-scale data systems, with a strong focus on designing and maintaining efficient data pipelines. Proficiency in Apache Druid and Imply platforms, along with expertise in cloud-based services like GCP, is essential. You should also have a solid understanding of Python for building and optimizing data flows, as well as experience with data governance and quality assurance practices. Furthermore, familiarity with event-driven architectures, tools like Apache Airflow, and distributed processing frameworks such as Spark will be beneficial. Your ability to apply complex algorithms and statistical techniques to large datasets, along with experience in working with relational databases and non-interactive reporting solutions, will be valuable assets in this role. Joining the Blis team means engaging in high-impact work in a data-intensive environment, collaborating with brilliant engineers, and being part of an innovative culture that prioritizes client obsession and agility. With a global reach and a commitment to diversity and inclusion, Blis offers a dynamic work environment where your contributions can make a tangible difference in the world of advertising technology.,

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-thinking approach to integrating cutting-AI technologies, particularly LLM Mesh for Generative AI applications. If you are passionate about building scalable data solutions and are eager to explore the cutting edge of AI, we encourage you to apply. Key Responsibilities : - Dataiku Leadership : Drive data engineering initiatives with a strong emphasis on leveraging Dataiku capabilities for data preparation, analysis, visualization, and the deployment of data solutions. - Data Pipeline Development : Design, develop, and optimize robust and scalable data pipelines to support various business intelligence and advanced analytics projects. This includes developing and maintaining ETL/ELT processes to automate data extraction, transformation, and loading from diverse sources. - Data Modeling & Architecture : Apply expertise in data modeling techniques to design efficient and scalable database structures, ensuring data integrity and optimal performance. - ETL/ELT Expertise : Implement and manage ETL processes and tools to ensure efficient and reliable data flow, maintaining high data quality and accessibility. - Gen AI Integration : Explore and implement solutions leveraging LLM Mesh for Generative AI applications, contributing to the development of innovative AI-powered features. - Programming & Scripting : Utilize programming languages such as Python and SQL for data manipulation, analysis, automation, and the development of custom data solutions. - Cloud Platform Deployment : Deploy and manage scalable data solutions on cloud platforms such as AWS or Azure, leveraging their respective services for optimal performance and cost-efficiency. - Data Quality & Governance : Ensure seamless integration of data sources, maintaining high data quality, consistency, and accessibility across all data assets. Implement data governance best practices. - Collaboration & Mentorship : Collaborate closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver impactful solutions. Potentially mentor junior team members. - Performance Optimization : Continuously monitor and optimize the performance of data pipelines and data systems. Required Skills & Experience : - Proficiency in Dataiku : Demonstrable expertise in Dataiku for data preparation, analysis, visualization, and building end-to-end data pipelines and applications. - Expertise in Data Modeling : Strong understanding and practical experience in various data modeling techniques (e.g., dimensional modeling, Kimball, Inmon) to design efficient and scalable database structures. - ETL/ELT Processes & Tools : Extensive experience with ETL/ELT processes and a proven track record of using various ETL tools (e.g., Dataiku's built-in capabilities, Apache Airflow, Talend, SSIS, etc.). - Familiarity with LLM Mesh : Familiarity with LLM Mesh or similar frameworks for Gen AI applications, understanding its concepts and potential for integration. - Programming Languages : Strong proficiency in Python for data manipulation, scripting, and developing data solutions. Solid command of SQL for complex querying, data analysis, and database interactions. - Cloud Platforms : Knowledge and hands-on experience with at least one major cloud platform (AWS or Azure) for deploying and managing scalable data solutions (e.g., S3, EC2, Azure Data Lake, Azure Synapse, etc.). - Gen AI Concepts : Basic understanding of Generative AI concepts and their potential applications in data engineering. - Problem-Solving : Excellent analytical and problem-solving skills with a keen eye for detail. - Communication : Strong communication and interpersonal skills to collaborate effectively with cross-functional teams. Bonus Points (Nice to Have) : - Experience with other big data technologies (e.g., Spark, Hadoop, Snowflake). - Familiarity with data governance and data security best practices. - Experience with MLOps principles and tools. - Contributions to open-source projects related to data engineering or AI. Education : Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related quantitative field.

Job Summary: We are seeking a highly skilled Data Engineer to design, develop, and maintain robust data pipelines and architectures. The ideal candidate will transform raw, complex datasets into clean, structured, and scalable formats that enable analytics, reporting, and business intelligence across the organization. This role requires strong collaboration with data scientists, analysts, and cross-functional teams to ensure timely and accurate data availability and system performance. Key Responsibilities Design and implement scalable data pipelines to support real-time and batch processing. Develop and maintain ETL/ELT processes that move, clean, and organize data from multiple sources. Build and manage modern data architectures that support efficient storage, processing, and access. Collaborate with stakeholders to understand data needs and deliver reliable solutions. Perform data transformation, enrichment, validation, and normalization for analysis and reporting. Monitor and ensure the quality, integrity, and consistency of data across systems. Optimize workflows for performance, scalability, and cost-efficiency. Support cloud and on-premise data integrations, migrations, and automation initiatives. Document data flows, schemas, and infrastructure for operational and development purposes. • Apply best practices in data governance, security, and compliance. Required Qualifications & Skills: Bachelors or Masters degree in Computer Science, Data Engineering, or a related field. Proven 6+ Years experience in data engineering, ETL development, or data pipeline management. Proficiency with tools and technologies such as: SQL, Python, Spark, Scala ETL tools (e.g., Apache Airflow, Talend) Cloud platforms (e.g., AWS, GCP, Azure) Big Data tools (e.g., Hadoop, Hive, Kafka) Data warehouses (e.g., Snowflake, Redshift, BigQuery) Strong understanding of data modeling, data architecture, and data lakes. Experience with CI/CD, version control, and working in Agile environments. Preferred Qualifications: • Experience with data observability and monitoring tools. • Knowledge of data cataloging and governance frameworks. • AWS/GCP/Azure data certification is a plus.

About Logik Are you driven to innovate Are you energized by the excitement of building a high-growth startup with winning technology and proven product-market fit Are you looking to join a team of A-players who keep customers first and take their work but not themselves seriously Logik was founded in 2021 by the godfathers of CPQ our CEO Christopher Shutts and our Executive Chairman Godard Abel, who together co-founded BigMachines, the first-ever CPQ technology vendor, in the early 2000s. Today, were reimagining what CPQ can and should be with our composable, AI-enabled platform that provides advanced configuration, transaction management, guided selling, and more. Were a well-funded and fast-growing startup disrupting the CPQ space, with founders that created the category and a platform thats pushing boundaries in configure-price-quote and complex commerce. We're looking for an exceptional AI Backend Engineer to join our Bangalore team and help us build the next generation of AI-powered solutions. Position Summary: As an Senior Backend Engineer AI & ML Specialization Engineer, you will play a crucial role in designing and developing scalable, high-performance backend systems that support our AI models and data pipelines. You will work closely with data scientists, machine learning engineers, and other backend developers to ensure our platform delivers reliable, real-time insights and predictions. Key Responsibilities: Design and develop robust, scalable backend services and APIs that handle large volumes of data and traffic. Implement data ingestion and processing pipelines to efficiently collect, store, and transform data for AI models. Develop and maintain efficient data storage solutions, including databases and data warehouses. Optimize backend systems for performance, scalability, and security. Collaborate with data scientists and machine learning engineers to integrate AI models into backend infrastructure. Collaborate with Devops to implement ML Ops and integrate the models and data engineering pipelines into highly available and reliable tech stacks. Troubleshoot and resolve technical issues related to backend systems and data pipelines. Stay up-to-date with the latest advancements in backend technologies and AI. Requirements: Bachelor's or Master's degree in Computer Science, Engineering, or a related field. 6+ years of experience in backend development, with a focus on machine learning. Strong proficiency in Python and experience with popular frameworks such as Flask, Django, or FastAPI. Experience with SQL and NoSQL databases such as PostgreSQL, MySQL, MongoDB, or Redis. Experience with cloud platforms such as AWS, Azure, or GCP. Knowledge of date engineering, data pipelines and data processing frameworks such as Apache Airflow, Apache Spark, or Dask. Knowledge of ML Ops frameworks such as Kubeflow and experience with containerisation technologies such as Docker and Kubernetes. Knowledge of distributed computing and parallel programming. Excellent communication and problem-solving skills. Ability to work independently and as part of a team. Preferred Skills: Understanding of AI concepts and machine learning frameworks (e.g., TensorFlow, PyTorch) is a plus. 3 + Years of experience with Java or Go is a plus. Experience with real-time data processing and streaming technologies. What We Offer: Competitive salary and benefits package. Opportunity to work on cutting-edge AI projects. Collaborative and supportive work environment. Continuous learning and professional development opportunities.,

Job Summary: We are seeking a highly skilled and motivated Machine Learning Engineer with a strong foundation in programming and machine learning, hands-on experience with AWS Machine Learning services (especially SageMaker), and a solid understanding of Data Engineering and MLOps practices. You will be responsible for designing, developing, deploying, and maintaining scalable ML solutions in a cloud-native environment. Key Responsibilities: • Design and implement machine learning models and pipelines using AWS SageMaker and related services. • Develop and maintain robust data pipelines for training and inference workflows. • Collaborate with data scientists, engineers, and product teams to translate business requirements into ML solutions. • Implement MLOps best practices including CI/CD for ML, model versioning, monitoring, and retraining strategies. • Optimize model performance and ensure scalability and reliability in production environments. • Monitor deployed models for drift, performance degradation, and anomalies. • Document processes, architectures, and workflows for reproducibility and compliance. Required Skills & Qualifications: • Strong programming skills in Python and familiarity with ML libraries (e.g., scikitlearn, TensorFlow, PyTorch). • Solid understanding of machine learning algorithms, model evaluation, and tuning. • Hands-on experience with AWS ML services, especially SageMaker, S3, Lambda, Step Functions, and CloudWatch. • Experience with data engineering tools (e.g., Apache Airflow, Spark, Glue) and workflow orchestration. Machine Learning Engineer - Job Description • Proficiency in MLOps tools and practices (e.g., MLflow, Kubeflow, CI/CD pipelines, Docker, Kubernetes). • Familiarity with monitoring tools and logging frameworks for ML systems. • Excellent problem-solving and communication skills. Preferred Qualifications: • AWS Certification (e.g., AWS Certified Machine Learning Specialty). • Experience with real-time inference and streaming data. • Knowledge of data governance, security, and compliance in ML systems

The ideal candidate for this position should have advanced proficiency in Python, with a solid understanding of inheritance and classes. Additionally, the candidate should be well-versed in EMR, Athena, Redshift, AWS Glue, IAM roles, CloudFormation (CFT is optional), Apache Airflow, Git, SQL, Py-Spark, Open Metadata, and Data Lakehouse. Experience with metadata management is highly desirable, particularly with AWS Services such as S3. The candidate should possess the following key skills: - Creation of ETL Pipelines - Deploying code in EMR - Querying in Athena - Creating Airflow Dags for scheduling ETL pipelines - Knowledge of AWS Lambda and ability to create Lambda functions This role is for an individual contributor, and as such, the candidate is expected to autonomously manage client communication and proactively resolve technical issues without external assistance.,

As a Technical Program Manager (TPM), you will play a crucial role in establishing a strong connection between the business and engineering departments. Your primary responsibility will involve working on intricate business constraints and translating them into essential product requirements and features. Your technical expertise will be instrumental in guiding the team from the initial project stages through to its successful launch within strict timelines. A key aspect of your role will be to demonstrate exceptional leadership skills, inspiring teams to strive for excellence and develop top-notch products. Specific requirements for this role include having a fundamental understanding of various technologies, data orchestration tools, and frameworks such as Apache Airflow, API Integrations, Micro-services Architecture, and CI/CD. Your strong communication skills will be vital in ensuring effective collaboration within the team. Additionally, you should possess knowledge of data modeling and ETL processes, along with familiarity with data streaming and real-time data processing technologies. Proficiency in utilizing data visualization tools like Tableau and Power BI to generate reports and dashboards will be beneficial for this role. An important aspect of the job involves automating repetitive tasks and workflows using scripting or automation tools. It is imperative to stay updated with the latest data technologies and industry trends, showcasing your commitment to continuous learning. Furthermore, you should be capable of explaining technical concepts and flows to non-technical audiences in a clear and concise manner. Your written communication skills must be articulate, enabling you to convey complex information effectively to engineers and Software Development Engineers in Test (SDETs). Building and nurturing strong relationships, as well as collaborating with a diverse team comprising engineering, product, and business stakeholders, will be essential for achieving success in this role.,

Genpact is a global professional services and solutions firm focused on delivering outcomes that shape the future. With over 125,000 employees across 30+ countries, we are driven by curiosity, entrepreneurial agility, and the desire to create lasting value for our clients. Our purpose, the relentless pursuit of a world that works better for people, guides us as we serve and transform leading enterprises, including the Fortune Global 500. Our deep business and industry knowledge, digital operations services, and expertise in data, technology, and AI enable us to drive results and deliver innovative solutions. We are currently seeking applications for the role of Principal Consultant Software Engineer. In this role, you will have the opportunity to work individually or mentor and train a small group in a development and migration environment. Effective communication with global customers and collaboration within a team environment are essential to succeed in this role. Responsibilities: - Possess AWS Aurora experience and strong skills in Postgres. - Design Scalable, Fault-Tolerant Data Migration Solutions to migrate data from various sources to Postgres/local databases. - Proficient in integrating problem-specific programming languages such as SQL and Python. - Develop comprehensive documentation, including functional specifications, technical designs, and unit testing documents to ensure smooth collaboration across teams. - Enhance existing data pipelines, incorporating new business requirements, optimizing performance, and ensuring data quality. - Automate processes using Python scripts to reduce manual workload. - Schedule production jobs using Apache Airflow. - Write stored procedures in Postgres/SQL. - Understand the functional importance of data in projects. - Utilize GIT and Jira for version control and process tracking. Qualifications we seek in you: Minimum Qualifications/Skills: - AWS Aurora experience and proficiency in Postgres. - Strong Python programming skills. - Solid PostgreSQL database skills. - Experience in making REST API calls using Python scripts. - Proficiency in Airflow and Agile methodology. - Excellent communication and inquisitive nature. - Willingness to work flexible hours to accommodate US overlap. Preferred Qualifications/Skills: - AWS Aurora experience and proficiency in Postgres. - Strong Python programming skills. - Solid PostgreSQL database skills. - Experience in making REST API calls using Python scripts. - Proficiency in Airflow and Agile methodology. If you are a motivated individual with the required qualifications and skills, we encourage you to apply for the Principal Consultant Software Engineer position at Genpact. Location: India-Hyderabad Schedule: Full-time Education Level: Bachelor's/Graduation/Equivalent Job Posting Date: Aug 21, 2024, 11:02:25 AM Unposting Date: Ongoing Master Skills List: Digital Job Category: Full Time,

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-thinking approach to integrating cutting-AI technologies, particularly LLM Mesh for Generative AI applications. If you are passionate about building scalable data solutions and are eager to explore the cutting edge of AI, we encourage you to apply. Key Responsibilities : - Dataiku Leadership : Drive data engineering initiatives with a strong emphasis on leveraging Dataiku capabilities for data preparation, analysis, visualization, and the deployment of data solutions. - Data Pipeline Development : Design, develop, and optimize robust and scalable data pipelines to support various business intelligence and advanced analytics projects. This includes developing and maintaining ETL/ELT processes to automate data extraction, transformation, and loading from diverse sources. - Data Modeling & Architecture : Apply expertise in data modeling techniques to design efficient and scalable database structures, ensuring data integrity and optimal performance. - ETL/ELT Expertise : Implement and manage ETL processes and tools to ensure efficient and reliable data flow, maintaining high data quality and accessibility. - Gen AI Integration : Explore and implement solutions leveraging LLM Mesh for Generative AI applications, contributing to the development of innovative AI-powered features. - Programming & Scripting : Utilize programming languages such as Python and SQL for data manipulation, analysis, automation, and the development of custom data solutions. - Cloud Platform Deployment : Deploy and manage scalable data solutions on cloud platforms such as AWS or Azure, leveraging their respective services for optimal performance and cost-efficiency. - Data Quality & Governance : Ensure seamless integration of data sources, maintaining high data quality, consistency, and accessibility across all data assets. Implement data governance best practices. - Collaboration & Mentorship : Collaborate closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver impactful solutions. Potentially mentor junior team members. - Performance Optimization : Continuously monitor and optimize the performance of data pipelines and data systems. Required Skills & Experience : - Proficiency in Dataiku : Demonstrable expertise in Dataiku for data preparation, analysis, visualization, and building end-to-end data pipelines and applications. - Expertise in Data Modeling : Strong understanding and practical experience in various data modeling techniques (e.g., dimensional modeling, Kimball, Inmon) to design efficient and scalable database structures. - ETL/ELT Processes & Tools : Extensive experience with ETL/ELT processes and a proven track record of using various ETL tools (e.g., Dataiku's built-in capabilities, Apache Airflow, Talend, SSIS, etc.). - Familiarity with LLM Mesh : Familiarity with LLM Mesh or similar frameworks for Gen AI applications, understanding its concepts and potential for integration. - Programming Languages : Strong proficiency in Python for data manipulation, scripting, and developing data solutions. Solid command of SQL for complex querying, data analysis, and database interactions. - Cloud Platforms : Knowledge and hands-on experience with at least one major cloud platform (AWS or Azure) for deploying and managing scalable data solutions (e.g., S3, EC2, Azure Data Lake, Azure Synapse, etc.). - Gen AI Concepts : Basic understanding of Generative AI concepts and their potential applications in data engineering. - Problem-Solving : Excellent analytical and problem-solving skills with a keen eye for detail. - Communication : Strong communication and interpersonal skills to collaborate effectively with cross-functional teams. Bonus Points (Nice to Have) : - Experience with other big data technologies (e.g., Spark, Hadoop, Snowflake). - Familiarity with data governance and data security best practices. - Experience with MLOps principles and tools. - Contributions to open-source projects related to data engineering or AI. Education : Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related quantitative field.

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-thinking approach to integrating cutting-AI technologies, particularly LLM Mesh for Generative AI applications. If you are passionate about building scalable data solutions and are eager to explore the cutting edge of AI, we encourage you to apply. Key Responsibilities : - Dataiku Leadership : Drive data engineering initiatives with a strong emphasis on leveraging Dataiku capabilities for data preparation, analysis, visualization, and the deployment of data solutions. - Data Pipeline Development : Design, develop, and optimize robust and scalable data pipelines to support various business intelligence and advanced analytics projects. This includes developing and maintaining ETL/ELT processes to automate data extraction, transformation, and loading from diverse sources. - Data Modeling & Architecture : Apply expertise in data modeling techniques to design efficient and scalable database structures, ensuring data integrity and optimal performance. - ETL/ELT Expertise : Implement and manage ETL processes and tools to ensure efficient and reliable data flow, maintaining high data quality and accessibility. - Gen AI Integration : Explore and implement solutions leveraging LLM Mesh for Generative AI applications, contributing to the development of innovative AI-powered features. - Programming & Scripting : Utilize programming languages such as Python and SQL for data manipulation, analysis, automation, and the development of custom data solutions. - Cloud Platform Deployment : Deploy and manage scalable data solutions on cloud platforms such as AWS or Azure, leveraging their respective services for optimal performance and cost-efficiency. - Data Quality & Governance : Ensure seamless integration of data sources, maintaining high data quality, consistency, and accessibility across all data assets. Implement data governance best practices. - Collaboration & Mentorship : Collaborate closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver impactful solutions. Potentially mentor junior team members. - Performance Optimization : Continuously monitor and optimize the performance of data pipelines and data systems. Required Skills & Experience : - Proficiency in Dataiku : Demonstrable expertise in Dataiku for data preparation, analysis, visualization, and building end-to-end data pipelines and applications. - Expertise in Data Modeling : Strong understanding and practical experience in various data modeling techniques (e.g., dimensional modeling, Kimball, Inmon) to design efficient and scalable database structures. - ETL/ELT Processes & Tools : Extensive experience with ETL/ELT processes and a proven track record of using various ETL tools (e.g., Dataiku's built-in capabilities, Apache Airflow, Talend, SSIS, etc.). - Familiarity with LLM Mesh : Familiarity with LLM Mesh or similar frameworks for Gen AI applications, understanding its concepts and potential for integration. - Programming Languages : Strong proficiency in Python for data manipulation, scripting, and developing data solutions. Solid command of SQL for complex querying, data analysis, and database interactions. - Cloud Platforms : Knowledge and hands-on experience with at least one major cloud platform (AWS or Azure) for deploying and managing scalable data solutions (e.g., S3, EC2, Azure Data Lake, Azure Synapse, etc.). - Gen AI Concepts : Basic understanding of Generative AI concepts and their potential applications in data engineering. - Problem-Solving : Excellent analytical and problem-solving skills with a keen eye for detail. - Communication : Strong communication and interpersonal skills to collaborate effectively with cross-functional teams. Bonus Points (Nice to Have) : - Experience with other big data technologies (e.g., Spark, Hadoop, Snowflake). - Familiarity with data governance and data security best practices. - Experience with MLOps principles and tools. - Contributions to open-source projects related to data engineering or AI. Education : Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related quantitative field.

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-thinking approach to integrating cutting-AI technologies, particularly LLM Mesh for Generative AI applications. If you are passionate about building scalable data solutions and are eager to explore the cutting edge of AI, we encourage you to apply. Key Responsibilities : - Dataiku Leadership : Drive data engineering initiatives with a strong emphasis on leveraging Dataiku capabilities for data preparation, analysis, visualization, and the deployment of data solutions. - Data Pipeline Development : Design, develop, and optimize robust and scalable data pipelines to support various business intelligence and advanced analytics projects. This includes developing and maintaining ETL/ELT processes to automate data extraction, transformation, and loading from diverse sources. - Data Modeling & Architecture : Apply expertise in data modeling techniques to design efficient and scalable database structures, ensuring data integrity and optimal performance. - ETL/ELT Expertise : Implement and manage ETL processes and tools to ensure efficient and reliable data flow, maintaining high data quality and accessibility. - Gen AI Integration : Explore and implement solutions leveraging LLM Mesh for Generative AI applications, contributing to the development of innovative AI-powered features. - Programming & Scripting : Utilize programming languages such as Python and SQL for data manipulation, analysis, automation, and the development of custom data solutions. - Cloud Platform Deployment : Deploy and manage scalable data solutions on cloud platforms such as AWS or Azure, leveraging their respective services for optimal performance and cost-efficiency. - Data Quality & Governance : Ensure seamless integration of data sources, maintaining high data quality, consistency, and accessibility across all data assets. Implement data governance best practices. - Collaboration & Mentorship : Collaborate closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver impactful solutions. Potentially mentor junior team members. - Performance Optimization : Continuously monitor and optimize the performance of data pipelines and data systems. Required Skills & Experience : - Proficiency in Dataiku : Demonstrable expertise in Dataiku for data preparation, analysis, visualization, and building end-to-end data pipelines and applications. - Expertise in Data Modeling : Strong understanding and practical experience in various data modeling techniques (e.g., dimensional modeling, Kimball, Inmon) to design efficient and scalable database structures. - ETL/ELT Processes & Tools : Extensive experience with ETL/ELT processes and a proven track record of using various ETL tools (e.g., Dataiku's built-in capabilities, Apache Airflow, Talend, SSIS, etc.). - Familiarity with LLM Mesh : Familiarity with LLM Mesh or similar frameworks for Gen AI applications, understanding its concepts and potential for integration. - Programming Languages : Strong proficiency in Python for data manipulation, scripting, and developing data solutions. Solid command of SQL for complex querying, data analysis, and database interactions. - Cloud Platforms : Knowledge and hands-on experience with at least one major cloud platform (AWS or Azure) for deploying and managing scalable data solutions (e.g., S3, EC2, Azure Data Lake, Azure Synapse, etc.). - Gen AI Concepts : Basic understanding of Generative AI concepts and their potential applications in data engineering. - Problem-Solving : Excellent analytical and problem-solving skills with a keen eye for detail. - Communication : Strong communication and interpersonal skills to collaborate effectively with cross-functional teams. Bonus Points (Nice to Have) : - Experience with other big data technologies (e.g., Spark, Hadoop, Snowflake). - Familiarity with data governance and data security best practices. - Experience with MLOps principles and tools. - Contributions to open-source projects related to data engineering or AI. Education : Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related quantitative field.

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-thinking approach to integrating cutting-AI technologies, particularly LLM Mesh for Generative AI applications. If you are passionate about building scalable data solutions and are eager to explore the cutting edge of AI, we encourage you to apply. Key Responsibilities : - Dataiku Leadership : Drive data engineering initiatives with a strong emphasis on leveraging Dataiku capabilities for data preparation, analysis, visualization, and the deployment of data solutions. - Data Pipeline Development : Design, develop, and optimize robust and scalable data pipelines to support various business intelligence and advanced analytics projects. This includes developing and maintaining ETL/ELT processes to automate data extraction, transformation, and loading from diverse sources. - Data Modeling & Architecture : Apply expertise in data modeling techniques to design efficient and scalable database structures, ensuring data integrity and optimal performance. - ETL/ELT Expertise : Implement and manage ETL processes and tools to ensure efficient and reliable data flow, maintaining high data quality and accessibility. - Gen AI Integration : Explore and implement solutions leveraging LLM Mesh for Generative AI applications, contributing to the development of innovative AI-powered features. - Programming & Scripting : Utilize programming languages such as Python and SQL for data manipulation, analysis, automation, and the development of custom data solutions. - Cloud Platform Deployment : Deploy and manage scalable data solutions on cloud platforms such as AWS or Azure, leveraging their respective services for optimal performance and cost-efficiency. - Data Quality & Governance : Ensure seamless integration of data sources, maintaining high data quality, consistency, and accessibility across all data assets. Implement data governance best practices. - Collaboration & Mentorship : Collaborate closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver impactful solutions. Potentially mentor junior team members. - Performance Optimization : Continuously monitor and optimize the performance of data pipelines and data systems. Required Skills & Experience : - Proficiency in Dataiku : Demonstrable expertise in Dataiku for data preparation, analysis, visualization, and building end-to-end data pipelines and applications. - Expertise in Data Modeling : Strong understanding and practical experience in various data modeling techniques (e.g., dimensional modeling, Kimball, Inmon) to design efficient and scalable database structures. - ETL/ELT Processes & Tools : Extensive experience with ETL/ELT processes and a proven track record of using various ETL tools (e.g., Dataiku's built-in capabilities, Apache Airflow, Talend, SSIS, etc.). - Familiarity with LLM Mesh : Familiarity with LLM Mesh or similar frameworks for Gen AI applications, understanding its concepts and potential for integration. - Programming Languages : Strong proficiency in Python for data manipulation, scripting, and developing data solutions. Solid command of SQL for complex querying, data analysis, and database interactions. - Cloud Platforms : Knowledge and hands-on experience with at least one major cloud platform (AWS or Azure) for deploying and managing scalable data solutions (e.g., S3, EC2, Azure Data Lake, Azure Synapse, etc.). - Gen AI Concepts : Basic understanding of Generative AI concepts and their potential applications in data engineering. - Problem-Solving : Excellent analytical and problem-solving skills with a keen eye for detail. - Communication : Strong communication and interpersonal skills to collaborate effectively with cross-functional teams. Bonus Points (Nice to Have) : - Experience with other big data technologies (e.g., Spark, Hadoop, Snowflake). - Familiarity with data governance and data security best practices. - Experience with MLOps principles and tools. - Contributions to open-source projects related to data engineering or AI. Education : Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related quantitative field.

-Maintain and update our in-house Shiny dashboard (driven by R) -Build SQL data pipelines for merchants and business analytics team -Collaborate with merchants, sales, and other stakeholders to identify and implement solutions to business needs.

Must have skills required: GCP, support, Python Forbes Advisor is Looking for: Role Summary We are seeking a proactive and detail-oriented Data Support Engineer- to monitor production processes, manage incident tickets, and ensure seamless operations in our data platforms. The ideal candidate will have experience in Google Cloud Platform (GCP), Airflow, Python and SQL with a strong focus on enabling developer productivity and maintaining system reliability. Key Responsibilities: Production Monitoring: Monitor and ensure the smooth execution of production data pipelines and workflows. Identify and promptly address anomalies or failures in the production environment. Perform first-level investigation for issues, leveraging logs and monitoring tools. Incident Management: Create and manage tickets for identified production issues, ensuring accurate documentation of details and impact analysis. Assign tickets to the appropriate development teams and follow up to ensure timely resolution. Communication of incidents within the Data Team. Platform Support: Participate in daily standup and team meetings and contribute to platform improvement initiatives. Contribute to enhancing the platform to streamline development workflows and improve system usability. Required Skills: Bachelors degree with Minimum 1 year of experience working in supporting the production pipelines. Proficiency in SQL for debugging tasks. Familiarity with incident management tools like JIRA. Strong communication skills to interact with cross-functional teams and stakeholders. Good to have: Hands-on experience with Google Cloud Platform (GCP) services like BigQuery. Strong understanding of Apache Airflow and managing DAGs. Basic understanding of DevOps practices and automating CI/CD pipelines. Python Proficiency Note: This role requires candidates to work in UK timings. Saturday and Sunday will be working. Rotational off will be provided. Qualifications Bachelors degree in full time.

Job Title: Business Intelligence Analyst (R Shiny Developer) Position Overview: We are seeking an entry-level Business Intelligence Analyst (R Shiny Developer) to join our business analytics team. In this high-impact role, you will be responsible for maintaining and enhancing our in-house merchandising Shiny dashboard, adding new functionality to automate processes and analytics for our merchants. Additionally, you will be responsible for supporting the business analytics team with their business intelligence and data engineering needs. This position requires collaboration across multiple departments and offers an excellent opportunity to make a significant contribution to our company's success. Key Responsibilities: Maintain and update our in-house Shiny dashboard (driven by R) to ensure reliability and performance. Build SQL data pipelines for merchants and business analytics team. Develop new features and functionality to support automation and streamline merchant processes. Collaborate with merchants, sales, and other stakeholders to identify and implement solutions to business needs. Qualifications: Programming Skills: Proficiency in SQL, R, and Shiny is required. Experience with Apache Airflow is a plus. Education: A degree in Information Systems, Mathematics, Data Science, Computer Science, Statistics, or another STEM/technical business field. Experience: 2-3 years minimum in R Shiny development. Strong analytical and problem-solving skills. Excellent communication and collaboration skills. Self-starter with the ability to work independently and as part of a team.

Design and implement automated test cases for ETL processes and data pipelines Perform data validation, data transformation, and reconciliation testing Write and execute complex SQL queries to validate source-to-target data mappings Required Candidate profile Work closely ETL developers, business analysts, QA teams Log, track, and report defects using tools like JIRA, HP ALM, or TestRail Support regression testing, UAT, and performance testing for ETL jobs Perks and benefits Perks and Benefits

We are seeking a highly skilled Senior Data Engineer to join our dynamic team in Bangalore. You will design, develop, and maintain scalable data ingestion frameworks and ELT pipelines using tools such as DBT, Apache Airflow, and Prefect. The ideal candidate will have deep technical expertise in cloud platforms (especially AWS), data architecture, and orchestration tools. You will work with modern cloud data warehouses like Snowflake, Redshift, or Databricks and integrate pipelines with AWS services such as S3, Lambda, Step Functions, and Glue. A strong background in SQL, scripting, and CI/CD practices is essential. Experience with data systems in manufacturing is a plus.

We are looking for energetic, self-motivated and exceptional Data engineers to work on extraordinary enterprise products based on AI and Big Data engineering leveraging AWS/Databricks tech stack. He/she will work with a star team of Architects, Data Scientists/AI Specialists, Data Engineers and Integration. Skills and Qualifications: 5+ years of experience in DWH/ETL Domain; Databricks/AWS tech stack 2+ years of experience in building data pipelines with Databricks/ PySpark /SQL Experience in writing and interpreting SQL queries, designing data models and data standards. Experience in SQL Server databases, Oracle and/or cloud databases. Experience in data warehousing and data mart, Star and Snowflake model. Experience in loading data into databases from databases and files. Experience in analyzing and drawing design conclusions from data profiling results. Understanding business processes and relationship of systems and applications. Must be comfortable conversing with the end-users. Must have the ability to manage multiple projects/clients simultaneously. Excellent analytical, verbal and communication skills. Role and Responsibilities: Work with business stakeholders and build data solutions to address analytical & reporting requirements. Work with application developers and business analysts to implement and optimise Databricks/AWS-based implementations meeting data requirements. Design, develop, and optimize data pipelines using Databricks (Delta Lake, Spark SQL, PySpark), AWS Glue, and Apache Airflow Implement and manage ETL workflows using Databricks notebooks, PySpark and AWS Glue for efficient data transformation Develop/ optimize SQL scripts, queries, views, and stored procedures to enhance data models and improve query performance on managed databases. Conduct root cause analysis and resolve production problems and data issues. Create and maintain up to date documentation of the data model, data flow and field level mappings. Provide support for production problems and daily batch processing. Provide ongoing maintenance and optimization of database schemas, data lake structures (Delta Tables, Parquet), and views to ensure data integrity and performance

Experience : 3+ years Shift : (GMT+05:30) Asia/Kolkata (IST) Must have Skills required: Python, Django, Fast API, Prompt Engineering, Apache Airflow, Mongo DB As a Python Developer at Infrrd, your job responsibilities will include: Create python scripts and services. Document inside code and designs. Ensure the deliverables are of the highest quality in terms of functional and technical aspects. Ensure the in sprint defects are closed with the highest quality and any production defects are being taken care of for a specific duration post release. Work along with the PM, Architect and the BA to manage the deliverables from technical and functional aspects Work along with DevOps, and other teams to fulfill roles and see the big picture. Carry out regular code reviews, enforce coding best practices, source control, and deliverables Below is a list of the background we would like our Python Developer to have: 3 -5 years of expertise in Python 3 software engineering Skills Experience building systems that interact with REST APIs or depend on web frameworks like Flask (preferred) or Django. Python Language Proficiency: Strong understanding of Python syntax and libraries like Pandas. Design, Develop, test, and maintain high-quality software using Python programming language. Experience with MongoDB Strong object-oriented design and programming Skills Experience with SQL and NoSQL databases. Proficiency with Git. Nice to Have: Familiarity with Linux. Experience with microservices and architectural design. Experience with Python testing frameworks (unit test, pytest). Knowledge of using OpenAPI specifications for designing REST Endpoints. Experience working with message brokers (RabbitMQ, Redis) Experience in deploying machine learning models in production environments and made them accessible through APIs. Experience with DevOps processes along with Kubernetes and container ecosystem is an advantage. Experience working with AWS services or other cloud services. Skills Python, Django, Fast API, Prompt Engineering, Apache Airflow, Mongo DB

We are looking for energetic, self-motivated and exceptional Data engineer to work on extraordinary enterprise products based on AI and Big Data engineering leveraging AWS/Databricks tech stack. He/she will work with star team of Architects, Data Scientists/AI Specialists, Data Engineers and Integration. Skills and Qualifications: 5+ years of experience in DWH/ETL Domain; Databricks/AWS tech stack 2+ years of experience in building data pipelines with Databricks/ PySpark /SQL Experience in writing and interpreting SQL queries, designing data models and data standards. Experience in SQL Server databases, Oracle and/or cloud databases. Experience in data warehousing and data mart, Star and Snowflake model. Experience in loading data into database from databases and files. Experience in analyzing and drawing design conclusions from data profiling results. Understanding business process and relationship of systems and applications. Must be comfortable conversing with the end-users. Must have ability to manage multiple projects/clients simultaneously. Excellent analytical, verbal and communication skills. Role and Responsibilities: Work with business stakeholders and build data solutions to address analytical & reporting requirements. Work with application developers and business analysts to implement and optimise Databricks/AWS-based implementations meeting data requirements. Design, develop, and optimize data pipelines using Databricks (Delta Lake, Spark SQL, PySpark), AWS Glue, and Apache Airflow Implement and manage ETL workflows using Databricks notebooks, PySpark and AWS Glue for efficient data transformation Develop/ optimize SQL scripts, queries, views, and stored procedures to enhance data models and improve query performance on managed databases. Conduct root cause analysis and resolve production problems and data issues. Create and maintain up to date documentation of the data model, data flow and field level mappings. Provide support for production problems and daily batch processing. Provide ongoing maintenance and optimization of database schemas, data lake structures (Delta Tables, Parquet), and views to ensure data integrity and performance. Immediate Joiners.

We are looking for energetic, self-motivated and exceptional Data engineer to work on extraordinary enterprise products based on AI and Big Data engineering leveraging AWS/Databricks tech stack. He/she will work with star team of Architects, Data Scientists/AI Specialists, Data Engineers and Integration. Skills and Qualifications: 5+ years of experience in DWH/ETL Domain; Databricks/AWS tech stack 2+ years of experience in building data pipelines with Databricks/ PySpark /SQL Experience in writing and interpreting SQL queries, designing data models and data standards. Experience in SQL Server databases, Oracle and/or cloud databases. Experience in data warehousing and data mart, Star and Snowflake model. Experience in loading data into database from databases and files. Experience in analyzing and drawing design conclusions from data profiling results. Understanding business process and relationship of systems and applications. Must be comfortable conversing with the end-users. Must have ability to manage multiple projects/clients simultaneously. Excellent analytical, verbal and communication skills. Role and Responsibilities: Work with business stakeholders and build data solutions to address analytical & reporting requirements. Work with application developers and business analysts to implement and optimise Databricks/AWS-based implementations meeting data requirements. Design, develop, and optimize data pipelines using Databricks (Delta Lake, Spark SQL, PySpark), AWS Glue, and Apache Airflow Implement and manage ETL workflows using Databricks notebooks, PySpark and AWS Glue for efficient data transformation Develop/ optimize SQL scripts, queries, views, and stored procedures to enhance data models and improve query performance on managed databases. Conduct root cause analysis and resolve production problems and data issues. Create and maintain up to date documentation of the data model, data flow and field level mappings. Provide support for production problems and daily batch processing. Provide ongoing maintenance and optimization of database schemas, data lake structures (Delta Tables, Parquet), and views to ensure data integrity and performance

5 to 7 years of experience in data engineering Architect and maintain scalable, secure, and reliable data platforms and pipelines Design and implement data lake/data warehouse solutions such as Redshift, BigQuery, Snowflake, or Delta Lake Build real-time and batch data pipelines using tools like Apache Airflow, Kafka, Spark, and DBT Ensure data governance, lineage, quality, and observability

Notice period: Immediate 15days Profile source: Tamil Nadu Timings: 1:00pm 10:00pm (IST) Work Mode: WFO (Mon-Fri) We are seeking a detail-oriented and highly motivated Data Engineer to join our growing Data & Analytics team. In this role, you will be responsible for designing, building, and maintaining robust data pipelines and infrastructure that power insights across the organization. Youll work closely with data scientists, analysts, and engineers to ensure the integrity, accessibility, and scalability of our data systems. Key Responsibilities Responsibilities: Design, develop, and maintain scalable data pipelines and ETL Build and optimize data architecture to ensure data quality and Integrate data from diverse internal and external Collaborate with cross-functional teams to define data requirements and deliver Implement best practices for data governance, security, and Monitor pipeline performance and perform real-time troubleshooting of data Participate in code reviews and contribute to documentation and Required Qualifications & Skills: Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent practical experience). Minimum of 2 years of professional experience in data engineering or software Solid understanding of SQL and proficiency in at least one programming language, such as Python, Java, or Scala. Practical experience building and maintaining data pipelines using tools like Apache Airflow or Hands-on experience with cloud platforms (AWS, GCP, Azure) and data warehousing solutions (Redshift, BigQuery, Snowflake). Familiarity with big data technologies and frameworks, including Spark, Kafka, and Demonstrated ability to solve complex problems with a strong focus on Experience implementing CI/CD practices for data Working knowledge of data modeling principles and schema Exposure to machine learning pipelines or real-time analytics systems is a plus.