340 Apache Airflow Jobs - Page 7

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-thinking approach to integrating cutting-AI technologies, particularly LLM Mesh for Generative AI applications. If you are passionate about building scalable data solutions and are eager to explore the cutting edge of AI, we encourage you to apply. Key Responsibilities : - Dataiku Leadership : Drive data engineering initiatives with a strong emphasis on leveraging Dataiku capabilities for data preparation, analysis, visualization, and the deployment of data solutions. - Data Pipeline Development : Design, develop, and optimize robust and scalable data pipelines to support various business intelligence and advanced analytics projects. This includes developing and maintaining ETL/ELT processes to automate data extraction, transformation, and loading from diverse sources. - Data Modeling & Architecture : Apply expertise in data modeling techniques to design efficient and scalable database structures, ensuring data integrity and optimal performance. - ETL/ELT Expertise : Implement and manage ETL processes and tools to ensure efficient and reliable data flow, maintaining high data quality and accessibility. - Gen AI Integration : Explore and implement solutions leveraging LLM Mesh for Generative AI applications, contributing to the development of innovative AI-powered features. - Programming & Scripting : Utilize programming languages such as Python and SQL for data manipulation, analysis, automation, and the development of custom data solutions. - Cloud Platform Deployment : Deploy and manage scalable data solutions on cloud platforms such as AWS or Azure, leveraging their respective services for optimal performance and cost-efficiency. - Data Quality & Governance : Ensure seamless integration of data sources, maintaining high data quality, consistency, and accessibility across all data assets. Implement data governance best practices. - Collaboration & Mentorship : Collaborate closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver impactful solutions. Potentially mentor junior team members. - Performance Optimization : Continuously monitor and optimize the performance of data pipelines and data systems. Required Skills & Experience : - Proficiency in Dataiku : Demonstrable expertise in Dataiku for data preparation, analysis, visualization, and building end-to-end data pipelines and applications. - Expertise in Data Modeling : Strong understanding and practical experience in various data modeling techniques (e.g., dimensional modeling, Kimball, Inmon) to design efficient and scalable database structures. - ETL/ELT Processes & Tools : Extensive experience with ETL/ELT processes and a proven track record of using various ETL tools (e.g., Dataiku's built-in capabilities, Apache Airflow, Talend, SSIS, etc.). - Familiarity with LLM Mesh : Familiarity with LLM Mesh or similar frameworks for Gen AI applications, understanding its concepts and potential for integration. - Programming Languages : Strong proficiency in Python for data manipulation, scripting, and developing data solutions. Solid command of SQL for complex querying, data analysis, and database interactions. - Cloud Platforms : Knowledge and hands-on experience with at least one major cloud platform (AWS or Azure) for deploying and managing scalable data solutions (e.g., S3, EC2, Azure Data Lake, Azure Synapse, etc.). - Gen AI Concepts : Basic understanding of Generative AI concepts and their potential applications in data engineering. - Problem-Solving : Excellent analytical and problem-solving skills with a keen eye for detail. - Communication : Strong communication and interpersonal skills to collaborate effectively with cross-functional teams. Bonus Points (Nice to Have) : - Experience with other big data technologies (e.g., Spark, Hadoop, Snowflake). - Familiarity with data governance and data security best practices. - Experience with MLOps principles and tools. - Contributions to open-source projects related to data engineering or AI. Education : Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related quantitative field.

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-thinking approach to integrating cutting-AI technologies, particularly LLM Mesh for Generative AI applications. If you are passionate about building scalable data solutions and are eager to explore the cutting edge of AI, we encourage you to apply. Key Responsibilities : - Dataiku Leadership : Drive data engineering initiatives with a strong emphasis on leveraging Dataiku capabilities for data preparation, analysis, visualization, and the deployment of data solutions. - Data Pipeline Development : Design, develop, and optimize robust and scalable data pipelines to support various business intelligence and advanced analytics projects. This includes developing and maintaining ETL/ELT processes to automate data extraction, transformation, and loading from diverse sources. - Data Modeling & Architecture : Apply expertise in data modeling techniques to design efficient and scalable database structures, ensuring data integrity and optimal performance. - ETL/ELT Expertise : Implement and manage ETL processes and tools to ensure efficient and reliable data flow, maintaining high data quality and accessibility. - Gen AI Integration : Explore and implement solutions leveraging LLM Mesh for Generative AI applications, contributing to the development of innovative AI-powered features. - Programming & Scripting : Utilize programming languages such as Python and SQL for data manipulation, analysis, automation, and the development of custom data solutions. - Cloud Platform Deployment : Deploy and manage scalable data solutions on cloud platforms such as AWS or Azure, leveraging their respective services for optimal performance and cost-efficiency. - Data Quality & Governance : Ensure seamless integration of data sources, maintaining high data quality, consistency, and accessibility across all data assets. Implement data governance best practices. - Collaboration & Mentorship : Collaborate closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver impactful solutions. Potentially mentor junior team members. - Performance Optimization : Continuously monitor and optimize the performance of data pipelines and data systems. Required Skills & Experience : - Proficiency in Dataiku : Demonstrable expertise in Dataiku for data preparation, analysis, visualization, and building end-to-end data pipelines and applications. - Expertise in Data Modeling : Strong understanding and practical experience in various data modeling techniques (e.g., dimensional modeling, Kimball, Inmon) to design efficient and scalable database structures. - ETL/ELT Processes & Tools : Extensive experience with ETL/ELT processes and a proven track record of using various ETL tools (e.g., Dataiku's built-in capabilities, Apache Airflow, Talend, SSIS, etc.). - Familiarity with LLM Mesh : Familiarity with LLM Mesh or similar frameworks for Gen AI applications, understanding its concepts and potential for integration. - Programming Languages : Strong proficiency in Python for data manipulation, scripting, and developing data solutions. Solid command of SQL for complex querying, data analysis, and database interactions. - Cloud Platforms : Knowledge and hands-on experience with at least one major cloud platform (AWS or Azure) for deploying and managing scalable data solutions (e.g., S3, EC2, Azure Data Lake, Azure Synapse, etc.). - Gen AI Concepts : Basic understanding of Generative AI concepts and their potential applications in data engineering. - Problem-Solving : Excellent analytical and problem-solving skills with a keen eye for detail. - Communication : Strong communication and interpersonal skills to collaborate effectively with cross-functional teams. Bonus Points (Nice to Have) : - Experience with other big data technologies (e.g., Spark, Hadoop, Snowflake). - Familiarity with data governance and data security best practices. - Experience with MLOps principles and tools. - Contributions to open-source projects related to data engineering or AI. Education : Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related quantitative field.

We are looking for lead or principal software engineers to join our Data Cloud team. Our Data Cloud team is responsible for the Zeta Identity Graph platform, which captures billions of behavioural, demographic, environmental, and transactional signals, for people-based marketing. As part of this team, the data engineer will be designing and growing our existing data infrastructure to democratize data access, enable complex data analyses, and automate optimization workflows for business and marketing operations. Job Description: Essential Responsibilities: As a Lead or Principal Data Engineer, your responsibilities will include: Building, refining, tuning, and maintaining our real-time and batch data infrastructure Daily use technologies such as HDFS, Spark, Snowflake, Hive, HBase, Scylla, Django, FastAPI, etc. Maintaining data quality and accuracy across production data systems Working with Data Engineers to optimize data models and workflows Working with Data Analysts to develop ETL processes for analysis and reporting Working with Product Managers to design and build data products Working with our DevOps team to scale and optimize our data infrastructure Participate in architecture discussions, influence the road map, take ownership and responsibility over new projects Participating in 24/7 on-call rotation (be available by phone or email in case something goes wrong) Desired Characteristics: Minimum 7 years of software engineering experience. Proven long term experience and enthusiasm for distributed data processing at scale, eagerness to learn new things. Expertise in designing and architecting distributed low latency and scalable solutions in either cloud and onpremises environment. Exposure to the whole software development lifecycle from inception to production and monitoring Fluency in Python or solid experience in Scala, Java Proficient with relational databases and Advanced SQL Expert in usage of services like Spark, HDFS, Hive, HBase Experience in adequate usage of any scheduler such as Apache Airflow, Apache Luigi, Chronos etc. Experience in adequate usage of cloud services (AWS) at scale Experience in agile software development processes Excellent interpersonal and communication skills Nice to have: Experience with large scale / multi-tenant distributed systems Experience with columnar / NoSQL databases Vertica, Snowflake, HBase, Scylla, Couchbase Experience in real team streaming frameworks Flink, Storm Experience with web frameworks such as Flask, Django .

About the Job :We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects.This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-thinking approach to integrating cutting-AI technologies, particularly LLM Mesh for Generative AI applications. If you are passionate about building scalable data solutions and are eager to explore the cutting edge of AI, we encourage you to apply.Key Responsibilities :- Dataiku Leadership : Drive data engineering initiatives with a strong emphasis on leveraging Dataiku capabilities for data preparation, analysis, visualization, and the deployment of data solutions.- Data Pipeline Development : Design, develop, and optimize robust and scalable data pipelines to support various business intelligence and advanced analytics projects. This includes developing and maintaining ETL/ELT processes to automate data extraction, transformation, and loading from diverse sources.- Data Modeling & Architecture : Apply expertise in data modeling techniques to design efficient and scalable database structures, ensuring data integrity and optimal performance.- ETL/ELT Expertise : Implement and manage ETL processes and tools to ensure efficient and reliable data flow, maintaining high data quality and accessibility.- Gen AI Integration : Explore and implement solutions leveraging LLM Mesh for Generative AI applications, contributing to the development of innovative AI-powered features.- Programming & Scripting : Utilize programming languages such as Python and SQL for data manipulation, analysis, automation, and the development of custom data solutions.- Cloud Platform Deployment : Deploy and manage scalable data solutions on cloud platforms such as AWS or Azure, leveraging their respective services for optimal performance and cost-efficiency.- Data Quality & Governance : Ensure seamless integration of data sources, maintaining high data quality, consistency, and accessibility across all data assets. Implement data governance best practices.- Collaboration & Mentorship : Collaborate closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver impactful solutions. Potentially mentor junior team members.- Performance Optimization : Continuously monitor and optimize the performance of data pipelines and data systems.Required Skills & Experience :- Proficiency in Dataiku : Demonstrable expertise in Dataiku for data preparation, analysis, visualization, and building end-to-end data pipelines and applications.- Expertise in Data Modeling : Strong understanding and practical experience in various data modeling techniques (e.g., dimensional modeling, Kimball, Inmon) to design efficient and scalable database structures.- ETL/ELT Processes & Tools : Extensive experience with ETL/ELT processes and a proven track record of using various ETL tools (e.g., Dataiku's built-in capabilities, Apache Airflow, Talend, SSIS, etc.). - Familiarity with LLM Mesh : Familiarity with LLM Mesh or similar frameworks for Gen AI applications, understanding its concepts and potential for integration.- Programming Languages : Strong proficiency in Python for data manipulation, scripting, and developing data solutions. Solid command of SQL for complex querying, data analysis, and database interactions.- Cloud Platforms : Knowledge and hands-on experience with at least one major cloud platform (AWS or Azure) for deploying and managing scalable data solutions (e.g., S3, EC2, Azure Data Lake, Azure Synapse, etc.).- Gen AI Concepts : Basic understanding of Generative AI concepts and their potential applications in data engineering.- Problem-Solving : Excellent analytical and problem-solving skills with a keen eye for detail.- Communication : Strong communication and interpersonal skills to collaborate effectively with cross-functional teams.Bonus Points (Nice to Have) :- Experience with other big data technologies (e.g., Spark, Hadoop, Snowflake).- Familiarity with data governance and data security best practices.- Experience with MLOps principles and tools.- Contributions to open-source projects related to data engineering or AI.Education : Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related quantitative field.

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-thinking approach to integrating cutting-AI technologies, particularly LLM Mesh for Generative AI applications. If you are passionate about building scalable data solutions and are eager to explore the cutting edge of AI, we encourage you to apply. Key Responsibilities : - Dataiku Leadership : Drive data engineering initiatives with a strong emphasis on leveraging Dataiku capabilities for data preparation, analysis, visualization, and the deployment of data solutions. - Data Pipeline Development : Design, develop, and optimize robust and scalable data pipelines to support various business intelligence and advanced analytics projects. This includes developing and maintaining ETL/ELT processes to automate data extraction, transformation, and loading from diverse sources. - Data Modeling & Architecture : Apply expertise in data modeling techniques to design efficient and scalable database structures, ensuring data integrity and optimal performance. - ETL/ELT Expertise : Implement and manage ETL processes and tools to ensure efficient and reliable data flow, maintaining high data quality and accessibility. - Gen AI Integration : Explore and implement solutions leveraging LLM Mesh for Generative AI applications, contributing to the development of innovative AI-powered features. - Programming & Scripting : Utilize programming languages such as Python and SQL for data manipulation, analysis, automation, and the development of custom data solutions. - Cloud Platform Deployment : Deploy and manage scalable data solutions on cloud platforms such as AWS or Azure, leveraging their respective services for optimal performance and cost-efficiency. - Data Quality & Governance : Ensure seamless integration of data sources, maintaining high data quality, consistency, and accessibility across all data assets. Implement data governance best practices. - Collaboration & Mentorship : Collaborate closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver impactful solutions. Potentially mentor junior team members. - Performance Optimization : Continuously monitor and optimize the performance of data pipelines and data systems. Required Skills & Experience : - Proficiency in Dataiku : Demonstrable expertise in Dataiku for data preparation, analysis, visualization, and building end-to-end data pipelines and applications. - Expertise in Data Modeling : Strong understanding and practical experience in various data modeling techniques (e.g., dimensional modeling, Kimball, Inmon) to design efficient and scalable database structures. - ETL/ELT Processes & Tools : Extensive experience with ETL/ELT processes and a proven track record of using various ETL tools (e.g., Dataiku's built-in capabilities, Apache Airflow, Talend, SSIS, etc.). - Familiarity with LLM Mesh : Familiarity with LLM Mesh or similar frameworks for Gen AI applications, understanding its concepts and potential for integration. - Programming Languages : Strong proficiency in Python for data manipulation, scripting, and developing data solutions. Solid command of SQL for complex querying, data analysis, and database interactions. - Cloud Platforms : Knowledge and hands-on experience with at least one major cloud platform (AWS or Azure) for deploying and managing scalable data solutions (e.g., S3, EC2, Azure Data Lake, Azure Synapse, etc.). - Gen AI Concepts : Basic understanding of Generative AI concepts and their potential applications in data engineering. - Problem-Solving : Excellent analytical and problem-solving skills with a keen eye for detail. - Communication : Strong communication and interpersonal skills to collaborate effectively with cross-functional teams. Bonus Points (Nice to Have) : - Experience with other big data technologies (e.g., Spark, Hadoop, Snowflake). - Familiarity with data governance and data security best practices. - Experience with MLOps principles and tools. - Contributions to open-source projects related to data engineering or AI. Education : Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related quantitative field.

We seek a professional to develop ETL pipelines with PySpark, Airflow, and Python, work with large datasets, write Oracle SQL queries, manage schemas, optimize performance, and maintain data warehouses, while guiding the team on scalable solutions.

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-thinking approach to integrating cutting-AI technologies, particularly LLM Mesh for Generative AI applications. If you are passionate about building scalable data solutions and are eager to explore the cutting edge of AI, we encourage you to apply. Key Responsibilities : - Dataiku Leadership : Drive data engineering initiatives with a strong emphasis on leveraging Dataiku capabilities for data preparation, analysis, visualization, and the deployment of data solutions. - Data Pipeline Development : Design, develop, and optimize robust and scalable data pipelines to support various business intelligence and advanced analytics projects. This includes developing and maintaining ETL/ELT processes to automate data extraction, transformation, and loading from diverse sources. - Data Modeling & Architecture : Apply expertise in data modeling techniques to design efficient and scalable database structures, ensuring data integrity and optimal performance. - ETL/ELT Expertise : Implement and manage ETL processes and tools to ensure efficient and reliable data flow, maintaining high data quality and accessibility. - Gen AI Integration : Explore and implement solutions leveraging LLM Mesh for Generative AI applications, contributing to the development of innovative AI-powered features. - Programming & Scripting : Utilize programming languages such as Python and SQL for data manipulation, analysis, automation, and the development of custom data solutions. - Cloud Platform Deployment : Deploy and manage scalable data solutions on cloud platforms such as AWS or Azure, leveraging their respective services for optimal performance and cost-efficiency. - Data Quality & Governance : Ensure seamless integration of data sources, maintaining high data quality, consistency, and accessibility across all data assets. Implement data governance best practices. - Collaboration & Mentorship : Collaborate closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver impactful solutions. Potentially mentor junior team members. - Performance Optimization : Continuously monitor and optimize the performance of data pipelines and data systems. Required Skills & Experience : - Proficiency in Dataiku : Demonstrable expertise in Dataiku for data preparation, analysis, visualization, and building end-to-end data pipelines and applications. - Expertise in Data Modeling : Strong understanding and practical experience in various data modeling techniques (e.g., dimensional modeling, Kimball, Inmon) to design efficient and scalable database structures. - ETL/ELT Processes & Tools : Extensive experience with ETL/ELT processes and a proven track record of using various ETL tools (e.g., Dataiku's built-in capabilities, Apache Airflow, Talend, SSIS, etc.). - Familiarity with LLM Mesh : Familiarity with LLM Mesh or similar frameworks for Gen AI applications, understanding its concepts and potential for integration. - Programming Languages : Strong proficiency in Python for data manipulation, scripting, and developing data solutions. Solid command of SQL for complex querying, data analysis, and database interactions. - Cloud Platforms : Knowledge and hands-on experience with at least one major cloud platform (AWS or Azure) for deploying and managing scalable data solutions (e.g., S3, EC2, Azure Data Lake, Azure Synapse, etc.). - Gen AI Concepts : Basic understanding of Generative AI concepts and their potential applications in data engineering. - Problem-Solving : Excellent analytical and problem-solving skills with a keen eye for detail. - Communication : Strong communication and interpersonal skills to collaborate effectively with cross-functional teams. Bonus Points (Nice to Have) : - Experience with other big data technologies (e.g., Spark, Hadoop, Snowflake). - Familiarity with data governance and data security best practices. - Experience with MLOps principles and tools. - Contributions to open-source projects related to data engineering or AI. Education : Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related quantitative field.

You are looking for a GCP Cloud Engineer for a position based in Pune. As a GCP Data Engineer, you will be responsible for designing, implementing, and optimizing data solutions on Google Cloud Platform. Your expertise in GCP services, solution design, and programming skills will be crucial for developing scalable and efficient cloud solutions. Your key responsibilities will include designing and implementing GCP-based data solutions following best practices, developing workflows and pipelines using Cloud Composer and Apache Airflow, building and managing data processing clusters using Dataproc, working with GCP services like Cloud Functions, Cloud Run, and Cloud Storage, and integrating multiple data sources through ETL/ELT workflows. You will be expected to write clean, efficient, and scalable code in languages such as Python, Java, or similar, apply logical problem-solving skills to address business challenges, and collaborate with stakeholders to design end-to-end GCP solution architectures. To be successful in this role, you should have hands-on experience with Dataproc, Cloud Composer, Cloud Functions, and Cloud Run, strong programming skills in Python, Java, or similar languages, a good understanding of GCP architecture, and experience in setting task dependencies in Airflow DAGs. Logical and analytical thinking, strong communication, and documentation skills are also essential for cross-functional collaboration. Preferred qualifications include GCP Professional Data Engineer or Architect Certification, experience in data lake and data warehouse solutions on GCP (e.g., BigQuery, Dataflow), and familiarity with CI/CD pipelines for GCP-based deployments.,

As a Data Engineer at our company, you will be responsible for designing scalable and robust AI/ML systems in production, focusing on high-performance and cost-effective solutions. Your expertise in various technologies, including GCP services like BigQuery, Cloud Dataflow, Pub/Sub, Dataproc, and Cloud Storage, along with programming languages such as Python, Java/Scala, and SQL, will be crucial for the success of our projects. Additionally, your experience with data processing tools like Apache Beam, Apache Kafka, and Cloud Dataprep, as well as orchestration tools like Apache Airflow and Terraform, will play a significant role in implementing efficient data pipelines. Knowledge of security protocols such as IAM, Cloud Identity, and Cloud Security Command Center, and containerization technologies like Docker and Kubernetes (GKE) will also be essential in ensuring data integrity and system security. Moreover, your familiarity with machine learning platforms like Google AI Platform, TensorFlow, and AutoML will enable you to develop and deploy cutting-edge AI models. Certification in Google Cloud Data Engineer and Cloud Architect is preferred, demonstrating your commitment to continuous learning and professional growth. In this role, you will collaborate with cross-functional teams, mentor engineers, and provide leadership to ensure that our projects meet business objectives. Your ability to implement MLOps practices, deploy models, monitor performance, and manage version control will be critical for the success of our AI/ML initiatives. Furthermore, your deep understanding of frameworks such as TensorFlow, PyTorch, and Scikit-learn, coupled with experience in data engineering principles, scalable pipelines, and distributed systems like Apache Kafka, Spark, and Kubernetes, will be invaluable assets in designing and deploying advanced machine learning models. The ideal candidate will possess strong leadership and mentorship capabilities, problem-solving skills, project management abilities, and a collaborative mindset. By fostering a positive and productive work environment, you will contribute to the success of our team and the timely delivery of high-quality solutions. At our company, you will have the opportunity to work on cutting-edge projects, collaborate with a highly motivated team, and enjoy a competitive salary, flexible schedule, and a comprehensive benefits package. Join us at Grid Dynamics, a leading provider of technology consulting and engineering services, and be part of our journey to solve complex technical challenges and drive positive business outcomes for our clients worldwide.,

You will be joining McDonald's Corporation, one of the world's largest employers with a presence in over 100 countries, at their corporate opportunity in Hyderabad. The global offices in Hyderabad serve as innovation and operations hubs, fostering McDonald's global talent pool and in-house expertise. In this role, you will play a pivotal part in developing impactful solutions for the business and customers worldwide, focusing on business, technology, analytics, and AI. As a Data Engineer at the G4 level, you will be responsible for creating scalable and efficient data solutions to support the Brand Marketing and Menu function, with a specific emphasis on the Menu Data product and associated initiatives. Collaborating with data scientists, analysts, and cross-functional teams, you will ensure the availability, reliability, and performance of data systems. Your role will involve leading initiatives to establish trust in Menu data, support decision-making, and work closely with business and technology teams to deliver scalable data solutions that provide insights into menu performance, customer preferences, and marketing effectiveness. Your expertise in cloud computing platforms, technologies, and data engineering best practices will be pivotal in this domain. Key Responsibilities: - Develop and maintain reliable Menu data products supporting menu and marketing Analytics. - Implement new technology solutions to enhance data reliability and observability. - Lead data engineering initiatives for Product Mix Analytics, ensuring timely and accurate delivery of marketing and menu-related products. - Define business rules with the Product owner to ensure high-quality Menu datasets. - Drive best practices for pipeline development, data governance, security, and quality across marketing and menu-related datasets. - Ensure scalability, maintainability, and quality of data systems supporting menu item tracking, promotion data, and marketing analytics. - Stay updated on emerging data engineering technologies, trends, and best practices for evolving Product Mix analytics needs. - Document data engineering processes, workflows, and solutions for knowledge sharing and future reference. - Mentor and coach junior data engineers, particularly in areas related to menu item tracking, promotion data, and marketing analytics. - Coordinate and collaborate with teams distributed across time zones, as required. Requirements: - Lead teams to implement scalable data engineering practices within the Menu Data ecosystem. - Hold a Bachelor's or Master's degree in computer science or related engineering field with extensive Cloud computing experience. - Possess over 5 years of professional experience in data engineering or related fields. - Proficiency in Python, Java, or Scala for data processing and automation. - Hands-on experience with data orchestration tools (e.g., Apache Airflow, Luigi) and big data ecosystems (e.g., Hadoop, Spark, NoSQL). - Expertise in Data quality functions like cleansing, standardization, parsing, de-duplication, mapping, hierarchy management, etc. - Ability to perform comprehensive data analysis using various tools. - Proven capability to mentor team members and lead technical initiatives across multiple workstreams. - Effective communication and stakeholder management skills to drive alignment and adoption of data engineering standards. - Demonstrated experience in data management and governance capabilities. - Familiarity with data warehousing principles and best practices. - Excellent problem-solving skills to utilize data and technology for resolving complex issues. - Strong collaboration skills to work efficiently in cross-functional teams. Location: Hyderabad, India Work Pattern: Full-time role Work Mode: Hybrid,

The ideal candidate should be highly interested and available immediately. Please submit your resume along with your total experience, current CTC, notice period, and current location details to Nitin.patil@ust.com. You will be responsible for designing, developing, and optimizing data pipelines and ETL workflows. Your work will involve collaborating with Apache Hadoop, Airflow, Kubernetes, and Containers to streamline data processing. Additionally, you will implement data analytics and mining techniques to derive valuable business insights. Managing cloud-based big data solutions on GCP and Azure will also be part of your job. Lastly, you will troubleshoot Hadoop log files and utilize multiple data processing engines for scalable data solutions. To excel in this role, you must possess proficiency in Scala, Spark, PySpark, Python, and SQL. Hands-on experience with the Hadoop ecosystem, Hive, Pig, and MapReduce is essential. Previous experience in ETL, Data Warehouse Design, and Data Cleansing will be highly beneficial. Familiarity with data pipeline orchestration tools like Apache Airflow is required. Knowledge of Kubernetes, Containers, and cloud platforms such as GCP and Azure is also necessary. If you are a seasoned big data engineer with a passion for Scala and cloud technologies, we encourage you to apply for this exciting opportunity.,

You should have at least 5 years of experience working as a Data Engineer. Your expertise should include a strong background in Azure Cloud services and proficiency in tools such as Azure Databricks, PySpark, and Delta Lake. It is essential to have solid experience in Python and FastAPI for API development, as well as familiarity with Azure Functions for serverless API deployments. Experience in managing ETL pipelines using Apache Airflow is also required. Hands-on experience with databases like PostgreSQL and MongoDB is necessary. Strong SQL skills and the ability to work with large datasets are key for this role.,

As a skilled developer with 5-8 years of experience, you will be responsible for developing, updating, and maintaining applications to meet specified requirements, scale efficiently, and ensure high performance. Your role will involve analyzing project requirements, designing effective solutions within the broader product architecture, and deploying APIs and web services with reusable, testable, and efficient code. You will implement low-latency, scalable applications with optimized performance and create Docker files for containerization, deploying applications within a Kubernetes environment. Your ability to adapt quickly to a dynamic, start-up style environment, demonstrate strong problem-solving skills, and a resourceful approach will be key to driving results. Your qualifications should include proficiency in Python, particularly with Fast API/Flask, along with familiarity with other web frameworks like Django and web2py. Deep understanding of RESTful API design, HTTP, JSON, database expertise in RDBMS and document-based databases, design patterns, and best practices, containerization, orchestration, scalable architecture knowledge, as well as unit testing and quality assurance are essential. You should also be proficient with Git for source code management and collaborative development. In addition to technical skills, hands-on experience in ETL processes, data pipelines, cloud services (especially AWS), microservices architecture, and CI/CD tools will be valuable. Working on technical challenges with global impact, self-development opportunities, sponsored certifications, tech talks, hackathons, and a generous benefits package including health insurance, retirement benefits, flexible work hours, and more are some of the reasons why you will love working with us. This role offers you an exciting opportunity to contribute to cutting-edge solutions and advance your career in a dynamic and collaborative environment.,

You will be responsible for developing, deploying, monitoring, and maintaining ETL Jobs as well as all data engineering and pipeline activities. Your role will involve having a good understanding of DB activities and providing support in DB solutions. Additionally, you must possess proven expertise in SQL queries. Your key responsibilities will include designing and constructing various enterprise procedure constructs using any ETL tool, preferably PentahoDI. You will be expected to provide accurate work estimates, manage efforts across multiple lines of work, design and develop exception handling and data cleansing/standardization procedures, gather requirements from various stakeholders related to ETL automation, as well as design and create data extraction, transformation, and load functions. Moreover, you will be involved in data modeling of complex large data sets, conducting tests, validating data flows, preparing ETL processes according to business requirements, and incorporating all business requirements into design specifications. As for qualifications and experience, you should hold a B.E./B.Tech/MCA degree with at least 10 years of experience in designing and developing large-scale enterprise ETL solutions. Prior experience in any ETL tool, primarily PentahoDI, and a good understanding of databases along with expertise in writing SQL queries are essential. In terms of skills and knowledge, you should have experience in full lifecycle software development and production support for DWH systems, data analysis, modeling, and design specific to a DWH/BI environment. Exposure to developing ETL packages and jobs using SPOON, scheduling Pentaho ETL Jobs in crontab, as well as familiarity with Hadoop, Hive, PIG, SQL scripting, data loading tools like Flume, Sqoop, workflow/schedulers like Oozie, and migrating existing dataflows into Big Data platforms are required. Experience in any open-source BI and databases will be considered advantageous. Joining us will provide you with impactful work where you will play a pivotal role in safeguarding Tanla's assets, data, and reputation in the industry. You will have tremendous growth opportunities as part of a rapidly growing company in the telecom and CPaaS space, with chances for professional development. Moreover, you will have the opportunity to work in an innovative environment alongside a world-class team, where innovation is celebrated. Tanla is an equal opportunity employer that champions diversity and is committed to creating an inclusive environment for all employees.,

As a Project Leader at BCN Labs, you will be an integral part of a Center of Excellence (CoE) within Bain & Company, working on delivering innovative data-driven solutions across various sectors and industries. Your role will involve collaborating with other CoEs and Practices at Bain to provide end-to-end analytical solutions that drive high-impact results for clients globally. Your primary responsibilities will include designing and implementing scalable data pipelines using modern data engineering tools, leading project teams in framing business problems and delivering strategic solutions, mentoring a team of engineers and analysts, and engaging with clients and stakeholders to communicate technical concepts effectively. Additionally, you will contribute to data infrastructure innovation and collaborate with data scientists to enable well-governed data environments and workflows. To excel in this role, you should have a Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field, along with at least 5 years of experience in data engineering, software development, and building scalable data pipelines in a production environment. Your technical skills should include expertise in Python, SQL, HTML, CSS, JavaScript, and experience with frameworks like FastAPI, Django, React, and Vue.js. Familiarity with AWS or Azure, container orchestration, and tools like Apache Airflow, PySpark, and Snowflake is highly preferred. At BCN Labs, we foster a team-oriented environment where collaboration and support are key. We believe in creating a diverse and inclusive workplace where employees can thrive both personally and professionally. As a part of Bain & Company, you will have the opportunity to work with exceptional talents and contribute to building world-class solutions that redefine industries and drive extraordinary results for our clients.,

As a Data Engineer, you will be responsible for designing and building efficient data pipelines using Azure Databricks (PySpark). Your role will involve implementing business logic for data transformation and enrichment at scale and managing and optimizing Delta Lake storage solutions. You will also be involved in developing REST APIs using FastAPI to expose processed data and deploying APIs on Azure Functions for scalable and serverless data access. Additionally, you will develop and manage Airflow DAGs to orchestrate ETL processes, ingest and process data from various internal and external sources on a scheduled basis. Database management will be a key part of your responsibilities, including handling data storage and access using PostgreSQL and MongoDB. You will write optimized SQL queries to support downstream applications and analytics. Collaboration is an essential aspect of this role, requiring you to work cross-functionally with teams to deliver reliable, high-performance data solutions. You will need to follow best practices in code quality, version control, and documentation. To excel in this role, you should have at least 5 years of hands-on experience as a Data Engineer. Strong experience with Azure Cloud services is required, along with proficiency in Azure Databricks, PySpark, and Delta Lake. Solid experience with Python and FastAPI for API development, as well as experience with Azure Functions for serverless API deployments, is essential. You should also be skilled in managing ETL pipelines using Apache Airflow and have hands-on experience with PostgreSQL and MongoDB. Strong SQL skills and experience handling large datasets are also necessary.,

As a Data Engineer (Snowflake and Python) Vice President in the Technology department, you will need to have 8-12 years of experience in designing and building highly resilient data platforms. Your strong knowledge of data engineering, architecture, and data modeling will be crucial for this role. You should possess strong analytical and problem-solving skills and have experience working with platforms like Snowflake, DBT, and Apache AirFlow. Experience in building applications on the cloud, specifically Azure, is required. Proficiency in programming skills such as Python, and optionally Java or Scala, along with a willingness to learn other languages is essential. Prior experience in working with Git, Bitbucket, Jenkins, PRs, JIRA, and following the Scrum Agile methodology will be advantageous. Your key responsibilities will include leading and mentoring a team of data management and distribution professionals to foster a culture of innovation and continuous improvement. You will collaborate with cross-functional global teams, including IT, analytics, and business units, to align data management and distribution strategies with organizational goals. Providing strategic guidance to senior leadership on data-related opportunities and risks will be a key part of your role. A deep understanding of trade lifecycles of complex OTC derivative products like Swaps, Options, FX Forward, and other financial derivatives in the context of regulatory reporting is expected. Familiarity with standard DevOps tools such as Jenkins, Sonar, and Jira will also be beneficial for this position.,

As a Lead Data Engineer at Rearc, you will play a crucial role in establishing and maintaining technical excellence within our data engineering team. Your extensive experience in data architecture, ETL processes, and data modeling will be key in optimizing data workflows for efficiency, scalability, and reliability. Collaborating closely with cross-functional teams, you will design and implement robust data solutions that align with business objectives and adhere to best practices in data management. Building strong partnerships with technical teams and stakeholders is essential as you drive data-driven initiatives and ensure their successful implementation. With over 10 years of experience in data engineering or related fields, you bring a wealth of expertise in managing and optimizing data pipelines and architectures. Your proficiency in Java and/or Python, along with experience in data pipeline orchestration using platforms like Airflow, Databricks, DBT, or AWS Glue, will be invaluable. Hands-on experience with data analysis tools and libraries such as Pyspark, NumPy, Pandas, or Dask is required, while proficiency with Spark and Databricks is highly desirable. Your proven track record of leading complex data engineering projects, coupled with hands-on experience in ETL processes, data warehousing, and data modeling tools, enables you to deliver efficient and robust data pipelines. You possess in-depth knowledge of data integration tools and best practices, as well as a strong understanding of cloud-based data services and technologies like AWS Redshift, Azure Synapse Analytics, and Google BigQuery. Your strategic and analytical skills will enable you to solve intricate data challenges and drive data-driven decision-making. In this role, you will collaborate with stakeholders to understand data requirements and challenges, implement data solutions with a DataOps mindset using modern tools and frameworks, lead data engineering projects, mentor junior team members, and promote knowledge sharing through technical blogs and articles. Your exceptional communication and interpersonal skills will facilitate collaboration with cross-functional teams and effective stakeholder engagement at all levels. At Rearc, we empower engineers to build innovative products and experiences by providing them with the best tools possible. If you are a cloud professional with a passion for problem-solving and a desire to make a difference, join us in our mission to solve problems and drive innovation in the field of data engineering.,

As the Lead Python Software Engineer at Citi, you will have the exciting opportunity to build the foundational backend systems for "Citi Assist", a Generative AI assistant that is set to reach every Citi employee globally. Joining at the ground floor, you will play a pivotal role in developing a high-visibility platform that establishes the standard for the safe and scalable adoption of this technology by a global bank. In this hands-on role, your primary responsibility will be to build, code, and deliver high-quality, tested Python backend features efficiently. Taking ownership of the features you develop, you will drive them from conception to completion, ensuring security and compliance are integral parts of your work. Collaboration and teamwork are essential as you contribute to our engineering culture through constructive code reviews, assisting peers, and sharing knowledge. Operating with a security-first mindset, you will implement best practices to address emerging challenges in AI safety. Proactive communication and autonomy are key aspects of this role, requiring you to take initiative, seek clarifications, and provide progress updates without prompting. Upholding engineering discipline, you will adhere to clean code practices, meaningful commits, and rigorous CI/CD processes in every aspect of your work. The ideal candidate for this role is a pragmatic builder who takes pride in crafting enduring software solutions. With a bias for action and a focus on delivering value, you will excel in a collaborative environment where technical excellence is valued. Your deep experience in building, deploying, and maintaining production-grade Python services, coupled with familiarity with security considerations and software design patterns, will be invaluable assets in this role. Furthermore, your proficiency in managing applications in containerized environments, experience with CI/CD pipelines, and expertise in relational databases like Postgres will be crucial for success in this position. If you have a solutions-first mindset, a drive for technical excellence, and a passion for collaborative problem-solving, we invite you to join our team at Citi and be part of our mission to build "Citi Assist" and enable the safe adoption of Generative AI across the organization.,

As an MLOps Engineer at Barclays, you will be responsible for operationalizing cutting-edge machine learning and generative AI solutions. Your main focus will be on ensuring scalable, secure, and efficient deployment across the infrastructure. Collaborating closely with data scientists, ML engineers, and business stakeholders, you will build and maintain robust MLOps pipelines. These pipelines will enable rapid experimentation and reliable production implementation of AI models, including LLMs and real-time analytics systems. To excel in this role, you should have strong programming skills in Python and hands-on experience with ML libraries such as scikit-learn, TensorFlow, and PyTorch. Additionally, familiarity with automation tools like Jenkins, GitHub Actions, or GitLab CI/CD for automating ML pipelines is essential. Proficiency in Docker and Kubernetes for scalable deployments, along with deep experience in utilizing AWS services like SageMaker, Bedrock, Lambda, and CloudFormation, is crucial. Your responsibilities will also involve managing infrastructure for training and inference using AWS services such as S3, EC2, EKS, and Step Functions. Experience with Infrastructure as Code tools like Terraform and AWS CDK, as well as model lifecycle management tools like MLflow and SageMaker Model Registry, will be beneficial. Understanding DevOps principles applied to ML workflows is key to success in this role. Additionally, highly valued skills may include experience with Snowflake, Databricks for collaborative ML development, and knowledge of data engineering tools like Apache Airflow, Kafka, and Spark. An understanding of model interpretability, responsible AI, and governance will further enhance your contributions. Involvement in open-source MLOps tools or communities, strong leadership, communication, and cross-functional collaboration skills, as well as knowledge of data privacy, model governance, and regulatory compliance in AI systems, will be advantageous. This role is based out of Pune and aims to build and maintain infrastructure platforms and products that support applications and data systems. Your accountability will involve developing, delivering, and maintaining high-quality infrastructure solutions that meet business requirements. You will play a crucial role in incident management, automation, security implementation, teamwork, and continuous learning to ensure the reliability, availability, and scalability of systems and platforms. If you are an Assistant Vice President in this role, you will be expected to advise and influence decision-making, contribute to policy development, and ensure operational effectiveness. Collaboration with other functions/business divisions, leadership of complex tasks, and coaching employees towards objectives will be part of your responsibilities. For individual contributors, leading collaborative assignments, guiding team members, identifying new directions for projects, and consulting on complex issues will be key tasks. All colleagues are expected to demonstrate Barclays Values of Respect, Integrity, Service, Excellence, and Stewardship, along with the Barclays Mindset of Empower, Challenge, and Drive.,

As a member of the Dun & Bradstreet team, you will play a crucial role in unlocking the power of data through analytics to create a better tomorrow. Our global community of over 6,000 team members is dedicated to accelerating creativity, innovation, and growth in order to help clients turn uncertainty into confidence, risk into opportunity, and potential into prosperity. We welcome bold and diverse thinkers who are passionate about making a positive impact. You will be responsible for designing and developing data pipelines within our Big Data ecosystem using technologies such as Apache Spark and Apache Airflow. Your role will involve architecting, building, and deploying scalable and efficient data pipelines while ensuring clarity and maintainability through proper documentation. Additionally, you will demonstrate expertise in data architecture and management, including familiarity with data lakes, modern data warehousing practices, and distributed data processing solutions. Your programming and scripting skills in Python will be put to the test as you write clean, efficient, and maintainable code to support cloud-based infrastructures such as AWS and GCP. You will be tasked with managing and optimizing cloud-based data infrastructure to ensure efficient data storage and retrieval. Workflow orchestration using Apache Airflow will also be a key aspect of your responsibilities, requiring you to develop and manage workflows for scheduling and orchestrating data processing jobs. Innovation and optimization are at the core of what we do, and you will be expected to create detailed designs and proof-of-concepts to enable new workloads and technical capabilities on our platform. Collaboration with platform and infrastructure engineers will be essential to implement these capabilities in production. Your strong knowledge of Big Data architecture, coupled with hands-on experience in technologies like Hadoop, Spark, and Hive, will be invaluable in this role. To be successful in this position, you should have a minimum of 8 years of hands-on experience with Big Data technologies, including at least 3 years of experience with Spark. Hands-on experience with dataproc and managing solutions deployed in the Cloud are highly desirable. Additionally, a minimum of 6 years of experience in Cloud environments, preferably GCP, and any experience with NoSQL and Graph databases will be beneficial. Experience working in a Global company, particularly in a DevOps model, is considered a plus. If you are ready to join a dynamic team of passionate individuals who are committed to driving innovation and growth, we invite you to explore career opportunities at Dun & Bradstreet by visiting https://www.dnb.com/about-us/careers-and-people/joblistings.html and https://jobs.lever.co/dnb. Official communication from Dun & Bradstreet will come from an email address ending in @dnb.com.,

You are an experienced Senior Data Engineer with a strong background in Databricks, Apache Airflow, SQL, and Python. Your main responsibility will involve designing and optimizing data pipelines, ETL workflows, and cloud-based data solutions. Your expertise in big data processing, cloud-based analytics, and automation will be crucial in developing efficient and scalable data engineering solutions. You will be responsible for tasks such as designing, developing, and optimizing data pipelines using Databricks and Apache Airflow. Implementing PySpark-based transformations and processing in Databricks to handle large-scale data, and maintaining SQL-based data pipelines with a focus on performance tuning and optimization. Additionally, you will create Python scripts for automation, data transformation, and API-based data ingestion. Your role will also involve working with Airflow DAGs to efficiently schedule and orchestrate data workflows, optimizing data lake and data warehouse performance for scalability and reliability, and integrating data pipelines with cloud platforms like AWS, Azure, or GCP along with various data storage solutions. Ensuring adherence to data security, governance, and compliance standards will be a key aspect of your responsibilities. In terms of required skills and qualifications, you should have 8-9 years of experience in Data Engineering or related fields. You must possess strong expertise in Databricks (PySpark, Delta Lake, DBSQL), proficiency in Apache Airflow for scheduling and orchestrating workflows, advanced SQL skills for data extraction, transformation, and performance tuning, as well as strong programming skills in Python (pandas, NumPy, PySpark, APIs). Experience with big data technologies, distributed computing, cloud platforms (AWS / Azure / GCP), data warehousing, data modeling concepts, CI/CD pipelines, version control (Git), data governance, security, and compliance best practices are also essential. Excellent troubleshooting, debugging, and performance optimization skills are expected from you.,

You will work as a Machine Learning Ops Engineer at Netsmart, contributing to the design, implementation, and maintenance of ML pipelines for model training, validation, and deployment. Your role involves automating model deployment processes using CI/CD pipelines and containerization technologies. You will be responsible for monitoring model performance, data drift, and system health in production environments, collaborating with data scientists to operationalize machine learning models and algorithms. Additionally, you will implement version control for models, datasets, and ML experiments using MLOps tools, optimize ML infrastructure for scalability, reliability, and cost-effectiveness, and troubleshoot and resolve issues related to model deployment and production systems. It will be part of your responsibility to maintain documentation for ML workflows, deployment processes, and system architecture. This position may require availability outside of standard business hours as part of a rotational on-call schedule. To qualify for this role, you must hold a Bachelor's degree in computer science, Information Management, or a related field. You are expected to have 2-4 years of experience in software development, DevOps, or data engineering. Proficiency in Python, SQL, and at least one ML framework such as TensorFlow, PyTorch, or Scikit-learn is required. Experience with containerization (Docker) and orchestration tools (Kubernetes), cloud platforms like AWS, Azure, GCP, and their ML services, as well as knowledge of CI/CD pipelines, version control (Git), and infrastructure as code are essential. Moreover, familiarity with monitoring tools, logging frameworks for production systems, data pipeline tools like Apache Airflow, Kubeflow, or similar, strong problem-solving skills, and the ability to work in fast-paced, collaborative environments are expected. Preferred skills include experience with MLOps platforms like MLflow, Weights & Biases, Neptune, knowledge of streaming data processing (e.g., Kafka, Kinesis), familiarity with infrastructure monitoring tools such as Prometheus, Grafana, understanding of model interpretability and explainability techniques, experience with feature stores and data versioning tools, and certification in cloud platforms like AWS ML, Azure AI, GCP ML. At Netsmart, you will be part of an inclusive environment that prioritizes your growth, offering professional development opportunities, competitive compensation packages, and comprehensive benefits. Netsmart is an Equal Opportunity Employer.,

As a Senior/Lead Data Engineer at Srijan Technologies PVT LTD, you will play a crucial role in designing, developing, and maintaining data pipelines for our clients. Your expertise in AWS Glue, Apache Airflow, Kafka, SQL, Python, and DataOps tools will be key in ensuring the efficient and reliable flow of data across the organization. You will collaborate with cross-functional teams to understand data requirements, implement DataOps practices, and optimize data pipeline performance. Monitoring data pipeline performance, troubleshooting data-related issues, and staying up-to-date with the latest industry trends in data engineering and DataOps will also be part of your responsibilities. Key Responsibilities Design, Develop, and Maintain Data Pipelines: - Develop robust and scalable data pipelines using AWS Glue, Apache Airflow, and other relevant technologies. - Integrate various data sources like SAP HANA, Kafka, and SQL databases for seamless data flow and processing. - Optimize data pipelines for performance and reliability. Data Management And Transformation - Design and implement data transformation processes to clean, enrich, and structure data for analytical purposes. - Utilize SQL and Python for data extraction, transformation, and loading (ETL) tasks. - Ensure data quality and integrity through rigorous testing and validation processes. Collaboration And Communication - Work closely with data scientists, analysts, and stakeholders to deliver solutions that meet their needs. - Collaborate with cross-functional teams to implement DataOps practices and improve data life cycle management. Monitoring And Optimization - Monitor data pipeline performance and implement improvements for enhanced efficiency. - Troubleshoot and resolve data-related issues to minimize disruption to data workflows. - Implement and manage monitoring and alerting systems proactively. Documentation And Best Practices - Maintain comprehensive documentation of data pipelines, transformations, and processes. - Adhere to best practices in data engineering, including code versioning, testing, and deployment procedures. - Stay updated with the latest industry trends and technologies in data engineering and DataOps. Required Skills and Qualifications: Technical Expertise - Extensive experience with AWS Glue for data integration and transformation. - Proficient in Apache Airflow for workflow orchestration. - Strong knowledge of Kafka for real-time data streaming and processing. - Advanced SQL skills for managing relational databases. - Proficiency in Python for scripting and automation tasks. - Experience with SAP HANA for data storage and management. - Familiarity with Data Ops tools and methodologies for continuous integration in data engineering. Preferred Skills - Knowledge of Snowflake for cloud-based data warehousing solutions. - Experience with AWS data services like Redshift, S3, and Athena. - Familiarity with big data technologies such as Hadoop, Spark, and Hive. Soft Skills - Strong analytical and problem-solving skills. - Excellent communication and collaboration abilities. - Detail-oriented with a focus on data quality and accuracy. - Ability to work independently and manage multiple projects simultaneously. Qualifications - Bachelor's or master's degree in computer science, Information Technology, or a related field. - 4 years of experience in data engineering or a related role. - Proven track record of designing and implementing complex data pipelines and workflows. Join us at Srijan Technologies PVT LTD and be part of a global digital engineering firm that values innovation, collaboration, and excellence. As part of our team, you will have the opportunity to work on high-impact projects, benefit from professional development and mentorship, and enjoy a hybrid work mode with a remote-friendly workplace. Additionally, we offer health and family insurance, 40 leaves per year, wellness programs, and more. If you are passionate about data engineering and looking to make a difference in the industry, apply now and be a part of our dynamic team.,

One of our prestigious clients, a TOP MNC Giant with a global presence, is currently seeking a Lead Enterprise Architect to join their team in Pune, Mumbai, or Bangalore. **Qualifications and Certifications:** **Education:** - Bachelors or masters degree in Computer Science, Information Technology, Engineering, or a related field. **Experience:** - A minimum of 10+ years of experience in data engineering, with at least 4 years of hands-on experience with GCP cloud platforms. - Proven track record in designing and implementing data workflows using GCP services such as BigQuery, Dataform Cloud Dataflow, Cloud Pub/Sub, and Cloud Composer. **Certifications:** - Google Cloud Professional Data Engineer certification is preferred. **Key Skills:** **Mandatory Skills:** - Advanced proficiency in Python for developing data pipelines and automation. - Strong SQL skills for querying, transforming, and analyzing large datasets. - Hands-on experience with various GCP services including Cloud Storage, Dataflow, Cloud Pub/Sub, Cloud SQL, BigQuery, Dataform, Compute Engine, and Kubernetes Engine (GKE). - Familiarity with CI/CD tools like Jenkins, GitHub, or Bitbucket. - Proficiency in Docker, Kubernetes, Terraform, or Ansible for containerization, orchestration, and infrastructure as code (IaC). - Knowledge of workflow orchestration tools such as Apache Airflow or Cloud Composer. - Strong understanding of Agile/Scrum methodologies. **Nice-to-Have Skills:** - Experience with other cloud platforms like AWS or Azure. - Familiarity with data visualization tools such as Power BI, Looker, or Tableau. - Understanding of machine learning workflows and their integration with data pipelines. **Soft Skills:** - Strong problem-solving and critical-thinking abilities. - Excellent communication skills to effectively collaborate with both technical and non-technical stakeholders. - Proactive attitude towards innovation and continuous learning. - Ability to work independently and as part of a collaborative team. If you are interested in this opportunity, please reply back with your updated CV and provide the following details: - Total Experience: - Relevant experience in Data Engineering: - Relevant experience in GCP cloud platforms: - Relevant experience as an Enterprise Architect: - Availability to join ASAP: - Preferred location (Pune / Mumbai / Bangalore): We will contact you once we receive your CV along with the above-mentioned details. Thank you, Kavita.A,