491 Data Pipeline Jobs - Page 5

Hi Candidates, we have job openings in one of our MNC Company Interested candidates can apply here and share details to chandrakala.c@i-q.co Note: NP-0-15 days only serving Role & responsibilities Key Responsibilities Complete Data Modelling Tasks o Initiate and manage Gap Analysis and Source-to-Target Mapping Exercises. o Gain a comprehensive understanding of the EA extract. o Map the SAP source used in EA extracts to the AWS Transform Zone, AWS Conform Zone, and AWS Enrich Zone. Develop a matrix view of all Excel/Tableau reports to identify any missing fields or tables from SAP in the Transform Zone. Engage with SMEs to finalize the Data Model (DM). Obtain email confirmation and approval for the finalized DM. o Perform data modelling using ER Studio and STTM. o Generate DDL scripts for data engineers to facilitate implementation. Complete Data Engineering Tasks o Set up infrastructure for pipelines this includes Glue Jobs, crawlers, scheduling, step functions etc. Build, deploy, test and run pipelines on demand in lower environments. Verify data integrity: no duplicates, all columns in final table etc. Write unit tests for methods used in pipeline and use standard tools for testing. o Code formatting and linting. o Collaborate with other Modelling Engineers to align on correct approach. o Update existing pipelines for CZ tables (SDLF and OF) where necessary with new columns if they are required for EZ tables. Raise DDP requests to register databases and tables, and to load data into the raw zone. Create comprehensive good documentation. Ensure each task is accompanied by detailed notes specific to its functional area for clear tracking and reference. Analyse and manage bugs, and change requests raised by business/SMEs. Collaborate with Data Analyst and Virtual Engineers (VE) to refine and enhance semantic modelling in Power BI. Plan out work using Microsoft Azure, ADO. Dependencies, status and effort is correctly reflected. Required Skills and Experience: Proven experience in data modelling and data pipeline development. Proficiency with tools like ER Studio, STTM, AWS Glue, Redshift & Athena, and Power BI. Strong SQL and experience with generating DDL scripts. Experience working in SAP data environments. Experience in any of these domain areas is highly desirable: Logistics, Supply Planning, Exports and IFOT. Familiarity with cloud platforms, particularly AWS. Hands-on experience with DevOps and Agile methodologies (e.g., Azure ADO). Strong communication and documentation skills. Ability to work collaboratively with cross-functional teams. Preferred candidate profile

Role Description: As a Data Engineering Lead, you will play a crucial role in overseeing the design, development, and maintenance of our organization's data architecture and infrastructure. You will be responsible for designing and developing the architecture for the data platform that ensures the efficient and effective processing of large volumes of data, enabling the business to make informed decisions based on reliable and high-quality data. The ideal candidate will have a strong background in data engineering, excellent leadership skills, and a proven track record of successfully managing complex data projects. Responsibilities : Data Architecture and Design: Design and implement scalable and efficient data architectures to support the organization's data processing needs Work closely with cross-functional teams to understand data requirements and ensure that data solutions align with business objectives ETL Development: Oversee the development of robust ETL processes to extract, transform, and load data from various sources into the data warehouse Ensure data quality and integrity throughout the ETL process, implementing best practices for data cleansing and validation Big Data Technology - Stay abreast of emerging trends and technologies in big data and analytics, and assess their applicability to the organization's data strategy Implement and optimize big data technologies to process and analyze large datasets efficiently Cloud Integration: Collaborate with the IT infrastructure team to integrate data engineering solutions with cloud platforms, ensuring scalability, security, and performance. Performance Monitoring and Optimization: Implement monitoring tools and processes to track the performance of data pipelines and proactively address any issues Optimize data processing. Documentation: Maintain comprehensive documentation for data engineering processes, data models, and system architecture Ensure that team members follow documentation standards and best practices. Collaboration and Communication: Collaborate with data scientists, analysts, and other stakeholders to understand their data needs and deliver solutions that meet those requirements Communicate effectively with technical and non-technical stakeholders, providing updates on project status, challenges, and opportunities. Qualifications: Bachelor's or Master's degree in Computer Science, Information Technology, or a related field. 6-8 years of professional experience in data engineering In-depth knowledge of data modeling, ETL processes, and data warehousing. In-depth knowledge of building the data warehouse using Snowflake Should have experience in data ingestion, data lakes, data mesh and data governance Must have experience in Python programming Strong understanding of big data technologies and frameworks, such as Hadoop, Spark, and Kafka. Experience with cloud platforms, such as AWS, Azure, or Google Cloud. Familiarity with database systems like SQL, NoSQL, and data pipeline orchestration tools .Excellent problem-solving and analytical skills .Strong communication and interpersonal skills. Proven ability to work collaboratively in a fast-paced, dynamic environment.

Job Title: Data Engineer Location: Pune Experience: Fresher 5 years Employment Type: Full-Time About the Role: We are looking for talented Data Engineers to join our expanding team. You will be responsible for integrating, managing, and optimizing data systems thus contributing to innovative and challenging projects. This is a great opportunity to enhance your technical skills and become a part of a successful team. Key Responsibilities: Develop, customize, and manage data integration tools, databases (MySQL), and data warehouses using Python, Java, and other relevant technologies. Write and execute complex queries and automation scripts for processing operational data. Collaborate with the testing team to assess the reliability and performance of systems. Deploy data models into production, manage computing resources, and implement monitoring tools. Create tools for data visualization, reporting, and support for data analysis. Monitor and optimize data pipeline performance and stability, ensuring alignment with evolving data models. Work with senior engineers to implement improvements and process enhancements. Stay updated on data engineering best practices, focusing on automation and efficiency. Assist in the development of Machine Learning models by providing clean and structured data pipelines. Mentor and collaborate with peers, contributing to a learning-focused work environment. Technology Stack & Required Skills: Strong understanding of ETL processes, experience with tools like Airflow. Hands-on experience with Python, Java and relational databases (e.g., MySQL) and NoSQL databases (e.g., Cassandra, MongoDB). Knowledge of cloud platforms such as AWS or Azure, with experience in tools like Azure Data Factory or AWS Snowflake. Basic understanding of Machine Learning concepts and experience supporting ML models (optional but preferred). Experience with Big Data tools such as Spark and streaming tools like Kafka is a plus. Familiarity with data visualization tools like PowerBI or Tableau. Strong ability to handle large data volumes efficiently. Knowledge of GenAI LLM models. Qualifications: BE/ME/MCA in Computer Engineering or related field. Fresher-5 years of experience in Data Engineering roles. Prior experience working in Agile teams and contributing to complex projects.

Qualification BCA. MCA preferable Required Skill Set 5+ years in Data Engineering, with at least 2 years on GCP/BigQuery Strong Python and SQL expertise (Airflow, dbt or similar) Deep understanding of ETL patterns, change-data-capture, and data-quality frameworks Experience with IoT or time-series data pipelines a plus Excellent communication skills and track record of leading cross-functional teams Job Description / Responsibilities Design, build, and maintain scalable ETL/ELT pipelines in Airflow and BigQuery Define and enforce data-modeling standards, naming conventions, and testing frameworks Develop and review core transformations: IoT enrichment (batch-ID assignment, stage tagging) Transactional ETL (ERPNext/MariaDB BigQuery) Finance automation pipelines (e.g., bank reconciliation) Create and manage schema definitions for staging, enriched_events, and erp_batch_overview tables Implement data-quality tests (using dbt or custom Airflow operators) and oversee QA handoff Collaborate closely with DevOps to ensure CI/CD, monitoring, and cost-efficient operations Drive documentation, runbooks, and knowledge transfer sessions Mentor and coordinate with freelance data engineers and analytics team members Desired profile of the Proficiency in Python and SQL , including working with Airflow and dbt or similar tools. Strong understanding of ETL/ELT design patterns , CDC (Change Data Capture) , and data governance best practices. Excellent communication skills and the ability to translate technical requirements into business outcomes.

The successful candidate for the Full Stack Developer position at U.S. Pharmacopeial Convention (USP) will have a demonstrated understanding of the organization's mission and a commitment to excellence through inclusive and equitable behaviors and practices. They should possess the ability to quickly build credibility with stakeholders. As a Full Stack Developer, you will be part of the Digital & Innovation group at USP, responsible for building innovative digital products using cutting-edge cloud technologies. Your role will be crucial in creating an amazing digital experience for customers. Your responsibilities will include building scalable applications and platforms using the latest cloud technologies, ensuring systems are regularly reviewed and upgraded based on governance principles and security policies. You will participate in code reviews, architecture discussions, and agile development processes to maintain high-quality, maintainable, and scalable code. Additionally, you will provide technical guidance and mentorship to junior developers and team members, as well as document and communicate technical designs, processes, and solutions to both technical and non-technical stakeholders. To qualify for this role, you should have a Bachelor's or Master's degree in Computer Science, Engineering, or a related field, along with 6-10 years of experience in software development with a focus on cloud computing. Strong knowledge of cloud platforms such as AWS, Azure, and Google Cloud, as well as services like compute, storage, networking, and security, is essential. Experience in leading and mentoring junior software developers, extensive knowledge of Java spring boot applications, and proficiency in programming languages like Python or Node.js are also required. Moreover, you should have experience with AWS/Azure services, containerization technologies like Docker and Kubernetes, front-end technologies like React.js/Node.js, and microservices. Familiarity with cloud architecture patterns, best practices, security principles, data pipelines, and ETL tools is a plus. Experience in leading initiatives related to continuous improvement or new technology implementations, strong analytical and problem-solving skills, and the ability to manage multiple projects and priorities in a fast-paced environment are also desirable attributes. Additional preferences include experience with scientific chemistry nomenclature, pharmaceutical datasets, knowledge graphs, and the ability to explain complex technical issues to a non-technical audience. Strong communication skills, the ability to work independently, make tough decisions, and prioritize tasks are essential for this role. As a Full Stack Developer at USP, you will have supervisory responsibilities and will be eligible for a comprehensive benefits package that includes company-paid time off, healthcare options, and retirement savings. USP is an independent scientific organization dedicated to developing quality standards for medicines, dietary supplements, and food ingredients in collaboration with global health and science authorities. The organization values inclusivity, mentorship, and professional growth, emphasizing Diversity, Equity, Inclusion, and Belonging in its mission to ensure quality in health and healthcare worldwide.,

As a Data Engineer2 at GoKwik, you will have the opportunity to closely collaborate with product managers, data scientists, business intelligence teams, and SDEs to develop and implement data-driven strategies. Your role will involve identifying, designing, and executing process improvements to enhance data models, architectures, pipelines, and applications. You will play a vital role in continuously optimizing data processes, overseeing data management, governance, security, and analysis to ensure data quality and security across all product verticals. Additionally, you will design, create, and deploy new data models and pipelines as necessary to achieve high performance, operational excellence, accuracy, and reliability in the system. Your responsibilities will include utilizing tools and technologies to establish a data architecture that supports new data initiatives and next-gen products. You will focus on building test-driven products and pipelines that are easily maintainable and reusable. Furthermore, you will design and construct an infrastructure for data extraction, transformation, and loading from various data sources, supporting the marketing and sales team. To excel in this role, you should possess a Bachelor's or Master's degree in Computer Science, Mathematics, or relevant computer programming training, along with a minimum of 4 years of experience in the Data Engineering field. Proficiency in SQL, relational databases, query authoring, data pipelines, architectures, and working with cross-functional teams in a dynamic environment is essential. Experience with Python, data pipeline tools, and AWS cloud services is also required. We are looking for individuals who are independent, resourceful, analytical, and adept at problem-solving. The ability to adapt to changing environments, excellent communication skills, and a collaborative mindset are crucial for success in this role. If you are passionate about tackling challenging problems at scale and making a significant impact within a dynamic and entrepreneurial setting, we welcome you to join our team at GoKwik.,

We are looking for someone who is enthusiastic to contribute to the implementation of a metadata-driven platform managing the full lifecycle of batch and streaming Big Data pipelines. This role involves applying ML and AI techniques in data management, such as anomaly detection for identifying and resolving data quality issues and data discovery. The platform facilitates the delivery of Visa's core data assets to both internal and external customers. You will provide Platform-as-a-Service offerings that are easy to consume, scalable, secure, and reliable using open source-based Cloud solutions for Big Data technologies. Working at the intersection of infrastructure and software engineering, you will design and deploy data and pipeline management frameworks using open-source components like Hadoop, Hive, Spark, HBase, Kafka streaming, and other Big Data technologies. Collaboration with various teams is essential to build and maintain innovative, reliable, secure, and cost-effective distributed solutions. Facilitating knowledge transfer to the Engineering and Operations team, you will work on technical challenges and process improvements with geographically distributed teams. Your responsibilities will include designing and implementing agile-innovative data pipeline and workflow management solutions that leverage technology advances for cost reduction, standardization, and commoditization. Driving the adoption of open standard toolsets to reduce complexity and support operational goals for increasing automation across the enterprise is a key aspect of this role. As a champion for the adoption of open infrastructure solutions that are fit for purpose, you will keep technology relevant. The role involves spending 80% of the time writing code in different languages, frameworks, and technology stacks. At Visa, your uniqueness is valued. Working here provides an opportunity to make a global impact, invest in your career growth, and be part of an inclusive and diverse workplace. Join our global team of disruptors, trailblazers, innovators, and risk-takers who are driving economic growth worldwide, moving the industry forward creatively, and engaging in meaningful work that brings financial literacy and digital commerce to millions of unbanked and underserved consumers. This position is hybrid, and the expectation of days in the office will be confirmed by your hiring manager. **Basic Qualifications**: - Minimum of 6 months of work experience or a bachelor's degree - Bachelor's degree in Computer Science, Computer Engineering, or a related field - Good understanding of data structures and algorithms - Good analytical and problem-solving skills **Preferred Qualifications**: - 1 or more years of work experience or an Advanced Degree (e.g., Masters) in Computer Science - Excellent programming skills with experience in at least one of the following: Python, Node.js, Java, Scala, GoLang - MVC (model-view-controller) for end-to-end development - Knowledge of SQL/NoSQL technology. Familiarity with Databases like Oracle, DB2, SQL Server, etc. - Proficiency in Unix-based operating systems and bash scripts - Strong communication skills, including clear and concise written and spoken communications with professional judgment - Team player with excellent interpersonal skills - Demonstrated ability to lead and navigate through ambiguity **Additional Information**:,

Position summary: We are seeking a Senior Software Development Engineer – Data Engineering with 3-5 years of experience to design, develop, and optimize data pipelines and analytics workflows using Snowflake, Databricks, and Apache Spark. The ideal candidate will have a strong background in big data processing, cloud data platforms, and performance optimization to enable scalable data-driven solutions. Key Responsibilities: Work with cloud-based data solutions (Azure, AWS, GCP). Implement data modeling and warehousing solutions. Developing and maintaining data pipelines for efficient data extraction, transformation, and loading (ETL) processes. Designing and optimizing data storage solutions, including data warehouses and data lakes. Ensuring data quality and integrity through data validation, cleansing, and error handling. Collaborating with data analysts, data architects, and software engineers to understand data requirements and deliver relevant data sets (e.g., for business intelligence). Implementing data security measures and access controls to protect sensitive information. Monitor and troubleshoot issues in data pipelines, notebooks, and SQL queries to ensure seamless data processing. Develop and maintain Power BI dashboards and reports. Work with DAX and Power Query to manipulate and transform data. Basic Qualifications Bachelor’s or master’s degree in computer science or data science 3-5 years of experience in data engineering, big data processing, and cloud-based data platforms. Proficient in SQL, Python, or Scala for data manipulation and processing. Proficient in developing data pipelines using Azure Synapse, Azure Data Factory, Microsoft Fabric. Experience with Apache Spark, Databricks and Snowflake is highly beneficial for handling big data and cloud-based analytics solutions. Preferred Qualifications Knowledge of streaming data processing (Apache Kafka, Flink, Kinesis, Pub/Sub). Experience in BI and analytics tools (Tableau, Power BI, Looker). Familiarity with data observability tools (Monte Carlo, Great Expectations). Contributions to open-source data engineering projects.

Role & responsibilities Design and implement AI agent workflows. Develop end-to-end intelligent pipelines and multi-agent systems (e.g., LangGraph/LangChain workflows) that coordinate multiple LLM-powered agents to solve complex tasks. Create graph-based or state-machine architectures for AI agents, chaining prompts and tools as needed. Build and fine-tune generative models. Develop, train, and fine-tune advanced generative models (transformers, diffusion models, VAEs, GANs, etc.) on domain-specific data. Deploy and optimize foundation models (such as GPT, LLaMA, Mistral) in production, adapting them to our use cases through prompt engineering and supervised fine-tuning. Develop data pipelines. Build robust data collection, preprocessing, and synthetic data generation pipelines to feed training and inference workflows. Implement data cleansing, annotation, and augmentation processes to ensure high-quality inputs for model training and evaluation. Implement LLM-based agents and automation. Integrate generative AI agents (e.g., chatbots, AI copilots, content generators) into business processes to automate data processing and decision-making tasks. Use Retrieval-Augmented Generation (RAG) pipelines and external knowledge sources to enhance agent capabilities. Leverage multimodal inputs when applicable. Optimize performance and safety. Continuously evaluate and improve model/system performance. Use GenAI-specific benchmarks and metrics (e.g., BLEU, ROUGE, TruthfulQA) to assess results, and iterate to optimize accuracy, latency, and resource efficiency. Implement safeguards and monitoring to mitigate issues like bias, hallucination, or inappropriate outputs. Collaborate and document. Work closely with product managers, engineers, and other stakeholders to gather requirements and integrate AI solutions into production systems. Document data workflows, model architectures, and experimentation results. Maintain code and tooling (prompt libraries, model registries) to ensure reproducibility and knowledge sharing. Education: Bachelors or Master’s degree in Computer Science, Data Science, Artificial Intelligence. Programming proficiency: Expert-level skills in Python experience in machine learning and deep learning frameworks (PyTorch, TensorFlow). Generative model expertise: Demonstrated ability to build, fine-tune, and deploy large-scale generative models Familiarity with transformer architectures and generative techniques (LLMs, diffusion models, GANs) Experience working with model repositories and fine-tuning frameworks (Hugging Face, etc.). LLM and agent frameworks: Strong understanding of LLM-based systems and agent-oriented AI patterns. Experience with frameworks like LangGraph/LangChain or similar multi-agent platforms. AI integration and MLOps: Experience integrating AI components with existing systems via APIs and services. . Proficiency in retrieval-augmented generation (RAG) setups, vector databases,Familiarity with machine learning deployment and MLOps tools (Docker, Kubernetes, MLflow, KServe, etc.) Familiarity with GenAI tools: Hands-on experience with state-of-the-art GenAI models and APIs (OpenAI GPT, Anthropic, Claude, etc.) and with popular libraries (Hugging Face Transformers, LangChain, etc.). Awareness of the current GenAI tooling ecosystem and best practices.

As a Mid Databricks Engineer, you will play a pivotal role in designing, implementing, and optimizing data processing pipelines and analytics solutions on the Databricks platform. You will collaborate closely with cross-functional teams to understand business requirements, architect scalable solutions, and ensure the reliability and performance of our data infrastructure. This role requires deep expertise in Databricks, strong programming skills, and a passion for solving complex engineering challenges. What you'll do : - Design and develop data processing pipelines and analytics solutions using Databricks. - Architect scalable and efficient data models and storage solutions on the Databricks platform. - Collaborate with architects and other teams to migrate current solution to use Databricks. - Optimize performance and reliability of Databricks clusters and jobs to meet SLAs and business requirements. - Use best practices for data governance, security, and compliance on the Databricks platform. - Mentor junior engineers and provide technical guidance. - Stay current with emerging technologies and trends in data engineering and analytics to drive continuous improvement. You'll be expected to have : - Bachelor's or master's degree in computer science, Engineering, or a related field. - 5 to 8 years of overall experience and 2+ years of experience designing and implementing data solutions on the Databricks platform. - Proficiency in programming languages such as Python, Scala, or SQL. - Strong understanding of distributed computing principles and experience with big data technologies such as Apache Spark. - Experience with cloud platforms such as AWS, Azure, or GCP, and their associated data services. - Proven track record of delivering scalable and reliable data solutions in a fast-paced environment. - Excellent problem-solving skills and attention to detail. - Strong communication and collaboration skills with the ability to work effectively in cross-functional teams. - Good to have experience with containerization technologies such as Docker and Kubernetes. - Knowledge of DevOps practices for automated deployment and monitoring of data pipelines.

Key Responsibilities Design, develop, and maintain scalable ETL workflows using Cloud Data Fusion and Apache Airflow . Configure and manage various data connectors (e.g., Cloud Storage, Pub/Sub, JDBC, SaaS APIs) for batch and streaming data ingestion. Implement data transformations, cleansing, and enrichment logic in Python (and SQL) to meet analytic requirements. Optimize BigQuery data models (fact/dimension tables, partitioning, clustering) for performance and cost-efficiency. Monitor, troubleshoot, and tune pipeline performance; implement robust error-handling and alerting mechanisms. Collaborate with data analysts, BI developers, and architects to understand data requirements and deliver accurate datasets. Maintain documentation for data pipelines, schemas, and operational runbooks. Ensure data security and governance best practices are followed across the data lifecycle. Minimum Qualifications 3+ years of hands-on experience in data engineering, with a focus on cloud-native ETL. Proven expertise with Google Cloud Data Fusion , including pipeline authoring and custom plugin development. Solid experience building and orchestrating pipelines in Apache Airflow (DAG design, operators, hooks). Strong Python programming skills for data manipulation and automation. Deep understanding of BigQuery : schema design, SQL scripting, performance tuning, and cost management. Familiarity with additional GCP services: Cloud Storage, Pub/Sub, Dataflow, and IAM. Experience with version control (Git), CI/CD pipelines, and DevOps practices for data projects. Excellent problem-solving skills, attention to detail, and the ability to work independently in a fast-paced environment. Immediate availability to join. Preferred (Nice-to-Have) Experience with other data integration tools (e.g., Dataflow, Talend, Informatica). Knowledge of containerization (Docker, Kubernetes) for scalable data workloads. Familiarity with streaming frameworks (Apache Beam, Spark Streaming). Background in data modeling methodologies (Star/Snowflake schemas). Exposure to metadata management, data cataloguing, and data governance frameworks.

Position: Senior Data Engineer Location: Bangalore, India About Dodge Dodge Construction Network exists to deliver the comprehensive data and connections the construction industry needs to build thriving communities. Our legacy is deeply rooted in empowering our customers with transformative insights, igniting their journey towards unparalleled business expansion and success. We serve decision-makers who seek reliable growth and who value relationships built on trust and quality. By combining our proprietary data with cutting-edge software, we deliver to our customers the essential intelligence needed to excel within their respective landscapes. We propel the construction industry forward by transforming data into tangible guidance, driving unparalleled advancement. Dodge is the catalyst for modern construction. https://www.construction.com/ About Symphony Technology Group (STG) STG is a Silicon Valley (California) based private equity firm that has a long and successful track record of transforming high potential software and software-enabled services companies, as well as insights-oriented companies into definitive market leaders. The firm brings expertise, flexibility, and resources to build strategic value and unlock the potential of innovative companies. Partnering to build customer-centric, market winning portfolio companies, STG creates sustainable foundations for growth that bring value to all existing and future stakeholders. The firm is dedicated to transforming and building outstanding technology companies in partnership with world class management teams. With over $5.0 billion in assets under management, including a recently raised $2.0 billion fund. STGs expansive portfolio has consisted of more than 30 global companies. STG Labs is the incubation center for many of STGs portfolio companies, building their engineering, professional services, and support delivery teams in India. STG Labs offers an entrepreneurial start-up environment for software and AI engineers, data scientists and analysts, project and product managers and provides a unique opportunity to work directly for a software or technology company. Based in Bangalore, STG Labs supports hybrid working. https://stg.com Roles and Responsibilities Design, build, and maintain scalable data pipelines and ETL processes leveraging AWS services. Collaborate closely with data architects, business analysts, and DevOps teams to translate business requirements into technical data solutions. Apply SDLC best practices, including planning, coding standards, code reviews, testing, and deployment. Automate workflows and optimize data pipelines for efficiency, performance, and reliability. Implement monitoring and logging to ensure the health and performance of data systems. Ensure data security and compliance through adherence to industry and internal standards. Participate actively in agile development processes and contribute to sprint planning, stand-ups, retrospectives, and documentation efforts. Qualifications Hands-on working knowledge and experience is required in: Data Structures Memory Management Basic Algos (Search, Sort, etc) Hands-on working knowledge and experience is preferred in: Memory Management Algorithms: Search, Sort, etc. AWS Data Services: Glue, EMR, Kinesis, Lambda, Athena, Redshift, S3 Scripting & Programming Languages: Python, Bash, SQL Version Control & CI/CD Tools: Git, Jenkins, Bitbucket Database Systems & Data Engineering: Data modeling, data warehousing principles Infrastructure as Code (IaC): Terraform, CloudFormation Containerization & Orchestration: Docker, Kubernetes Certifications Preferred : AWS Certifications (Data Analytics Specialty, Solutions Architect Associate).(Preferred Skill).

Role & Apache Nifi 5+ years of hands-on experience with Apache NiFi, including developing, managing, and optimizing complex data flows in production environments. Proven experience with Cloudera NiFi (CDP Data Flow) in enterprise environments, including integration with Cloudera Manager. Experience migrating NiFi flows across major version upgrades with strong understanding of backward compatibility Strong proficiency in Groovy scripting, used for ExecuteScript and InvokeScriptedProcessor processors. Solid understanding of SSH and SFTP protocols, including authentication schemes (key-based, password), session negotiation, and file permissions handling in NiFi processors (e.g., ListSFTP, FetchSFTP, PutSFTP). Good grasp of data encryption mechanisms, key management, and secure flowfile handling using processors like EncryptContent. Experience integrating NiFi with MongoDB, including reading/writing documents via processors like GetMongo, PutMongo, and QueryMongo. Experience working with Apache Kafka, including producing and consuming from Kafka topics using NiFi (PublishKafka, ConsumeKafka), and handling schema evolution with Confluent Schema Registry. Strong knowledge of Red Hat Enterprise Linux (RHEL) environments, including systemd services, filesystem permissions, log rotation, and resource tuning for JVM-based applications like NiFi. NiFi-Specific Technical Requirements: In-depth knowledge of NiFi flow design principles, including proper use of queues, back pressure, prioritizers, and connection tuning. Mastery of controller services, including SSLContextService, DBCPConnectionPool, and RecordReader/RecordWriter services. Experience with Record-based processing using Avro, JSON, CSV schemas and Record processors like ConvertRecord, QueryRecord, and LookupRecord. Ability to debug and optimize NiFi flows using Data Provenance, bulletins, and log analysis. Familiarity with custom processor development in Java/Groovy (optional but preferred). Experience setting up secure NiFi clusters, configuring user authentication (LDAP, OIDC), TLS certificates, and access policies. Proficiency in parameter contexts, variable registry, and flow versioning using NiFi Registry. Understanding of Zero-Master clustering model, node coordination, and site-to-site protocol. Experience deploying and monitoring NiFi in high-availability, production-grade environments, including using Prometheus/Grafana or Cloudera Manager for metrics and alerting. Preferred Qualifications: Experience working in regulated or secure environments, with strict data handling and audit requirements. Familiarity with DevOps workflows, including version-controlled flow templates (JSON/XML), CI/CD integration for NiFi Registry, and automated deployment strategies. Strong written and verbal communication skills, with ability to document flows and onboard other engineers. responsibilities Preferred candidate profile

Job Role: Snowflake / SQL Architect Duration: 6 months contract Job Description: • Architect and manage scalable data solutions using Snowflake and advanced SQL, optimizing performance for analytics and reporting. • Design and implement data pipelines, data warehouses, and data lakes, ensuring efficient data ingestion and transformation. • Develop best practices for data security, access control, and compliance within cloud-based data environmentsCollaborate with cross-functional teams to understand business needs and translate them into robust data architectures. • Evaluate and integrate third-party tools and technologies to enhance the Snowflake ecosystem and overall data strategy.

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-thinking approach to integrating cutting-AI technologies, particularly LLM Mesh for Generative AI applications. If you are passionate about building scalable data solutions and are eager to explore the cutting edge of AI, we encourage you to apply. Key Responsibilities : - Dataiku Leadership : Drive data engineering initiatives with a strong emphasis on leveraging Dataiku capabilities for data preparation, analysis, visualization, and the deployment of data solutions. - Data Pipeline Development : Design, develop, and optimize robust and scalable data pipelines to support various business intelligence and advanced analytics projects. This includes developing and maintaining ETL/ELT processes to automate data extraction, transformation, and loading from diverse sources. - Data Modeling & Architecture : Apply expertise in data modeling techniques to design efficient and scalable database structures, ensuring data integrity and optimal performance. - ETL/ELT Expertise : Implement and manage ETL processes and tools to ensure efficient and reliable data flow, maintaining high data quality and accessibility. - Gen AI Integration : Explore and implement solutions leveraging LLM Mesh for Generative AI applications, contributing to the development of innovative AI-powered features. - Programming & Scripting : Utilize programming languages such as Python and SQL for data manipulation, analysis, automation, and the development of custom data solutions. - Cloud Platform Deployment : Deploy and manage scalable data solutions on cloud platforms such as AWS or Azure, leveraging their respective services for optimal performance and cost-efficiency. - Data Quality & Governance : Ensure seamless integration of data sources, maintaining high data quality, consistency, and accessibility across all data assets. Implement data governance best practices. - Collaboration & Mentorship : Collaborate closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver impactful solutions. Potentially mentor junior team members. - Performance Optimization : Continuously monitor and optimize the performance of data pipelines and data systems. Required Skills & Experience : - Proficiency in Dataiku : Demonstrable expertise in Dataiku for data preparation, analysis, visualization, and building end-to-end data pipelines and applications. - Expertise in Data Modeling : Strong understanding and practical experience in various data modeling techniques (e.g., dimensional modeling, Kimball, Inmon) to design efficient and scalable database structures. - ETL/ELT Processes & Tools : Extensive experience with ETL/ELT processes and a proven track record of using various ETL tools (e.g., Dataiku's built-in capabilities, Apache Airflow, Talend, SSIS, etc.). - Familiarity with LLM Mesh : Familiarity with LLM Mesh or similar frameworks for Gen AI applications, understanding its concepts and potential for integration. - Programming Languages : Strong proficiency in Python for data manipulation, scripting, and developing data solutions. Solid command of SQL for complex querying, data analysis, and database interactions. - Cloud Platforms : Knowledge and hands-on experience with at least one major cloud platform (AWS or Azure) for deploying and managing scalable data solutions (e.g., S3, EC2, Azure Data Lake, Azure Synapse, etc.). - Gen AI Concepts : Basic understanding of Generative AI concepts and their potential applications in data engineering. - Problem-Solving : Excellent analytical and problem-solving skills with a keen eye for detail. - Communication : Strong communication and interpersonal skills to collaborate effectively with cross-functional teams. Bonus Points (Nice to Have) : - Experience with other big data technologies (e.g., Spark, Hadoop, Snowflake). - Familiarity with data governance and data security best practices. - Experience with MLOps principles and tools. - Contributions to open-source projects related to data engineering or AI. Education : Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related quantitative field.

An Opportunity to Work with One of India's Leading Credit Card Tech Innovators BOBCARD (A Bank of Baroda Subsidiary) Educational Qualifications: Bachelors/Masters degree in Computer Science, Data Engineering, Information Technology, or a related field . Experience Requirement: 5+ years Applicants should possess the following attributes: Technical Skills: Should have knowledge database design and development skills in Non- Relational Databases, SQL /. PL SQL, Oracle/MS SQL/MySQL Following GCP skillsets are necessary - BigQuery Storage, BigQuery ML, Cloud Data Fusion, Looker Studio Pro, Looker ML, DataStream Google Cloud platform especially Big Query Dataflow Experience with Java and Python and Google Cloud SDK API Scripting Experience in writing ETL/ELT Good Knowledge of Data modelling, Dimensional Modelling and Data Visualisation Proficient in SQL, Excel, Data Analytics, Data integration, Metadata Management Experience in Implementing complete project on GCP -AI / ML Experience Tableau / Power BI / Qlik Sense/ Looker Studio Use the full breadth of capabilities within Google Looker Studio to develop dashboards, Visual components, data Reports and analytical tools Proficiency in Oracle R, Python will be an added advantage Presenting complex data in visually appealing and insightful ways to aid story- telling and communication Strong knowledge of SDLC methodologies Agile/Scrum / Solid Principles Good understanding of Database Design Concepts Certification of Analytics with AI/ML would be an added advantage. Role & Responsibilities This position would include the mentioned set of responsibilities but not limited to: The candidate should be self-motivated, multi-tasker, and demonstrated team-player. Responsible for the managing the product development and maintenance in IT team of the organisation. Must have worked with large-scale applications and frameworks Learn and enhance own skillset and of team. Stay up to date to latest technology. Writing clean, high-quality, high-performance, maintainable code. Coordinate cross-functionally to ensure project meets business objectives and compliance standards Ensuring high performance of the application. Develop, test, and deploy fast and scalable system Designing and maintenance of fully functional applications and databases. Timely deployment of web apps on the cloud and on-prim/monolithic. Understanding of server management and cloud-based infrastructure. Establishment and integration of development tools as required Identification of application issues when deploying the apps App deployment on the cloud along with solving debugging issues Coding architecture for frontend and backend Building interactive consumer data Testing applications and fixing bugs, along with security and data protection features Solve Technical Challenges Cooperating with the back-end developer in the process of building the Cloud based Analytics platform Develop and support Analytics Platform Interested Professionals can also apply via our careers page: https://bobcard.turbohire.co/job/publicjobs/9IJ04bfVo3kTnyMKczuuhM3wpm81HcOhip9k0uecVaDF1y6XWQ9jRsFiuPYS7dQw

Full time with top German MNC for location Bangalore - Experience on SCALA is a must Job Overview: To work on development, monitoring and maintenance of Data pipelines across clusters. Primary responsibilities: Develop, Monitor and Maintain data pipeline for various plants. Create and maintain optimal data pipeline architecture. Assemble large, complex data sets that meet functional / non-functional business requirements. Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability. Work with stakeholders including the Data officers and stewards to assist with data-related technical issues and support their data infrastructure needs. Work on incidents highlighted by the data officers. Incident diagnosis, routing, evaluation & resolution. Analyze the root cause of incidents. Create incident closure report. Qualifications Qualifications Bachelors degree in Computer Science, Electronics & Communication Engineering, a related technical field, or equivalent practical experience. 6-8 years of experience in Spark, Scala software development. Experience in large-scale software development. Excellent software engineering skills (i.e., data structures, algorithms, software design). Excellent problem-solving, investigative, and troubleshooting skills. Experience in Kafka is mandatory Additional Information Skills Self-starter and empowered professional with strong execution and project management capabilities Ability to collaborate effectively, well developed inter personal relationships with all levels in the organization and outside contacts. Outstanding written and verbal communication skills. High Collaboration & a perseverance to drive performance & change Additional information Key Competencies- Distributed computing systems Experience with CI/CD tools such as Jenkins or Github Actions Experience with Python programming Working knowledge of Docker & Kubernetes Experience in developing data pipelines using spark & scala. Experience in debugging pipeline issues. Experience in writing python and shell scripts. In-Depth Knowledge of SQL and Other Database Solutions Having a strong understanding of Apache Hadoop-based analytics Hands on experience on InteliJ, Github /Bitbucket, HUE.

About Agoda Agoda is an online travel booking platform that offers accommodations, flights, and more to travelers worldwide. With a global network of 4.7M hotels and holiday properties, as well as flights, activities, and more, we are dedicated to connecting travelers with seamless travel experiences. As part of Booking Holdings and based in Asia, our team of 7,100+ employees from 95+ nationalities across 27 markets creates a work environment that thrives on diversity, creativity, and collaboration. At Agoda, we foster a culture of innovation through experimentation and ownership, allowing our customers to explore and enjoy the world. Our Purpose: Bridging the World Through Travel We believe that travel enriches lives by providing opportunities to learn, experience, and appreciate the beauty of our world. By bringing people and cultures closer together, travel promotes empathy, understanding, and happiness. The Data Team at Agoda The Data department at Agoda is responsible for overseeing all data-related requirements within the company. Our primary objective is to enhance the utilization of data through innovative approaches and the implementation of robust resources such as operational and analytical databases, queue systems, BI tools, and data science technology. We recruit talented individuals from diverse backgrounds globally to tackle this challenge, providing them with the necessary knowledge and tools for personal growth and success while upholding our company's values of diversity and experimentation. The Data team at Agoda plays a crucial role in supporting business users, product managers, engineers, and others in their decision-making processes. We are committed to improving the search experience for our customers by delivering faster results and ensuring protection against fraudulent activities. The abundance of data available to us presents both a challenge and a reward, driving our passion for excellence within the Data department. The Opportunity As a senior data pipeline engineer at Agoda, you will be working on distributed systems that span multiple data centers, thousands of servers, and process hundreds of billions of messages daily. Ensuring data quality, integrity, and accuracy is fundamental to our operations. You will be involved in designing scalable systems to handle the increasing volume of data, including auditing and monitoring functionalities. This role provides you with the opportunity to lead projects with a small team, enhancing your ownership and leadership skills. You will tackle complex problems related to managing and interpreting large datasets, such as schema registry, real-time data-ingestion, cross-data center replication, data enrichment, storage, and analytics. In This Role, You'll Get to - Build, administer, and scale data pipelines processing hundreds of billions of messages daily across multiple data centers - Develop and enhance existing frameworks used by teams throughout Agoda to contribute messages to the data pipeline - Manage data ingestion into various systems (Hadoop, ElasticSearch, other Distributed Systems) - Create tools to monitor high data accuracy SLAs for the data pipeline - Explore new technologies to improve data quality, processes, and flow - Develop high-quality software through design reviews, code reviews, and test-driven development What You'll Need To Succeed - Bachelors degree in Computer Science, Information Systems, Computer Engineering, or a related field - 8+ years of industry experience, preferably in a tech company - Strong knowledge of data architecture principles - Experience in debugging production issues - Proficient in coding and building purpose-driven, scalable, well-tested, and maintainable systems - Detail-oriented with a focus on considering all outcomes of decisions - Excellent communication skills in technical English, both verbally and in writing - Proficiency in multiple programming languages (e.g., Golang, Java, Scala, Python, C#) - Good understanding of Kafka and experience as a Kafka Administrator - Experience with data ingestion from Kafka into Hadoop, ElasticSearch, and other Distributed Systems - Strong systems administration skills in Linux - Previous involvement in or contribution to Open Source Projects Equal Opportunity Employer Agoda is an equal opportunity employer. We value diversity and welcome applications from individuals with a variety of backgrounds and experiences. We will retain your application for future vacancies and allow you to request the removal of your details if desired. For more information, please refer to our privacy policy. Note: Agoda does not accept third-party resumes. Kindly refrain from sending resumes to our jobs alias, Agoda employees, or any other organizational location. Agoda will not be liable for any fees associated with unsolicited resumes.,

- Own and enhance the full lifecycle of our data pipelines: from ingestion to transformation to delivery - Design, implement, and scale robust data infrastructure in a cloud environment - Write high-quality, production-grade code using Python

Job Title: Senior Data Engineer Python, PySpark, AWS Experience Required: 6 to 12 Years Location: Hyderabad Job Type: Full Time / Permanent Job Description: We are looking for a passionate and experienced Senior Data Engineer to join our team in Hyderabad . The ideal candidate should have a strong background in data engineering on AWS , with hands-on expertise in Python, PySpark, and AWS services to build and maintain scalable data pipelines and ETL workflows. Mandatory Skills: Data Engineering Python PySpark AWS Services (S3, Glue, Lambda, Redshift, RDS, EC2, Data Pipeline) Key Responsibilities: Design and implement robust, scalable data pipelines using PySpark , AWS Glue , and AWS Data Pipeline . Develop and maintain efficient ETL workflows to handle large-scale data processing. Automate data workflows and job orchestration using AWS Data Pipeline Ensure smooth data integration across services like S3 , Redshift , and RDS . Optimize data processing for performance and cost efficiency on the cloud. Work with various file formats like CSV, Parquet, and Avro. Technical Requirements: 8+ years of experience in Data Engineering , particularly in cloud-based environments . Proficient in Python and PySpark for data transformation and manipulation. Strong experience with AWS Glue for ETL development, Data Catalog, and Crawlers. Solid knowledge of SQL for querying structured and semi-structured data. Familiar with Data Lake architectures , Amazon EMR , and Kinesis . Experience with Docker , Git , and CI/CD pipelines for deployment and versioning Interested Candidates can also share their CV at akanksha.s@esolglobal.com

When our values align, there's no limit to what we can achieve. At Parexel, we all share the same goal - to improve the world's health. From clinical trials to regulatory, consulting, and market access, every clinical development solution we provide is underpinned by something special - a deep conviction in what we do. Each of us, no matter what we do at Parexel, contributes to the development of a therapy that ultimately will benefit a patient. We take our work personally, we do it with empathy and we're committed to making a difference. Required Skills: With a focus on automating testing and DevOps, use Microsoft Azure data PaaS services, design, build, modify, and support data pipelines leveraging DataBricks and PowerBI in a medallion architecture setting. Working experience of Python. Automate the running of unit and integration tests on all created code – Create and run unit and integration tests throughout the development lifecycle. Support and Troubleshooting – Assist the Operations Team with any environmental issues that arise during application deployment in the Development, QA, Staging, and Production environments. If necessary, create prototypes to validate proposed ideas and solicit input from stakeholders. Excellent grasp of and expertise with test-driven development and continuous integration processes. Analysis and Design – Converts high-level design to low-level design and implements it. Collaborate with Team Leads to define/clarify business requirements, estimate development costs, and finalize work plans. Benchmark application code proactively to prevent performance and scalability concerns. Collaborate with the Quality Assurance Team on issue reporting, resolution, and change management. Assist other teams in resolving issues that may develop as a result of applications or the integration of multiple components. Education: Bachelor's Degree or equivalent combination of education and experience.

Were looking for a Senior Data Analyst to join our data-driven team at an ad-tech company that thrives on turning complexity into clarity. Our analysts play a critical role in transforming raw, noisy data into accurate, actionable signals that drive real-time decision-making and long-term strategy. Youll work closely with product, engineering, and business teams to uncover insights, shape KPIs, and guide performance optimization. Responsibilities: Analyze large-scale datasets from multiple sources to uncover actionable insights and drive business impact. Design, monitor, and maintain key performance indicators (KPIs) across ad delivery, bidding, and monetization systems. Partner with product, engineering, and operations teams to define metrics, run deep-dive analyses, and influence strategic decisions. Develop and maintain dashboards, automated reports, and data pipelines to ensure data accessibility and accuracy. Lead investigative analysis of anomalies or unexpected trends in campaign performance, traffic quality, or platform behavior. Requirements BA / BSc in Industrial Engineering and Management / Information Systems Engineering / Economics / Statistics / Mathematics / similar background. 3+ years of experience in Data Analysis and interpretation (Marketing/ Business/ Product). High proficiency in SQL. Experience with data visualization of large data sets using BI systems (Qlik Sense, Sisense, Tableau, Looker, etc.). Experience working with data warehouse/data lake tools like Athena / Redshift / Snowflake /BigQuery. Knowledge of Python - An advantage. Experience building ETL processes An advantage. Fluent in English both written and spoken - Must

About the job : - As a Mid Databricks Engineer, you will play a pivotal role in designing, implementing, and optimizing data processing pipelines and analytics solutions on the Databricks platform. - You will collaborate closely with cross-functional teams to understand business requirements, architect scalable solutions, and ensure the reliability and performance of our data infrastructure. - This role requires deep expertise in Databricks, strong programming skills, and a passion for solving complex engineering challenges. What You'll Do : - Design and develop data processing pipelines and analytics solutions using Databricks. - Architect scalable and efficient data models and storage solutions on the Databricks platform. - Collaborate with architects and other teams to migrate current solution to use Databricks. - Optimize performance and reliability of Databricks clusters and jobs to meet SLAs and business requirements. - Use best practices for data governance, security, and compliance on the Databricks platform. - Mentor junior engineers and provide technical guidance. - Stay current with emerging technologies and trends in data engineering and analytics to drive continuous improvement. You'll Be Expected To Have : - Bachelor's or Master's degree in Computer Science, Engineering, or a related field. - 3 to 6 years of overall experience and 2+ years of experience designing and implementing data solutions on the Databricks platform. - Proficiency in programming languages such as Python, Scala, or SQL. - Strong understanding of distributed computing principles and experience with big data technologies such as Apache Spark. - Experience with cloud platforms such as AWS, Azure, or GCP, and their associated data services. - Proven track record of delivering scalable and reliable data solutions in a fast-paced environment. - Excellent problem-solving skills and attention to detail. - Strong communication and collaboration skills with the ability to work effectively in cross-functional teams. - Good to have experience with containerization technologies such as Docker and Kubernetes. - Knowledge of DevOps practices for automated deployment and monitoring of data pipelines.

Role Responsibilities : - Design and implement data pipelines using MS Fabric. - Develop data models to support business intelligence and analytics. - Manage and optimize ETL processes for data extraction, transformation, and loading. - Collaborate with cross-functional teams to gather and define data requirements. - Ensure data quality and integrity in all data processes. - Implement best practices for data management, storage, and processing. - Conduct performance tuning for data storage and retrieval for enhanced efficiency. - Generate and maintain documentation for data architecture and data flow. - Participate in troubleshooting data-related issues and implement solutions. - Monitor and optimize cloud-based solutions for scalability and resource efficiency. - Evaluate emerging technologies and tools for potential incorporation in projects. - Assist in designing data governance frameworks and policies. - Provide technical guidance and support to junior data engineers. - Participate in code reviews and ensure adherence to coding standards. - Stay updated with industry trends and best practices in data engineering. Qualifications : - 8+ years of experience in data engineering roles. - Strong expertise in MS Fabric and related technologies. - Proficiency in SQL and relational database management systems. - Experience with data warehousing solutions and data modeling. - Hands-on experience in ETL tools and processes. - Knowledge of cloud computing platforms (Azure, AWS, GCP). - Familiarity with Python or similar programming languages. - Ability to communicate complex concepts clearly to non-technical stakeholders. - Experience in implementing data quality measures and data governance. - Strong problem-solving skills and attention to detail. - Ability to work independently in a remote environment. - Experience with data visualization tools is a plus. - Excellent analytical and organizational skills. - Bachelor's degree in Computer Science, Engineering, or related field. - Experience in Agile methodologies and project management.

About the job : Role : Microsoft Fabric Data Engineer Experience : 6+ years as Azure Data Engineer including at least 1 E2E Implementation in Microsoft Fabric. Responsibilities : - Lead the design and implementation of Microsoft Fabric-centric data platforms and data warehouses. - Develop and optimize ETL/ELT processes within the Microsoft Azure ecosystem, effectively utilizing relevant Fabric solutions. - Ensure data integrity, quality, and governance throughout Microsoft Fabric environment. - Collaborate with stakeholders to translate business needs into actionable data solutions. - Troubleshoot and optimize existing Fabric implementations for enhanced performance. Skills : - Solid foundational knowledge in data warehousing, ETL/ELT processes, and data modeling (dimensional, normalized). - Design and implement scalable and efficient data pipelines using Data Factory (Data Pipeline, Data Flow Gen 2 etc) in Fabric, Pyspark notebooks, Spark SQL, and Python. This includes data ingestion, data transformation, and data loading processes. - Experience ingesting data from SAP systems like SAP ECC/S4HANA/SAP BW etc will be a plus. - Nice to have ability to develop dashboards or reports using tools like Power BI. Coding Fluency : - Proficiency in SQL, Python, or other languages for data scripting, transformation, and automation.