627 Mapreduce Jobs - Page 16

Job Requirements Job Requirements Role/ Job Title: Senior Data Engineer Business: New Age Function/ Department: Data & Analytics Place of Work: Mumbai/Bangalore Roles & Responsibilities 'Minimum 6 years of Data Engineering experience and 3 years in large scale Data Lake ecosystem Proven expertise in SQL, Spark Python, Scala, Hadoop ecosystem, Have worked on multiple TBs/PBs of data volume from ingestion to consumption Work with business stakeholders to identify and document high impact business problems and potential solutions First-hand experience with the complete software development life cycle including requirement analysis, design, development, deployment, and support Advanced understanding of Data Lake/Lakehouse architecture and experience/exposure to Hadoop (cloudera,hortonworks) and AWS Work on end-to-end data lifecycle from Data Ingestion, Data Transformation and Data Consumption layer. Versed with API and its usability A suitable candidate will also be proficient Spark, Spark Streaming, AWS, and EMR A suitable candidate will also demonstrate machine learning experience and experience with big data infrastructure inclusive of MapReduce, Hive, HDFS, YARN, HBase, Oozie, etc. The candidate will additionally demonstrate substantial experience and a deep knowledge of data mining techniques, relational, and non-relational databases. Advanced skills in technical debugging of the architecture in case of issues Creating Technical Design Documentation (HLD/LLD) of the projects/pipelines Secondary Responsibilities 'Ability to work independently and handle your own development effort. Excellent oral and written communication skills Learn and use internally available analytic technologies Identify key performance indicators and establish strategies on how to deliver on these key points for analysis solutions Use educational background in data engineering and perform data mining analysis Work with BI analysts/engineers to create prototypes, implementing traditional classifiers and determiners, predictive and regressive analysis points Engage in the delivery and presentation of solutions Managerial & Leadership Responsibilities 'Lead moderately complex initiatives within Technology and contribute to large scale data processing framework initiatives related to enterprise strategy deliverables Build and maintain optimized and highly available data pipelines that facilitate deeper analysis and reporting Review and analyze moderately complex business, operational or technical challenges that require an in-depth evaluation of variable factors Oversee the data integration work, including integrating a data model with datalake, maintaining a data warehouse and analytics environment, and writing scripts for data integration and analysis Resolve moderately complex issues and lead teams to meet data engineering deliverables while leveraging solid understanding of data information policies, procedures and compliance requirements Collaborate and consult with colleagues and managers to resolve data engineering issues and achieve strategic goals Key Success Metrics 'Ensure timely deliverables. Spot Data fixes. Lead technical aspects of the projects. Error free deliverables.

Job Requirements Job Requirements Role/ Job Title: Data Engineer Business: New Age Function/ Department: Data & Analytics Place of Work: Mumbai Roles & Responsibilities 'Minimum 2-3 years of Data Engineering experience Proven experience in in SQL, Spark, Hadoop ecosystem Have worked on multiple TBsof data volume from ingestion to consumption Work with business stakeholders to identify and document high impact business problems and potential solutions Good understanding of Data Lake/Lakehouse architecture and experience/exposure to Hadoop (cloudera, hortonworks) and/or AWS Work on end-to-end data lifecycle from Data Ingestion, Data Transformation and Data Consumption layer. Versed with API and its usability A suitable candidate will also be proficient in Spark, Spark Streaming, hive, SQLs A suitable candidate will also demonstrate experience with big data infrastructure inclusive of MapReduce, Hive, HDFS, YARN, HBase, Oozie, etc. The candidate will additionally demonstrate substantial experience and a deep knowledge of relational databases. Good skills in technical debugging of the code in case of issues. Also, working with git for code versioning Creating Technical Design Documentation of the projects/pipelines Secondary Responsibilities 'Ability to work independently and handle your own development effort. Excellent oral and written communication skills Learn and use internally available analytic technologies Identify key performance indicators and create educational/deliverables path to achieve the same Use educational background in data engineering and perform data mining analysis Work with BI analysts/engineers to create prototypes Engage in the delivery and presentation of solutions Key Success Metrics 'Ensure timely deliverables. Write and maintain excellent code quality Spot Data fixes. Solutionize technical aspects of the projects. Error free deliverables.

Data Software Engineer Job Description: 1. 5-12 Years of in Big Data & Data related technology experience 2. Expert level understanding of distributed computing principles 3. Expert level knowledge and experience in Apache Spark 4. Hands on programming with Python 5. Proficiency with Hadoop v2, Map Reduce, HDFS, Sqoop 6. Experience with building stream-processing systems, using technologies such as Apache Storm or Spark-Streaming 7. Experience with messaging systems, such as Kafka or RabbitMQ 8. Good understanding of Big Data querying tools, such as Hive, and Impala 9. Experience with integration of data from multiple data sources such as RDBMS (SQL Server, Oracle), ERP, Files 10. Good understanding of SQL queries, joins, stored procedures, relational schemas 11. Experience with NoSQL databases, such as HBase, Cassandra, MongoDB 12. Knowledge of ETL techniques and frameworks 13. Performance tuning of Spark Jobs 14. Experience with native Cloud data services AWS or AZURE Databricks or GCP 15. Ability to lead a team efficiently 16. Experience with designing and implementing Big data solutions 17. Practitioner of AGILE methodology

Videonetics Technology Pvt Ltd. is looking for Video Streaming and Model Porting Engineer to join our dynamic team and embark on a rewarding career journey Analyzing customer needs to determine appropriate solutions for complex technical issues Creating technical diagrams, flowcharts, formulas, and other written documentation to support projects Providing guidance to junior engineers on projects within their areas of expertise Conducting research on new technologies and products in order to recommend improvements to current processes Developing designs for new products or systems based on customer specifications Researching existing technologies to determine how they could be applied in new ways to solve problems Reviewing existing products or concepts to ensure compliance with industry standards, regulations, and company policies Preparing proposals for new projects, identifying potential problems, and proposing solutions Estimating costs and scheduling requirements for projects and evaluating results

Artic Consulting is looking for Data Engineer - Microsoft Fabric Focus to join our dynamic team and embark on a rewarding career journey Liaising with coworkers and clients to elucidate the requirements for each task. Conceptualizing and generating infrastructure that allows big data to be accessed and analyzed. Reformulating existing frameworks to optimize their functioning. Testing such structures to ensure that they are fit for use. Preparing raw data for manipulation by data scientists. Detecting and correcting errors in your work. Ensuring that your work remains backed up and readily accessible to relevant coworkers. Remaining up-to-date with industry standards and technological advancements that will improve the quality of your outputs.

Introduction A career in IBM Consulting is rooted by long-term relationships and close collaboration with clients across the globe. You'll work with visionaries across multiple industries to improve the hybrid cloud and AI journey for the most innovative and valuable companies in the world. Your ability to accelerate impact and make meaningful change for your clients is enabled by our strategic partner ecosystem and our robust technology platforms across the IBM portfolio Your Role And Responsibilities Developer leads the cloud application development/deployment. A developer responsibility is to lead the execution of a project by working with a senior level resource on assigned development/deployment activities and design, build, and maintain cloud environments focusing on uptime, access, control, and network security using automation and configuration management tools Preferred Education Master's Degree Required Technical And Professional Expertise Strong proficiency in Java, Spring Framework, Spring boot, RESTful APIs, excellent understanding of OOP, Design Patterns. Strong knowledge of ORM tools like Hibernate or JPA, Java based Micro-services framework, Hands on experience on Spring boot Microservices, Primary Skills: - Core Java, Spring Boot, Java2/EE, Microservices- Hadoop Ecosystem (HBase, Hive, MapReduce, HDFS, Pig, Sqoop etc)- Spark Good to have Python. Strong knowledge of micro-service logging, monitoring, debugging and testing, In-depth knowledge of relational databases (e.g., MySQL) Experience in container platforms such as Docker and Kubernetes, experience in messaging platforms such as Kafka or IBM MQ, good understanding of Test-Driven-Development Familiar with Ant, Maven or other build automation framework, good knowledge of base UNIX commands, Experience in Concurrent design and multi-threading Preferred Technical And Professional Experience None

What the Candidate Will Need / Bonus Points ---- What the Candidate Will Do --- -Partner with engineers, analysts, and product managers to define technical solutions that support business goal sContribute to the architecture and implementation of distributed data systems and platform sIdentify inefficiencies in data processing and proactively drive improvements in performance, reliability, and cos tServe as a thought leader and mentor in data engineering best practices across the organizatio n ---- Basic Qualifications -- --7+ years of hands-on experience in software engineering with a focus on data engineeri ngProficiency in at least one programming language such as Python, Java, or Sca laStrong SQL skills and experience with large-scale data processing frameworks (e.g., Apache Spark, Flink, MapReduce, Prest o)Demonstrated experience designing, implementing, and operating scalable ETL pipelines and data platfor msProven ability to work collaboratively across teams and communicate technical concepts to diverse stakeholde rs ---- Preferred Qualifications - ---Deep understanding of data warehousing concepts and data modeling best practi cesHands-on experience with Hadoop ecosystem tools (e.g., Hive, HDFS, Oozie, Airflow, Spark, Pres to)Familiarity with streaming technologies such as Kafka or Sa mzaExpertise in performance optimization, query tuning, and resource-efficient data process ingStrong problem-solving skills and a track record of owning systems from design to product ion

Deveoper eads the coud appication deveopment/depoyment. A deveoper responsibiity is to ead the execution of a project by working with a senior eve resource on assigned deveopment/depoyment activities and design, buid, and maintain coud environments focusing on uptime, access, contro, and network security using automation and configuration management toos Required education Bacheor's Degree Preferred education Master's Degree Required technica and professiona expertise Strong proficiency in Java, Spring Framework, Spring boot, RESTfu APIs, exceent understanding of OOP, Design Patterns. Strong knowedge of ORM toos ike Hibernate or JPA, Java based Micro-services framework, Hands on experience on Spring boot Microservices Strong knowedge of micro-service ogging, monitoring, debugging and testing, In-depth knowedge of reationa databases (e.g., MySQL) Experience in container patforms such as Docker and Kubernetes, experience in messaging patforms such as Kafka or IBM MQ, Good understanding of Test-Driven-Deveopment Famiiar with Ant, Maven or other buid automation framework, good knowedge of base UNIX commands Preferred technica and professiona experience Experience in Concurrent design and muti-threading Primary Skis: - Core Java, Spring Boot, Java2/EE, Microservices - Hadoop Ecosystem (HBase, Hive, MapReduce, HDFS, Pig, Sqoop etc) - Spark Good to have Python

Organization: At CommBank, we never lose sight of the role we play in other people’s financial wellbeing. Our focus is to help people and businesses move forward to progress. To make the right financial decisions and achieve their dreams, targets, and aspirations. Regardless of where you work within our organisation, your initiative, talent, ideas, and energy all contribute to the impact that we can make with our work. Together we can achieve great things. Job Title: Associate Data Engineer-Big Data Location: Bengaluru Business & Team: RM & FS Data Engineering Impact & contribution: As a Associate Data engineer with expertise in software development / programming and a passion for building data-driven solutions, you’re ahead of trends and work at the forefront of Big Data and Data warehouse technologies. Which is why we’re the perfect fit for you. Here, you’ll be part of a team of engineers going above and beyond to improve the standard of digital banking. Using the latest tech to solve our customers’ most complex data-centric problems. To us, data is everything. It is what powers our cutting-edge features and it’s the reason we can provide seamless experiences for millions of customers from app to branch. We’re responsible for CommBank’s key analytics capabilities and work to create world-leading capabilities for analytics, information management and decisioning. We work across the Cloudera Hadoop Big Data, Teradata Group Data Warehouse and Ab Initio platforms. Roles & Responsibilities: Passionate about building next generation data platforms and data pipeline solution across the bank. Enthusiastic, be able to contribute and learn from wider engineering talent in the team. Ready to execute state-of-the-art coding practices, driving high quality outcomes to solve core business objectives and minimise risks. Capable to create both technology blueprints and engineering roadmaps, for a multi- year data transformational journey. Can lead and drive a culture where quality, excellence and openness are championed. Constantly thinking outside the box and breaking boundaries to solve complex data problems. Are experienced in providing data driven solutions that source data from various enterprise data platform into Cloudera Hadoop Big Data environment, using technologies like Spark, MapReduce, Hive, Sqoop, Kafka; transform and process the source data to produce data assets; and transform and egression to other data platforms like Teradata or RDBMS system. Are confident in building group data products or data assets from scratch, by integrating large sets of data derived from hundreds of internal and external sources. Can collaborate, co-create and contribute to existing Data Engineering practices in the team. Have experience and responsible for data security and data management. Have a natural drive to educate, communicate and coordinate with different internal stakeholders. Essential Skills: Preferably with at least 3+ years of hands-on experience in a Data Engineering role. Experience in Ab Initio software products (GDE, Co>Operating System, Express>It, etc.) is mandatory. Experience in designing, building, and delivering enterprise-wide data ingestion, data integration and data pipeline solutions using common programming language (Scala, Java, or Python) in a Big Data and Data Warehouse platform. Experience in building data solution in Hadoop platform, using Spark, MapReduce, Sqoop, Kafka and various ETL frameworks for distributed data storage and processing. Preferably with at least 3+ years of hands-on experience. Experience in building data solution using AWS Cloud technology (EMR, Glue, Iceberg, Kinesis, MSK/Kafka, Redshift, DocumentDB, S3, etc.). Preferably with 1+ years of hands-on experience and certified AWS Data Engineer. Strong Unix/Linux Shell scripting and programming skills in Scala, Java, or Python. Proficient in SQL scripting, writing complex SQLs for building data pipelines. Experience in working in Agile teams, including working closely with internal business stakeholders. Familiarity with data warehousing and/or data mart build experience in Teradata, Oracle or RDBMS system is a plus. Certification on Cloudera CDP, Hadoop, Spark, Teradata, AWS, Ab Initio is a plus. Educational Qualifications: B.Tech and above If you're already part of the Commonwealth Bank Group (including Bankwest, x15ventures), you'll need to apply through Sidekick to submit a valid application. We’re keen to support you with the next step in your career. We're aware of some accessibility issues on this site, particularly for screen reader users. We want to make finding your dream job as easy as possible, so if you require additional support please contact HR Direct on 1800 989 696. Advertising End Date: 03/07/2025 Show more Show less

About The Role This is an Internal document. Job TitleSenior Data Engineer About The Role As a Senior Data Engineer, you will play a key role in designing and implementing data solutions @Kotak811. — You will be responsible for leading data engineering projects, mentoring junior team members, and collaborating with cross-functional teams to deliver high-quality and scalable data infrastructure. — Your expertise in data architecture, performance optimization, and data integration will be instrumental in driving the success of our data initiatives. Responsibilities 1. Data Architecture and Designa. Design and develop scalable, high-performance data architecture and data models. b. Collaborate with data scientists, architects, and business stakeholders to understand data requirements and design optimal data solutions. c. Evaluate and select appropriate technologies, tools, and frameworks for data engineering projects. d. Define and enforce data engineering best practices, standards, and guidelines. 2. Data Pipeline Development & Maintenancea. Develop and maintain robust and scalable data pipelines for data ingestion, transformation, and loading for real-time and batch-use-cases b. Implement ETL processes to integrate data from various sources into data storage systems. c. Optimise data pipelines for performance, scalability, and reliability. i. Identify and resolve performance bottlenecks in data pipelines and analytical systems. ii. Monitor and analyse system performance metrics, identifying areas for improvement and implementing solutions. iii. Optimise database performance, including query tuning, indexing, and partitioning strategies. d. Implement real-time and batch data processing solutions. 3. Data Quality and Governancea. Implement data quality frameworks and processes to ensure high data integrity and consistency. b. Design and enforce data management policies and standards. c. Develop and maintain documentation, data dictionaries, and metadata repositories. d. Conduct data profiling and analysis to identify data quality issues and implement remediation strategies. 4. ML Models Deployment & Management (is a plus) This is an Internal document. a. Responsible for designing, developing, and maintaining the infrastructure and processes necessary for deploying and managing machine learning models in production environments b. Implement model deployment strategies, including containerization and orchestration using tools like Docker and Kubernetes. c. Optimise model performance and latency for real-time inference in consumer applications. d. Collaborate with DevOps teams to implement continuous integration and continuous deployment (CI/CD) processes for model deployment. e. Monitor and troubleshoot deployed models, proactively identifying and resolving performance or data-related issues. f. Implement monitoring and logging solutions to track model performance, data drift, and system health. 5. Team Leadership and Mentorshipa. Lead data engineering projects, providing technical guidance and expertise to team members. i. Conduct code reviews and ensure adherence to coding standards and best practices. b. Mentor and coach junior data engineers, fostering their professional growth and development. c. Collaborate with cross-functional teams, including data scientists, software engineers, and business analysts, to drive successful project outcomes. d. Stay abreast of emerging technologies, trends, and best practices in data engineering and share knowledge within the team. i. Participate in the evaluation and selection of data engineering tools and technologies. Qualifications1. 3-5 years"™ experience with Bachelor's Degree in Computer Science, Engineering, Technology or related field required 2. Good understanding of streaming technologies like Kafka, Spark Streaming. 3. Experience with Enterprise Business Intelligence Platform/Data platform sizing, tuning, optimization and system landscape integration in large-scale, enterprise deployments. 4. Proficiency in one of the programming language preferably Java, Scala or Python 5. Good knowledge of Agile, SDLC/CICD practices and tools 6. Must have proven experience with Hadoop, Mapreduce, Hive, Spark, Scala programming. Must have in-depth knowledge of performance tuning/optimizing data processing jobs, debugging time consuming jobs. 7. Proven experience in development of conceptual, logical, and physical data models for Hadoop, relational, EDW (enterprise data warehouse) and OLAP database solutions. 8. Good understanding of distributed systems 9. Experience working extensively in multi-petabyte DW environment 10. Experience in engineering large-scale systems in a product environment

About The Opportunity A key player in the Big Data solutions space, we specialize in creating and implementing large-scale data processing frameworks. Our mission is to help clients harness the power of data analytics to drive business insights and operational efficiency. With a strong focus on leveraging cutting-edge technologies, we provide a collaborative environment conducive to professional growth and innovation. Role & Responsibilities Design and implement scalable data processing frameworks using Hadoop and Spark. Develop ETL processes for data ingestion, transformation, and loading from diverse sources. Collaborate with data architects and analysts to optimize data models and enhance performance. Ensure data quality and integrity through rigorous testing and validation. Create and maintain documentation for data workflows, processes, and architecture. Troubleshoot and resolve data-related issues in a timely manner. Skills & Qualifications Must-Have Proficiency in the Hadoop ecosystem (HDFS, MapReduce, Hive). Hands-on experience with Apache Spark and its components. Strong SQL skills for querying relational databases. Experience with ETL tools and data integration technologies. Knowledge of data modeling techniques and best practices. Familiarity with Python for scripting and automation. Preferred Experience with NoSQL databases (Cassandra, MongoDB). Ability to tune performance for large-scale data workflows. Exposure to cloud-based data solutions (AWS, Azure). Benefits & Culture Highlights Dynamic work environment focused on innovation and continuous learning. Opportunities for professional development and career advancement. Collaborative team atmosphere that values diverse perspectives. Skills: sql proficiency,big data debveloper,data modeling techniques,data integration technologies,python scripting,etl tools,gcp,performance tuning,python,sql,hadoop ecosystem (hdfs, mapreduce, hive),apache spark,data modeling,pyspark,data warehousing,hadoop ecosystem Show more Show less

About The Opportunity A key player in the Big Data solutions space, we specialize in creating and implementing large-scale data processing frameworks. Our mission is to help clients harness the power of data analytics to drive business insights and operational efficiency. With a strong focus on leveraging cutting-edge technologies, we provide a collaborative environment conducive to professional growth and innovation. Role & Responsibilities Design and implement scalable data processing frameworks using Hadoop and Spark. Develop ETL processes for data ingestion, transformation, and loading from diverse sources. Collaborate with data architects and analysts to optimize data models and enhance performance. Ensure data quality and integrity through rigorous testing and validation. Create and maintain documentation for data workflows, processes, and architecture. Troubleshoot and resolve data-related issues in a timely manner. Skills & Qualifications Must-Have Proficiency in the Hadoop ecosystem (HDFS, MapReduce, Hive). Hands-on experience with Apache Spark and its components. Strong SQL skills for querying relational databases. Experience with ETL tools and data integration technologies. Knowledge of data modeling techniques and best practices. Familiarity with Python for scripting and automation. Preferred Experience with NoSQL databases (Cassandra, MongoDB). Ability to tune performance for large-scale data workflows. Exposure to cloud-based data solutions (AWS, Azure). Benefits & Culture Highlights Dynamic work environment focused on innovation and continuous learning. Opportunities for professional development and career advancement. Collaborative team atmosphere that values diverse perspectives. Skills: sql proficiency,big data debveloper,data modeling techniques,data integration technologies,python scripting,etl tools,gcp,performance tuning,python,sql,hadoop ecosystem (hdfs, mapreduce, hive),apache spark,data modeling,pyspark,data warehousing,hadoop ecosystem Show more Show less

About The Opportunity A key player in the Big Data solutions space, we specialize in creating and implementing large-scale data processing frameworks. Our mission is to help clients harness the power of data analytics to drive business insights and operational efficiency. With a strong focus on leveraging cutting-edge technologies, we provide a collaborative environment conducive to professional growth and innovation. Role & Responsibilities Design and implement scalable data processing frameworks using Hadoop and Spark. Develop ETL processes for data ingestion, transformation, and loading from diverse sources. Collaborate with data architects and analysts to optimize data models and enhance performance. Ensure data quality and integrity through rigorous testing and validation. Create and maintain documentation for data workflows, processes, and architecture. Troubleshoot and resolve data-related issues in a timely manner. Skills & Qualifications Must-Have Proficiency in the Hadoop ecosystem (HDFS, MapReduce, Hive). Hands-on experience with Apache Spark and its components. Strong SQL skills for querying relational databases. Experience with ETL tools and data integration technologies. Knowledge of data modeling techniques and best practices. Familiarity with Python for scripting and automation. Preferred Experience with NoSQL databases (Cassandra, MongoDB). Ability to tune performance for large-scale data workflows. Exposure to cloud-based data solutions (AWS, Azure). Benefits & Culture Highlights Dynamic work environment focused on innovation and continuous learning. Opportunities for professional development and career advancement. Collaborative team atmosphere that values diverse perspectives. Skills: sql proficiency,big data debveloper,data modeling techniques,data integration technologies,python scripting,etl tools,gcp,performance tuning,python,sql,hadoop ecosystem (hdfs, mapreduce, hive),apache spark,data modeling,pyspark,data warehousing,hadoop ecosystem Show more Show less

About The Opportunity A key player in the Big Data solutions space, we specialize in creating and implementing large-scale data processing frameworks. Our mission is to help clients harness the power of data analytics to drive business insights and operational efficiency. With a strong focus on leveraging cutting-edge technologies, we provide a collaborative environment conducive to professional growth and innovation. Role & Responsibilities Design and implement scalable data processing frameworks using Hadoop and Spark. Develop ETL processes for data ingestion, transformation, and loading from diverse sources. Collaborate with data architects and analysts to optimize data models and enhance performance. Ensure data quality and integrity through rigorous testing and validation. Create and maintain documentation for data workflows, processes, and architecture. Troubleshoot and resolve data-related issues in a timely manner. Skills & Qualifications Must-Have Proficiency in the Hadoop ecosystem (HDFS, MapReduce, Hive). Hands-on experience with Apache Spark and its components. Strong SQL skills for querying relational databases. Experience with ETL tools and data integration technologies. Knowledge of data modeling techniques and best practices. Familiarity with Python for scripting and automation. Preferred Experience with NoSQL databases (Cassandra, MongoDB). Ability to tune performance for large-scale data workflows. Exposure to cloud-based data solutions (AWS, Azure). Benefits & Culture Highlights Dynamic work environment focused on innovation and continuous learning. Opportunities for professional development and career advancement. Collaborative team atmosphere that values diverse perspectives. Skills: sql proficiency,big data debveloper,data modeling techniques,data integration technologies,python scripting,etl tools,gcp,performance tuning,python,sql,hadoop ecosystem (hdfs, mapreduce, hive),apache spark,data modeling,pyspark,data warehousing,hadoop ecosystem Show more Show less

We're Nagarro. We are a Digital Product Engineering company that is scaling in a big way! We build products, services, and experiences that inspire, excite, and delight. We work at scale across all devices and digital mediums, and our people exist everywhere in the world (18000+ experts across 38 countries, to be exact). Our work culture is dynamic and non-hierarchical. We're looking for great new colleagues. That's where you come in! REQUIREMENTS: Total experience 5+ Years. Excellent knowledge and experience in Big data engineer. Strong hands-on experience with Apache Spark and Python. Solid experience with Hadoop, MapReduce , Hive, and SQL-lik e languages. Familiarity with GCP Pub/Sub, Kafka, and Trino. Experience building end-to-end data pipelines and integrating various data sources. Understanding of both relational (e.g., PostgreSQL) and NoSQL (e.g. , MongoDB) databases. Experience with Git, CI/CD tools, and Agile development practices. Experience working on Google Cloud Platform (GCP), particularly with BigQuery, GCS, Airflow, and Kubernetes. Excellent problem-solving and analytical skills. Strong verbal and written communication abilities. RESPONSIBILITIES: Writing and reviewing great quality code Understanding the client's business use cases and technical requirements and be able to convert them into technical design which elegantly meets the requirements Mapping decisions with requirements and be able to translate the same to developers Identifying different solutions and being able to narrow down the best option that meets the clients' requirements Defining guidelines and benchmarks for NFR considerations during project implementation Writing and reviewing design document explaining overall architecture, framework, and high-level design of the application for the developers Reviewing architecture and design on various aspects like extensibility, scalability, security, design patterns, user experience, NFRs, etc., and ensure that all relevant best practices are followed Developing and designing the overall solution for defined functional and non-functional requirements; and defining technologies, patterns, and frameworks to materialize it Understanding and relating technology integration scenarios and applying these learnings in projects Resolving issues that are raised during code/review, through exhaustive systematic analysis of the root cause, and being able to justify the decision taken Carrying out POCs to make sure that suggested design/technologies meet the requirements.

About the Role We are looking for a Senior Data Engineer to contribute to high-impact initiatives within the Payments Core and Payments Compliance Data Engineering team. In this critical role, you will design, build, and optimize data systems that support the scalability, accuracy, and compliance of our global payments infrastructure. You'll work closely with cross-functional partners in Engineering, Product, Risk, and Compliance to transform complex financial and transactional data into reliable pipelines, actionable insights, and audit-ready datasets. The ideal candidate is a strong individual contributor who combines deep technical expertise with a pragmatic approach to solving real-world data challenges. You are comfortable navigating ambiguity, driving projects forward, and have a passion for improving payment systems through robust, high-quality data solutions. What the Candidate Will Need / Bonus Points - What the Candidate Will Do - Partner with engineers, analysts, and product managers to define technical solutions that support business goals Contribute to the architecture and implementation of distributed data systems and platforms Identify inefficiencies in data processing and proactively drive improvements in performance, reliability, and cost Serve as a thought leader and mentor in data engineering best practices across the organization - Basic Qualifications - 7+ years of hands-on experience in software engineering with a focus on data engineering Proficiency in at least one programming language such as Python, Java, or Scala Strong SQL skills and experience with large-scale data processing frameworks (e.g., Apache Spark, Flink, MapReduce, Presto) Demonstrated experience designing, implementing, and operating scalable ETL pipelines and data platforms Proven ability to work collaboratively across teams and communicate technical concepts to diverse stakeholders - Preferred Qualifications - Deep understanding of data warehousing concepts and data modeling best practices Hands-on experience with Hadoop ecosystem tools (e.g., Hive, HDFS, Oozie, Airflow, Spark, Presto) Familiarity with streaming technologies such as Kafka or Samza Expertise in performance optimization, query tuning, and resource-efficient data processing Strong problem-solving skills and a track record of owning systems from design to production

About The Role We are looking for a Senior Data Engineer to contribute to high-impact initiatives within the Payments Core and Payments Compliance Data Engineering team. In this critical role, you will design, build, and optimize data systems that support the scalability, accuracy, and compliance of our global payments infrastructure. You'll work closely with cross-functional partners in Engineering, Product, Risk, and Compliance to transform complex financial and transactional data into reliable pipelines, actionable insights, and audit-ready datasets. The ideal candidate is a strong individual contributor who combines deep technical expertise with a pragmatic approach to solving real-world data challenges. You are comfortable navigating ambiguity, driving projects forward, and have a passion for improving payment systems through robust, high-quality data solutions. What The Candidate Will Need / Bonus Points ---- What the Candidate Will Do ---- Partner with engineers, analysts, and product managers to define technical solutions that support business goals Contribute to the architecture and implementation of distributed data systems and platforms Identify inefficiencies in data processing and proactively drive improvements in performance, reliability, and cost Serve as a thought leader and mentor in data engineering best practices across the organization Basic Qualifications 7+ years of hands-on experience in software engineering with a focus on data engineering Proficiency in at least one programming language such as Python, Java, or Scala Strong SQL skills and experience with large-scale data processing frameworks (e.g., Apache Spark, Flink, MapReduce, Presto) Demonstrated experience designing, implementing, and operating scalable ETL pipelines and data platforms Proven ability to work collaboratively across teams and communicate technical concepts to diverse stakeholders Preferred Qualifications Deep understanding of data warehousing concepts and data modeling best practices Hands-on experience with Hadoop ecosystem tools (e.g., Hive, HDFS, Oozie, Airflow, Spark, Presto) Familiarity with streaming technologies such as Kafka or Samza Expertise in performance optimization, query tuning, and resource-efficient data processing Strong problem-solving skills and a track record of owning systems from design to production Show more Show less

Job Title: Senior Data Engineer Location: Offshore – Remote – INDIA We need (at least) 3 offshore senior data engineers for Cedar Gate Migration Project. The working hours will be mainly IST with a potential overlap with EST. The project is a migration from Hadoop Mapreduce to Snowflake. The required skills are: 1) Spark coding/development experience/PySpark (must) 2) Hadoop Mapreduce experience (nice to have) 3) Snowflake experience (must) Show more Show less

Developer leads the cloud application development/deployment. A developer responsibility is to lead the execution of a project by working with a senior level resource on assigned development/deployment activities and design, build, and maintain cloud environments focusing on uptime, access, control, and network security using automation and configuration management tools Required education Bachelor's Degree Preferred education Master's Degree Required technical and professional expertise Strong proficiency in Java, Spring Framework, Spring boot, RESTful APIs, excellent understanding of OOP, Design Patterns. Strong knowledge of ORM tools like Hibernate or JPA, Java based Micro-services framework, Hands on experience on Spring boot Microservices, Primary Skills: - Core Java, Spring Boot, Java2/EE, Microservices- Hadoop Ecosystem (HBase, Hive, MapReduce, HDFS, Pig, Sqoop etc)- Spark Good to have Python. Strong knowledge of micro-service logging, monitoring, debugging and testing, In-depth knowledge of relational databases (e.g., MySQL) Experience in container platforms such as Docker and Kubernetes, experience in messaging platforms such as Kafka or IBM MQ, good understanding of Test-Driven-Development Familiar with Ant, Maven or other build automation framework, good knowledge of base UNIX commands, Experience in Concurrent design and multi-threading Preferred technical and professional experience None

Developer leads the cloud application development/deployment. A developer responsibility is to lead the execution of a project by working with a senior level resource on assigned development/deployment activities and design, build, and maintain cloud environments focusing on uptime, access, control, and network security using automation and configuration management tools Required education Bachelor's Degree Preferred education Master's Degree Required technical and professional expertise Strong proficiency in Java, Spring Framework, Spring boot, RESTful APIs, excellent understanding of OOP, Design Patterns. Strong knowledge of ORM tools like Hibernate or JPA, Java based Micro-services framework, Hands on experience on Spring boot Microservices Strong knowledge of micro-service logging, monitoring, debugging and testing, In-depth knowledge of relational databases (e.g., MySQL) Experience in container platforms such as Docker and Kubernetes, experience in messaging platforms such as Kafka or IBM MQ, Good understanding of Test-Driven-Development Familiar with Ant, Maven or other build automation framework, good knowledge of base UNIX commands Preferred technical and professional experience Experience in Concurrent design and multi-threading Primary Skills: - Core Java, Spring Boot, Java2/EE, Microservices - Hadoop Ecosystem (HBase, Hive, MapReduce, HDFS, Pig, Sqoop etc) - Spark Good to have Python

The Manager, Software Development Engineering leads a team of technical experts in successfully executing technology projects and solutions that align with the strategy and have broad business impact. The Manager, Software Development Engineering will work closely with development teams to identify and understand key features and their underlying functionality while also partnering closely with Product Management and UX Design. They may exercise influence and govern overall end-to-end software development life cycle related activities including management of support and maintenance releases, minor functional releases, and major projects. The Manager, Software Development Engineering will lead & provide technical guidance for process improvement programs while leveraging engineering best practices. In this people leadership role, Managers will recruit, train, motivate, coach, grow and develop Software Development Engineer team members at a variety of levels through their technical expertise and providing continuous feedback to ensure employee expectations, customer needs and product demands are met. About the Role: Lead and manage a team of engineers, providing mentorship and fostering a collaborative environment. Design, implement, and maintain scalable data pipelines and systems to support business analytics and data science initiatives. Collaborate with cross-functional teams to understand data requirements and ensure data solutions align with business goals. Ensure data quality, integrity, and security across all data processes and systems. Drive the adoption of best practices in data engineering, including coding standards, testing, and automation. Evaluate and integrate new technologies and tools to enhance data processing and analytics capabilities. Prepare and present reports on engineering activities, metrics, and project progress to stakeholders. About You: Proficiency in programming languages such as Python, Java, or Scala. Data Engineering with API & any programming language. Strong understanding of APIs and possess forward-looking knowledge of AI/ML tools or models and need to have some knowledge on software architecture. Experience with cloud platforms (e.g., AWS,Google Cloud) and big data technologies (e.g., Hadoop, Spark). Experience with Rest/Odata API's Strong problem-solving skills and the ability to work in a fast-paced environment. Excellent communication and interpersonal skills. Experience with data warehousing solutions such as BigQuery or snowflakes Familiarity with data visualization tools and techniques. Understanding of machine learning concepts and frameworks. #LI-AD2 Whats in it For You Hybrid Work Model Weve adopted a flexible hybrid working environment (2-3 days a week in the office depending on the role) for our office-based roles while delivering a seamless experience that is digitally and physically connected. Flexibility & Work-Life Balance: Flex My Way is a set of supportive workplace policies designed to help manage personal and professional responsibilities, whether caring for family, giving back to the community, or finding time to refresh and reset. This builds upon our flexible work arrangements, including work from anywhere for up to 8 weeks per year, empowering employees to achieve a better work-life balance. Career Development and Growth: By fostering a culture of continuous learning and skill development, we prepare our talent to tackle tomorrows challenges and deliver real-world solutions. Our Grow My Way programming and skills-first approach ensures you have the tools and knowledge to grow, lead, and thrive in an AI-enabled future. Industry Competitive Benefits We offer comprehensive benefit plans to include flexible vacation, two company-wide Mental Health Days off, access to the Headspace app, retirement savings, tuition reimbursement, employee incentive programs, and resources for mental, physical, and financial wellbeing. Culture: Globally recognized, award-winning reputation for inclusion and belonging, flexibility, work-life balance, and more. We live by our valuesObsess over our Customers, Compete to Win, Challenge (Y)our Thinking, Act Fast / Learn Fast, and Stronger Together. Social Impact Make an impact in your community with our Social Impact Institute. We offer employees two paid volunteer days off annually and opportunities to get involved with pro-bono consulting projects and Environmental, Social, and Governance (ESG) initiatives. Making a Real-World Impact: We are one of the few companies globally that helps its customers pursue justice, truth, and transparency. Together, with the professionals and institutions we serve, we help uphold the rule of law, turn the wheels of commerce, catch bad actors, report the facts, and provide trusted, unbiased information to people all over the world. Thomson Reuters informs the way forward by bringing together the trusted content and technology that people and organizations need to make the right decisions. We serve professionals across legal, tax, accounting, compliance, government, and media. Our products combine highly specialized software and insights to empower professionals with the data, intelligence, and solutions needed to make informed decisions, and to help institutions in their pursuit of justice, truth, and transparency. Reuters, part of Thomson Reuters, is a world leading provider of trusted journalism and news. We are powered by the talents of 26,000 employees across more than 70 countries, where everyone has a chance to contribute and grow professionally in flexible work environments. At a time when objectivity, accuracy, fairness, and transparency are under attack, we consider it our duty to pursue them. Sound excitingJoin us and help shape the industries that move society forward. As a global business, we rely on the unique backgrounds, perspectives, and experiences of all employees to deliver on our business goals. To ensure we can do that, we seek talented, qualified employees in all our operations around the world regardless of race, color, sex/gender, including pregnancy, gender identity and expression, national origin, religion, sexual orientation, disability, age, marital status, citizen status, veteran status, or any other protected classification under applicable law. Thomson Reuters is proud to be an Equal Employment Opportunity Employer providing a drug-free workplace. We also make reasonable accommodations for qualified individuals with disabilities and for sincerely held religious beliefs in accordance with applicable law. More information on requesting an accommodation here. Learn more on how to protect yourself from fraudulent job postings here. More information about Thomson Reuters can be found on thomsonreuters.com.

Responsibilities Ability to write clean, maintainable, and robust code in Python Understanding and expertise of software engineering concepts and best practices Knowledge of testing frameworks and libraries Experience with analytics (descriptive, predictive, EDA), feature engineer, algorithms, anomaly detection, data quality assessment and python visualization libraries - e.g. matplotlib, seaborn or other Comfortable with notebook and source code development - Jupyter, Pycharm/VScode Hands-on experience of technologies like Python, Spark/Pyspark, Hadoop/MapReduce/HIVE, Pandas etc. Familiarity with query languages and database technologies, CI/CD, testing and validation of data and software Tech stack and activities that you would use and preform on a daily basis : Python Spark (PySpark) Jupyter SQL and No-SQL DBMS Git (as source code versioning and CI/CD) Exploratory Data Analysis (EDA) Imputation Techniques Data Linking / Cleansing Feature Engineering Apache Airflow/ Jenkins scheduling and automation, Github and Github Actions Collaborative - able to build strong relations that enable robust debate, and resolve periodic disagreements regarding priorities. Excellent interpersonal, and communication skills Ability to communicate effectively with technical and non-technical audience Ability to work under pressure with a solid sense for setting priorities Ability to lead technical work with strong sense of ownership Strong command of English language (both verbal and written) Practical and action oriented Compelling communicator Excellent stakeholder management Foster and promote entrepreneurial spirit and curiosity amongst team members Team player Quick learner (ref:hirist.tech) Show more Show less

The Applications Development Programmer Analyst is an intermediate level position responsible for participation in the establishment and implementation of new or revised application systems and programs in coordination with the Technology team. The overall objective of this role is to contribute to applications systems analysis and programming activities. Responsibilities: Utilize knowledge of applications development procedures and concepts, and basic knowledge of other technical areas to identify and define necessary system enhancements Identify and analyze issues, make recommendations, and implement solutions Utilize knowledge of business processes, system processes, and industry standards to solve complex issues Analyze information and make evaluative judgements to recommend solutions and improvements Conduct testing and debugging, utilize script tools, and write basic code for design specifications Assess applicability of similar experiences and evaluate options under circumstances not covered by procedures Develop working knowledge of Citi’s information systems, procedures, standards, client server application development, network operations, database administration, systems administration, data center operations, and PC-based applications Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency. Additional Job Description We are looking for a Big Data Engineer that will work on the collecting, storing, processing, and analyzing of huge sets of data. The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them. You will also be responsible for integrating them with the architecture used across the company. Responsibilities Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities Implementing data wrangling, scarping, cleaning using both Java or Python Strong experience on data structure. Extensively work on API integration. Monitoring performance and advising any necessary infrastructure changes Defining data retention policies Skills And Qualifications Proficient understanding of distributed computing principles Proficient in Java or Pyhton and some part of machine learning Proficiency with Hadoop v2, MapReduce, HDFS,Pyspark,Spark Experience with building stream-processing systems, using solutions such as Storm or Spark-Streaming Good knowledge of Big Data querying tools, such as Pig, Hive, and Impala Experience with Spark Experience with integration of data from multiple data sources Experience with NoSQL databases, such as HBase, Cassandra, MongoDB Knowledge of various ETL techniques and frameworks, such as Flume Experience with various messaging systems, such as Kafka or RabbitMQ Experience with Big Data ML toolkits, such as Mahout, SparkML, or H2O Good understanding of Lambda Architecture, along with its advantages and drawbacks Experience with Cloudera/MapR/Hortonworks Qualifications: 0-2 years of relevant experience Experience in programming/debugging used in business applications Working knowledge of industry practice and standards Comprehensive knowledge of specific business area for application development Working knowledge of program languages Consistently demonstrates clear and concise written and verbal communication Education: Bachelor’s degree/University degree or equivalent experience This job description provides a high-level review of the types of work performed. Other job-related duties may be assigned as required. ------------------------------------------------------ Job Family Group: Technology ------------------------------------------------------ Job Family: Applications Development ------------------------------------------------------ Time Type: Full time ------------------------------------------------------ Most Relevant Skills Please see the requirements listed above. ------------------------------------------------------ Other Relevant Skills For complementary skills, please see above and/or contact the recruiter. ------------------------------------------------------ Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law. If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi. View Citi’s EEO Policy Statement and the Know Your Rights poster. Show more Show less

Developer leads the cloud application development/deployment. A developer responsibility is to lead the execution of a project by working with a level resource on assigned development/deployment activities and design, build, and maintain cloud environments focusing on uptime, access, control, and network security using automation and configuration management tools Required education Bachelor's Degree Preferred education Master's Degree Required technical and professional expertise Strong proficiency in Java, Spring Framework, Spring boot, RESTful APIs, excellent understanding of OOP, Design Patterns. Strong knowledge of ORM tools like Hibernate or JPA, Java based Micro-services framework, Hands on experience on Spring boot Microservices Strong knowledge of micro-service logging, monitoring, debugging and testing, In-depth knowledge of relational databases (e.g., MySQL) Experience in container platforms such as Docker and Kubernetes, experience in messaging platforms such as Kafka or IBM MQ, Good understanding of Test-Driven-Development Familiar with Ant, Maven or other build automation framework, good knowledge of base UNIX commands Preferred technical and professional experience Experience in Concurrent design and multi-threading Primary Skills: - Core Java, Spring Boot, Java2/EE, Microservices - Hadoop Ecosystem (HBase, Hive, MapReduce, HDFS, Pig, Sqoop etc) - Spark Good to have Python

As an Associate Software Developer at IBM, you'll work with clients to co-create solutions to major real-world challenges by using best practice technologies, tools, techniques, and products to translate system requirements into the design and development of customized systems Required education Bachelor's Degree Preferred education Master's Degree Required technical and professional expertise Core Java, Spring Boot, Java 2/EE, Microsservices - Hadoop Ecosystem (HBase, Hive, MapReduce, HDFS, Pig, Sqoop etc) Spark Good to have Python Preferred technical and professional experience None