2510 Hive Jobs - Page 25

Roles And Responsibilities Proficiency in building highly scalable ETL and streaming-based data pipelines using Google Cloud Platform (GCP) services and products like Biquark, Cloud Dataflow Proficiency in large scale data platforms and data processing systems such as Google Big Query, Amazon Redshift, Azure Data Lake Excellent Python, PySpark and SQL development and debugging skills, exposure to other Big Data frameworks like Hadoop Hive would be added advantage Experience building systems to retrieve and aggregate data from event-driven messaging frameworks (e.g. RabbitMQ and Pub/Sub) Secondary Skills : Cloud Big Table, AI/ML solutions, Compute Engine, Cloud Fusion (ref:hirist.tech) Show more Show less

Role: Data QA Lead Experience Required- 8+ Years Location- India/Remote Company Overview At Codvo.ai, software and people transformations go hand-in-hand. We are a global empathy-led technology services company. Product innovation and mature software engineering are part of our core DNA. Respect, Fairness, Growth, Agility, and Inclusiveness are the core values that we aspire to live by each day. We continue to expand our digital strategy, design, architecture, and product management capabilities to offer expertise, outside-the-box thinking, and measurable results. The Data Quality Analyst is responsible for ensuring the quality, accuracy, and consistency of data within the Customer and Loan Master Data API solution. This role will work closely with data owners, data modelers, and developers to identify and resolve data quality issues. Key Responsibilities Lead and manage end-to-end ETL/data validation activities. Design test strategy, plans, and scenarios for source-to-target validation. Build automated data validation frameworks (SQL/Python/Great Expectations). Integrate tests with CI/CD pipelines (Jenkins, Azure DevOps). Perform data integrity, transformation logic, and reconciliation checks. Collaborate with Data Engineering, Product, and DevOps teams. Drive test metrics reporting, defect triage, and root cause analysis. Mentor QA team members and ensure process adherence. Must-Have Skills 8+ years in QA with 4+ years in ETL testing. Strong SQL and database testing experience. Proficiency with ETL tools (Airbyte, DBT, Informatica, etc.). Automation using Python or similar scripting language. Solid understanding of data warehousing, SCD, deduplication. Experience with large datasets and structured/unstructured formats. Preferred Skills Knowledge of data orchestration tools (Prefect, Airflow). Familiarity with data quality/observability tools. Experience with big data systems (Spark, Hive). Hands-on with test data generation (Faker, Mockaroo). Show more Show less

Description We are looking for a skilled Data Engineer to join our existing team. Candidates will be working on Big Data applications using cutting edge technologies including Google Cloud Platform. The position offers 1st hand exposure to build data pipelines that can process peta-byte scale of data solving complex business problems Requirements Mandatory Skills: 4-6 years of hands-on experience in Data Engineering. Experience in writing and optimizing SQL queries in HIVE/Spark. Excellent coding and/or scripting skills in Python. Good experience in deploying spark applications in Kubernetes cluster. Good experience in development, deployment & troubleshooting of Spark applications. Exposure to any cloud environment (AWS/GCP preferred) Job responsibilities Role Description: Candidate will be part of an agile team. Development/Migration of new data pipelines. Optimizing/Fine-tuning existing workflows. Deploying Spark tasks on K8 clusters. Bringing new ideas for performance enhancement of data pipelines running on K8s.. Mandatory Skills: 4-6 years of hands-on experience in Data Engineering. Experience in writing and optimizing SQL queries in HIVE/Spark. Excellent coding and/or scripting skills in Python. Good experience in deploying spark applications in Kubernetes cluster. Good experience in development, deployment & troubleshooting of Spark applications. Exposure to any cloud environment (AWS/GCP preferred) What we offer Culture of caring. At GlobalLogic, we prioritize a culture of caring. Across every region and department, at every level, we consistently put people first. From day one, you’ll experience an inclusive culture of acceptance and belonging, where you’ll have the chance to build meaningful connections with collaborative teammates, supportive managers, and compassionate leaders. Learning and development. We are committed to your continuous learning and development. You’ll learn and grow daily in an environment with many opportunities to try new things, sharpen your skills, and advance your career at GlobalLogic. With our Career Navigator tool as just one example, GlobalLogic offers a rich array of programs, training curricula, and hands-on opportunities to grow personally and professionally. Interesting & meaningful work. GlobalLogic is known for engineering impact for and with clients around the world. As part of our team, you’ll have the chance to work on projects that matter. Each is a unique opportunity to engage your curiosity and creative problem-solving skills as you help clients reimagine what’s possible and bring new solutions to market. In the process, you’ll have the privilege of working on some of the most cutting-edge and impactful solutions shaping the world today. Balance and flexibility. We believe in the importance of balance and flexibility. With many functional career areas, roles, and work arrangements, you can explore ways of achieving the perfect balance between your work and life. Your life extends beyond the office, and we always do our best to help you integrate and balance the best of work and life, having fun along the way! High-trust organization. We are a high-trust organization where integrity is key. By joining GlobalLogic, you’re placing your trust in a safe, reliable, and ethical global company. Integrity and trust are a cornerstone of our value proposition to our employees and clients. You will find truthfulness, candor, and integrity in everything we do. About GlobalLogic GlobalLogic, a Hitachi Group Company, is a trusted digital engineering partner to the world’s largest and most forward-thinking companies. Since 2000, we’ve been at the forefront of the digital revolution – helping create some of the most innovative and widely used digital products and experiences. Today we continue to collaborate with clients in transforming businesses and redefining industries through intelligent products, platforms, and services. Show more Show less

Skill : Java, Spark, Kafka Experience : 10 to 16 years Location : Hyderabad As Data Engineer, you will : Support in designing and rolling out the data architecture and infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources Identify data source, design and implement data schema/models and integrate data that meet the requirements of the business stakeholders Play an active role in the end-to-end delivery of AI solutions, from ideation, feasibility assessment, to data preparation and industrialization. Work with business, IT and data stakeholders to support with data-related technical issues, their data infrastructure needs as well as to build the most flexible and scalable data platform. With a strong focus on DataOps, design, develop and deploy scalable batch and/or real-time data pipelines. Design, document, test and deploy ETL/ELT processes Find the right tradeoffs between the performance, reliability, scalability, and cost of the data pipelines you implement Monitor data processing efficiency and propose solutions for improvements. • Have the discipline to create and maintain comprehensive project documentation. • Build and share knowledge with colleagues and coach junior profiles.

Position Description Founded in 1976, CGI is among the world's largest independent IT and business consulting services firms. With 94,000 consultants and professionals globally, CGI delivers an end-to-end portfolio of capabilities, from strategic IT and business consulting to systems integration, managed IT and business process services, and intellectual property solutions. CGI works with clients through a local relationship model complemented by a global delivery network that helps clients digitally transform their organizations and accelerate results. CGI Fiscal 2024 reported revenue is CA$14.68 billion, and CGI shares are listed on the TSX (GIB.A) and the NYSE (GIB). Learn more at cgi.com. Position - Senior Software Engineer Experience - 4 - 7 Yrs Category - Software Development/Engineering Shift - 1 to 10 PM Location - BNG/HYD/CHN Position Id - J0125-0901 Work Type - Hybrid Employment Type - Full time Education - Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field. Your future duties and responsibilities We are looking for a talented Data Engg to join our team. In this role, you will develop, optimize, and maintain scalable applications, and be responsible for building efficient, testable, and reusable code. Your work will involve collaborating with cross-functional teams to deliver high-quality software that meets our clients' needs. Write reusable, testable, and efficient code. Implement security and data protection solutions. Develop and maintain robust and scalable backend systems and APIs using Python. Integrate user-facing elements developed by front-end developers with server-side logic. Work with various databases (SQL, NoSQL) to ensure efficient data storage and retrieval. Required Qualifications To Be Successful In This Role Programing Language : Python, Pyspark Bigdata Tech – Data Bricks, Spark, Hadoop, Hive Cloud – AWS Database – RDBMS & No SQL Shell Scripting Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field. CGI is an equal opportunity employer. In addition, CGI is committed to providing accommodation for people with disabilities in accordance with provincial legislation. Please let us know if you require reasonable accommodation due to a disability during any aspect of the recruitment process and we will work with you to address your needs. Together, as owners, let’s turn meaningful insights into action. Life at CGI is rooted in ownership, teamwork, respect and belonging. Here, you’ll reach your full potential because… You are invited to be an owner from day 1 as we work together to bring our Dream to life. That’s why we call ourselves CGI Partners rather than employees. We benefit from our collective success and actively shape our company’s strategy and direction. Your work creates value. You’ll develop innovative solutions and build relationships with teammates and clients while accessing global capabilities to scale your ideas, embrace new opportunities, and benefit from expansive industry and technology expertise. You’ll shape your career by joining a company built to grow and last. You’ll be supported by leaders who care about your health and well-being and provide you with opportunities to deepen your skills and broaden your horizons. Come join our team—one of the largest IT and business consulting services firms in the world. Show more Show less

Introduction In this role, you'll work in one of our IBM Consulting Client Innovation Centers (Delivery Centers), where we deliver deep technical and industry expertise to a wide range of public and private sector clients around the world. Our delivery centers offer our clients locally based skills and technical expertise to drive innovation and adoption of new technology. Your Role And Responsibilities As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform Responsibilities Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developing streaming pipelines Experience to work with Hadoop / AWS eco system components to implement scalable solutions to meet the ever-increasing data volumes, using big data/cloud technologies Apache Spark, Kafka, any Cloud computing etc Preferred Education Master's Degree Required Technical And Professional Expertise Minimum 4+ years of experience in Big Data technologies with extensive data engineering experience in Spark / Python or Scala. Minimum 3 years of experience on Cloud Data Platforms on AWS Experience in AWS EMR / AWS Glue / DataBricks, AWS RedShift, DynamoDB Good to excellent SQL skills Exposure to streaming solutions and message brokers like Kafka technologies Preferred Technical And Professional Experience Certification in AWS and Data Bricks or Cloudera Spark Certified developers Show more Show less

Entity: Technology Job Family Group: IT&S Group Job Description: You will work with Being part of a digital delivery data group supporting bp Solutions, you will apply your domain knowledge and familiarity with domain data processes to support the organisation. Part of bp’s Production & Operations business, bp Solutions has hubs in London, Pune, and Houston. The data team provides daily operational data management, data engineering and analytics support to this organisation across a broad range of activity from facilities and subsea engineering to logistics. Let me tell you about the role A data analyst collects, processes, and performs analyses on a variety of datasets. Their key responsibilities include interpreting sophisticated data sets to identify trends and patterns, using analytical tools and methods to generate actionable insights, and crafting visualizations and reports to communicate those insights and recommendations to support decision-making. Data analysts collaborate closely with business domain collaborators to understand their data analysis needs, ensure data accuracy, write and recommend data-driven solutions and tackle value impacting business problems. You might be a good fit for this role if you: Have strong domain knowledge in at least one of; facilities or subsea engineering, maintenance and reliability, operations, logistics. Strong analytical skills and proven capability in applying analytical techniques and Python scripting to solve practical problems. Are curious, and keen to apply new technologies, trends & methods to improve existing standards and the capabilities of the Subsurface community. Are well organized and self-motivated, you balance proactive and reactive approaches and across multiple priorities to complete tasks on time. Apply judgment and common sense – you use insight and good judgment to inform actions and respond to situations as they arise. What you will deliver Be a link between asset teams and Technology, combining in-depth understanding of one or more relevant domains with data & analytics skills Provide actionable, data-driven insights by combining deep statistical skills, data manipulation capabilities and business insight. Proactively identify impactful opportunities and autonomously complete data analysis. You apply existing data & analytics strategies relevant to your immediate scope. Clean, pre-process and analyse both structured and unstructured data Develop data visualisations to analyse and interrogate broad datasets (e.g. with tools such as Microsoft PowerBI, Spotfire or similar). Present results to peers and senior management, influencing decision making What you will need to be successful (experience and qualifications) Essential MSc or equivalent experience in a quantitative field, preferably statistics. have strong domain knowledge in at least one of; facilities or subsea engineering, maintenance and reliability, operations, logistics. Hands-on experience carrying out data analytics, data mining and product analytics in complex, fast-paced environments. Applied knowledge of data analytics and data pipelining tools and approaches across all data lifecycle stages. Deep understanding of a few and a high-level understanding of several commonly available statistics approaches. Advanced SQL knowledge. Advanced scripting experience in R or python. Ability to write and maintain moderately sophisticated data pipelines. Customer-centric and pragmatic approach. Focus on value delivery and swift execution, while maintaining attention to detail. Good communication and social skills, with the ability to effectively communicate ideas, expectations, and feedback to team members, partners, and customers. Foster collaboration and teamwork Desired Advanced analytics degree. Experience applying analytics to support engineering turnarounds Experience with big data technologies (e.g. Hadoop, Hive, and Spark) is a plus. About Bp Our purpose is to deliver energy to the world, today and tomorrow. For over 100 years, bp has focused on discovering, developing, and producing oil and gas in the nations where we operate. We are one of the few companies globally that can provide governments and customers with an integrated energy offering. Delivering our strategy sustainably is fundamental to achieving our ambition to be a net zero company by 2050 or sooner! Legal Disclaimer: We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, sex, gender, gender expression, sexual orientation, age, marital status, socioeconomic status, neurodiversity/neurocognitive functioning, veteran status or disability status. Individuals with an accessibility need may request an adjustment/accommodation related to bp’s recruiting process (e.g., accessing the job application, completing required assessments, participating in telephone screenings or interviews, etc.). If you would like to request an adjustment/accommodation related to the recruitment process, please contact us. If you are selected for a position and depending upon your role, your employment may be contingent upon adherence to local policy. This may include pre-placement drug screening, medical review of physical fitness for the role, and background checks. Travel Requirement Up to 10% travel should be expected with this role Relocation Assistance: This role is eligible for relocation within country Remote Type: This position is a hybrid of office/remote working Skills: Legal Disclaimer: We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, sex, gender, gender expression, sexual orientation, age, marital status, socioeconomic status, neurodiversity/neurocognitive functioning, veteran status or disability status. Individuals with an accessibility need may request an adjustment/accommodation related to bp’s recruiting process (e.g., accessing the job application, completing required assessments, participating in telephone screenings or interviews, etc.). If you would like to request an adjustment/accommodation related to the recruitment process, please contact us. If you are selected for a position and depending upon your role, your employment may be contingent upon adherence to local policy. This may include pre-placement drug screening, medical review of physical fitness for the role, and background checks. Show more Show less

At American Express, our culture is built on a 175-year history of innovation, shared values and Leadership Behaviors, and an unwavering commitment to back our customers, communities, and colleagues. As part of Team Amex, you'll experience this powerful backing with comprehensive support for your holistic well-being and many opportunities to learn new skills, develop as a leader, and grow your career. Here, your voice and ideas matter, your work makes an impact, and together, you will help us define the future of American Express. You Lead the Way. We’ve Got Your Back. With the right backing, people and businesses have the power to progress in incredible ways. When you join Team Amex, you become part of a global and diverse community of colleagues with an unwavering commitment to back our customers, communities and each other. Here, you’ll learn and grow as we help you create a career journey that’s unique and meaningful to you with benefits, programs, and flexibility that support you personally and professionally. At American Express, you’ll be recognized for your contributions, leadership, and impact—every colleague has the opportunity to share in the company’s success. Together, we’ll win as a team, striving to uphold our company values and powerful backing promise to provide the world’s best customer experience every day. And we’ll do it with the utmost integrity, and in an environment where everyone is seen, heard and feels like they belong. Join Team Amex and let's lead the way together. How will you make an impact in this role? The U.S. Consumer Services is responsible for our suite of consumer products, services, and experiences in the U.S., including our premium, cobrand, cash back, and lending portfolios, Membership Rewards, Global Travel & Lifestyle Services, the Centurion Lounge network, our Global Dining and Resy businesses, and best-in-class marketing programs across customer, prospect, and partner channels. The USCS team is focused on making membership an indisputable competitive advantage and helping each other become better leaders every day. The U.S. Consumer Data Office (USCDO) is responsible for advancing Enterprise Data Governance principles and policies across the data assets underpinning our consumer products. In this role, you will be responsible for end-to-end implementation of both data management and data governance opportunities across all aspects of U.S. Consumer data, developing new data assets, creating roadmap for data products & capabilities, ingesting data into Lumi, driving USCC teams compliance to AEMP79 policy (Data Risk Management Policy), implementing Data Quality Controls, and identifying and remediating data Issues. Responsibilities: Develop & maintain data products & solutions, highly relevant for USCC business growth – in collaboration with USCC business teams, tech partners and other teams Develop strong subject matter expertise on both internal and external data assets, identify synergies and create new strategic derived data assets (ODLs) Act as the custodian for data standardization, data governance, data quality and data ownership, while ensuring compliance and security of the data. Leverage statistical models and ML to create intelligent & accurate data quality checks across critical datasets Innovate with a focus on developing newer and better approaches using big data & cloud technologies Build strong relationships, operate effectively within large cross-functional teams, and influence business stakeholders to drive change. Minimum Qualifications: Proficiency in SQL. Good understanding of Python, Hive, Shell Scripting, GCP and related Data Warehousing solutions like BigQuery. Demonstrated capability in hypothesis testing & data-driven analysis Familiarity with Data Governance concepts (metadata management, data lineage, quality frameworks) Knowledge of industry-standard methodologies for data quality assurance, including statistical profiling, outlier detection, anomaly identification Ability to frame strategies, and communicate in a clear and consumable manner, tailored to the audience Ability to learn quickly and work independently with complex, unstructured initiatives Lead with an external perspective, challenge status quo and bring continuous innovation to our existing processes We back you with benefits that support your holistic well-being so you can be and deliver your best. This means caring for you and your loved ones' physical, financial, and mental health, as well as providing the flexibility you need to thrive personally and professionally: Competitive base salaries Bonus incentives Support for financial-well-being and retirement Comprehensive medical, dental, vision, life insurance, and disability benefits (depending on location) Flexible working model with hybrid, onsite or virtual arrangements depending on role and business need Generous paid parental leave policies (depending on your location) Free access to global on-site wellness centers staffed with nurses and doctors (depending on location) Free and confidential counseling support through our Healthy Minds program Career development and training opportunities American Express is an equal opportunity employer and makes employment decisions without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran status, disability status, age, or any other status protected by law. Offer of employment with American Express is conditioned upon the successful completion of a background verification check, subject to applicable laws and regulations. Show more Show less

Overview The Data Technology team at MSCI is responsible for meeting the data requirements across various business areas, including Index, Analytics, and Sustainability. Our team collates data from multiple sources such as vendors (e.g., Bloomberg, Reuters), website acquisitions, and web scraping (e.g., financial news sites, company websites, exchange websites, filings). This data can be in structured or semi-structured formats. We normalize the data, perform quality checks, assign internal identifiers, and release it to downstream applications. Responsibilities As data engineers, we build scalable systems to process data in various formats and volumes, ranging from megabytes to terabytes. Our systems perform quality checks, match data across various sources, and release it in multiple formats. We leverage the latest technologies, sources, and tools to process the data. Some of the exciting technologies we work with include Snowflake, Databricks, and Apache Spark. Qualifications Core Java, Spring Boot, Apache Spark, Spring Batch, Python. Exposure to sql databases like Oracle, Mysql, Microsoft Sql is a must. Any experience/knowledge/certification on Cloud technology preferrably Microsoft Azure or Google cloud platform is good to have. Exposures to non sql databases like Neo4j or Document database is again good to have. What we offer you Transparent compensation schemes and comprehensive employee benefits, tailored to your location, ensuring your financial security, health, and overall wellbeing. Flexible working arrangements, advanced technology, and collaborative workspaces. A culture of high performance and innovation where we experiment with new ideas and take responsibility for achieving results. A global network of talented colleagues, who inspire, support, and share their expertise to innovate and deliver for our clients. Global Orientation program to kickstart your journey, followed by access to our Learning@MSCI platform, LinkedIn Learning Pro and tailored learning opportunities for ongoing skills development. Multi-directional career paths that offer professional growth and development through new challenges, internal mobility and expanded roles. We actively nurture an environment that builds a sense of inclusion belonging and connection, including eight Employee Resource Groups. All Abilities, Asian Support Network, Black Leadership Network, Climate Action Network, Hola! MSCI, Pride & Allies, Women in Tech, and Women’s Leadership Forum. At MSCI we are passionate about what we do, and we are inspired by our purpose – to power better investment decisions. You’ll be part of an industry-leading network of creative, curious, and entrepreneurial pioneers. This is a space where you can challenge yourself, set new standards and perform beyond expectations for yourself, our clients, and our industry. MSCI is a leading provider of critical decision support tools and services for the global investment community. With over 50 years of expertise in research, data, and technology, we power better investment decisions by enabling clients to understand and analyze key drivers of risk and return and confidently build more effective portfolios. We create industry-leading research-enhanced solutions that clients use to gain insight into and improve transparency across the investment process. MSCI Inc. is an equal opportunity employer. It is the policy of the firm to ensure equal employment opportunity without discrimination or harassment on the basis of race, color, religion, creed, age, sex, gender, gender identity, sexual orientation, national origin, citizenship, disability, marital and civil partnership/union status, pregnancy (including unlawful discrimination on the basis of a legally protected parental leave), veteran status, or any other characteristic protected by law. MSCI is also committed to working with and providing reasonable accommodations to individuals with disabilities. If you are an individual with a disability and would like to request a reasonable accommodation for any part of the application process, please email Disability.Assistance@msci.com and indicate the specifics of the assistance needed. Please note, this e-mail is intended only for individuals who are requesting a reasonable workplace accommodation; it is not intended for other inquiries. To all recruitment agencies MSCI does not accept unsolicited CVs/Resumes. Please do not forward CVs/Resumes to any MSCI employee, location, or website. MSCI is not responsible for any fees related to unsolicited CVs/Resumes. Note on recruitment scams We are aware of recruitment scams where fraudsters impersonating MSCI personnel may try and elicit personal information from job seekers. Read our full note on careers.msci.com

PositionSenior Data Engineer - Airflow, PLSQL Experience5+ Years LocationBangalore/Hyderabad/Pune Seeking a Senior Data Engineer with strong expertise in Apache Airflow and Oracle PL/SQL, along with working experience in Snowflake and Agile methodologies. The ideal candidate will also take up Scrum Master responsibilities and lead a data engineering scrum team to deliver robust, scalable data solutions. Key Responsibilities: Design, develop, and maintain scalable data pipelines using Apache Airflow. Write and optimize complex PL/SQL queries, procedures, and packages on Oracle databases. Collaborate with cross-functional teams to design efficient data models and integration workflows. Work with Snowflake for data warehousing and analytics use cases. Own the delivery of sprint goals, backlog grooming, and facilitation of agile ceremonies as the Scrum Master. Monitor pipeline health and troubleshoot production data issues proactively. Ensure code quality, documentation, and best practices across the team. Mentor junior data engineers and promote a culture of continuous improvement. Required Skills and Qualifications: 5+ years of experience as a Data Engineer in enterprise environments. Strong expertise in Apache Airflow for orchestrating workflows. Expert in Oracle PL/SQL - stored procedures, performance tuning, debugging. Hands-on experience with Snowflake - data modeling, SQL, optimization. Working knowledge of version control (Git) and CI/CD practices. Prior experience or certification as a Scrum Master is highly desirable. Strong analytical and problem-solving skills with attention to detail. Excellent communication and leadership skills.

The SAS to Databricks Migration Developer will be responsible for migrating existing SAS code, data processes, and workflows to the Databricks platform This role requires expertise in both SAS and Databricks, with a focus on converting SAS logic into scalable PySpark and Python code The developer will design, implement, and optimize data pipelines, ensuring seamless integration and functionality within the Databricks environment Collaboration with various teams is essential to understand data requirements and deliver solutions that meet business needs

Job Overview: We are looking for experienced Data Engineers proficient in Hadoop, Hive, Python, SQL, and Pyspark/Spark to join our dynamic team. Candidates will be responsible for designing, developing, and maintaining scalable big data solutions. Key Responsibilities: Develop and optimize data pipelines for large-scale data processing. Work with structured and unstructured datasets to derive actionable insights. Collaborate with cross-functional teams to enhance data-driven decision-making. Ensure the performance, scalability, and reliability of data architectures. Implement best practices for data security and governance.

Allime Tech Solutions is looking for Hadoop with Scala Developer to join our dynamic team and embark on a rewarding career journey. A Developer is responsible for designing, developing, and maintaining software applications and systems They collaborate with a team of software developers, designers, and stakeholders to create software solutions that meet the needs of the business Key responsibilities:Design, code, test, and debug software applications and systemsCollaborate with cross-functional teams to identify and resolve software issuesWrite clean, efficient, and well-documented codeStay current with emerging technologies and industry trendsParticipate in code reviews to ensure code quality and adherence to coding standardsParticipate in the full software development life cycle, from requirement gathering to deploymentProvide technical support and troubleshooting for production issues Requirements:Strong programming skills in one or more programming languages, such as Python, Java, C++, or JavaScriptExperience with software development tools, such as version control systems (e g Git), integrated development environments (IDEs), and debugging toolsFamiliarity with software design patterns and best practicesGood communication and collaboration skills

Job Category: IT Job Type: Full Time Job Location: Bangalore Chennai Mumbai Pune Location- Mumbai, Pune, Bangalore, Chennai Experience- 5+ Data Engineer: Expertise in Python Language is MUST. SQL (should be able to write complex SQL Queries) is MUST Data Lake Development experience. Orchestration (Apache Airflow is preferred). Spark and Hive: Optimization of Spark/PySpark and Hive apps is MUST Trino/(AWS Athena) (Good to have) Snowflake (good to have). Data Quality (good to have). File Storage (S3 is good to have) Kind Note: Please apply or share your resume only if it matches the above criteria

Only for Immediate Joiners Core Responsibility: The project team will be spread between Paris and Bangalore. So, the candidate with an experience of 3-6 years is expected to work and coordinate on daily basis with the remote teams. Ability to learn new technology / framework / methodology. Hands-on individual responsible for producing excellent quality of code, adhering to expected coding standards and industry best practices. Must have strong knowledge and working experience on Big DATA ecosystem. Must have strong experience in SPARK/SCALA , NIFI, KAFKA, HIVE, PIG. Strong knowledge and experience working on HQL (hive Query Language) • Must have strong strong expertise in Debugging and Fixing Production Issues on BIG DATA eco System. Knowledge on code version management using Git & Jenkins, Nexus. • High levels of ownership and commitment on deliverables. Strong and Adaptive Communication Skills; Should be comfortable interacting with Paris counterparts to probe a technical problem or clarify requirement specifications. KEY SKILLS: Sound knowledge on SPARK/SCALA, NIFI, KAFKA - Must Have Sound Knowledge on HQL Knowledge on Kibana, Elastic Search Log stash Good to know Basic Awareness of CD/CI concepts & Technologies Big Data Ecosystem Good to know

job requisition idJR1027452 Overall Responsibilities: Data Pipeline Development: Design, develop, and maintain highly scalable and optimized ETL pipelines using PySpark on the Cloudera Data Platform, ensuring data integrity and accuracy. Data Ingestion: Implement and manage data ingestion processes from a variety of sources (e.g., relational databases, APIs, file systems) to the data lake or data warehouse on CDP. Data Transformation and Processing: Use PySpark to process, cleanse, and transform large datasets into meaningful formats that support analytical needs and business requirements. Performance Optimization: Conduct performance tuning of PySpark code and Cloudera components, optimizing resource utilization and reducing runtime of ETL processes. Data Quality and Validation: Implement data quality checks, monitoring, and validation routines to ensure data accuracy and reliability throughout the pipeline. Automation and Orchestration: Automate data workflows using tools like Apache Oozie, Airflow, or similar orchestration tools within the Cloudera ecosystem. Monitoring and Maintenance: Monitor pipeline performance, troubleshoot issues, and perform routine maintenance on the Cloudera Data Platform and associated data processes. Collaboration: Work closely with other data engineers, analysts, product managers, and other stakeholders to understand data requirements and support various data-driven initiatives. Documentation: Maintain thorough documentation of data engineering processes, code, and pipeline configurations. Software : Advanced proficiency in PySpark, including working with RDDs, DataFrames, and optimization techniques. Strong experience with Cloudera Data Platform (CDP) components, including Cloudera Manager, Hive, Impala, HDFS, and HBase. Knowledge of data warehousing concepts, ETL best practices, and experience with SQL-based tools (e.g., Hive, Impala). Familiarity with Hadoop, Kafka, and other distributed computing tools. Experience with Apache Oozie, Airflow, or similar orchestration frameworks. Strong scripting skills in Linux. Category-wise Technical Skills: PySpark: Advanced proficiency in PySpark, including working with RDDs, DataFrames, and optimization techniques. Cloudera Data Platform: Strong experience with Cloudera Data Platform (CDP) components, including Cloudera Manager, Hive, Impala, HDFS, and HBase. Data Warehousing: Knowledge of data warehousing concepts, ETL best practices, and experience with SQL-based tools (e.g., Hive, Impala). Big Data Technologies: Familiarity with Hadoop, Kafka, and other distributed computing tools. Orchestration and Scheduling: Experience with Apache Oozie, Airflow, or similar orchestration frameworks. Scripting and Automation: Strong scripting skills in Linux. Experience: 5-12 years of experience as a Data Engineer, with a strong focus on PySpark and the Cloudera Data Platform. Proven track record of implementing data engineering best practices. Experience in data ingestion, transformation, and optimization on the Cloudera Data Platform. Day-to-Day Activities: Design, develop, and maintain ETL pipelines using PySpark on CDP. Implement and manage data ingestion processes from various sources. Process, cleanse, and transform large datasets using PySpark. Conduct performance tuning and optimization of ETL processes. Implement data quality checks and validation routines. Automate data workflows using orchestration tools. Monitor pipeline performance and troubleshoot issues. Collaborate with team members to understand data requirements. Maintain documentation of data engineering processes and configurations. Qualifications: Bachelors or Masters degree in Computer Science, Data Engineering, Information Systems, or a related field. Relevant certifications in PySpark and Cloudera technologies are a plus. Soft Skills: Strong analytical and problem-solving skills. Excellent verbal and written communication abilities. Ability to work independently and collaboratively in a team environment. Attention to detail and commitment to data quality.

We are seeking a skilled and motivated Data Engineer to join our dynamic team. The ideal candidate will have experience in designing, developing, and maintaining scalable data pipelines and architectures using Hadoop, PySpark, ETL processes , and Cloud technologies . Role: Senior Data Engineer Experience: 4-8 years Job locations: Coimbatore, Chennai, Bangalore, Hyderabad Responsibilities Design, develop, and maintain data pipelines for processing large-scale datasets. Build efficient ETL workflows to transform and integrate data from multiple sources. Develop and optimize Hadoop and PySpark applications for data processing. Ensure data quality, governance, and security standards are met across systems. Implement and manage Cloud-based data solutions (AWS, Azure, or GCP). Collaborate with data scientists and analysts to support business intelligence initiatives. Troubleshoot performance issues and optimize query executions in big data environments. Stay updated with industry trends and advancements in big data and cloud technologies. Required Skills Strong programming skills in Python, Scala, or Java. Hands-on experience with Hadoop ecosystem (HDFS, Hive, Spark, etc.). Expertise in PySpark for distributed data processing. Proficiency in ETL tools and workflows (SSIS, Apache Nifi, or custom pipelines). Experience with Cloud platforms (AWS, Azure, GCP) and their data-related services. Knowledge of SQL and NoSQL databases. Familiarity with data warehousing concepts and data modeling techniques. Strong analytical and problem-solving skills. Interested can contact us at +91 7305206696/ saranyadevib@talentien.com Skills: sql,data warehousing,aws,cloud,hadoop,scala,java,python,data engineering,azure,cloud technologies (aws, azure, gcp),etl processes,data modeling,nosql,pyspark,etl Show more Show less

We are seeking a skilled and motivated Data Engineer to join our dynamic team. The ideal candidate will have experience in designing, developing, and maintaining scalable data pipelines and architectures using Hadoop, PySpark, ETL processes , and Cloud technologies . Role: Senior Data Engineer Experience: 4-8 years Job locations: Coimbatore, Chennai, Bangalore, Hyderabad Responsibilities Design, develop, and maintain data pipelines for processing large-scale datasets. Build efficient ETL workflows to transform and integrate data from multiple sources. Develop and optimize Hadoop and PySpark applications for data processing. Ensure data quality, governance, and security standards are met across systems. Implement and manage Cloud-based data solutions (AWS, Azure, or GCP). Collaborate with data scientists and analysts to support business intelligence initiatives. Troubleshoot performance issues and optimize query executions in big data environments. Stay updated with industry trends and advancements in big data and cloud technologies. Required Skills Strong programming skills in Python, Scala, or Java. Hands-on experience with Hadoop ecosystem (HDFS, Hive, Spark, etc.). Expertise in PySpark for distributed data processing. Proficiency in ETL tools and workflows (SSIS, Apache Nifi, or custom pipelines). Experience with Cloud platforms (AWS, Azure, GCP) and their data-related services. Knowledge of SQL and NoSQL databases. Familiarity with data warehousing concepts and data modeling techniques. Strong analytical and problem-solving skills. Interested can contact us at +91 7305206696/ saranyadevib@talentien.com Skills: sql,data warehousing,aws,cloud,hadoop,scala,java,python,data engineering,azure,cloud technologies (aws, azure, gcp),etl processes,data modeling,nosql,pyspark,etl Show more Show less

Minimum 12+ years of relevant experience in building software applications in data and analytics field Enhance the go-to-market strategy by designing new and relevant solution frameworks to accelerate our clients’ journeys for impacting patient outcomes. Pitch for these opportunities and craft winning proposals to grow the Data Science Practice. Build and lead a team of data scientists and analysts, fostering a collaborative and innovative environment. Oversee the design and delivery of the models, ensuring projects are completed on time and meet business objectives. Engaging in consultative selling with clients to grow/deliver business. Develop and operationalize scalable processes to deliver on large & complex client engagements. Extensive hands-on experience with Python, R, or Julia, focusing on data science and generative AI frameworks. Expertise in working with generative models such as GPT, DALL-E, Stable Diffusion, Codex, and MidJourney for various applications. Proficiency in fine-tuning and deploying generative models using libraries like Hugging Face Transformers, Diffusers, or PyTorch Lightning. Strong understanding of generative techniques, including GANs, VAEs, diffusion models, and autoregressive models. Experience in prompt engineering, zero-shot, and few-shot learning for optimizing generative AI outputs across different use cases. Expertise in managing generative AI data pipelines, including preprocessing large-scale multimodal datasets for text, image, or code generation.

Position Summary To be a technology expert architecting solutions and mentoring people in BI / Reporting processes with prior expertise in the Pharma domain. Job Responsibilities o Technology Leadership – Lead guide the team independently or with little support to design, implement deliver complex reporting and BI project assignments. o Technical portfolio – Expertise in a range of BI and hosting technologies like the AWS stack (Redshift, EC2), Qlikview, QlikSense, Tableau, Microstrategy, Spotfire o Project Management – Get accurate briefs from the Client and translate into tasks for team members with priorities and timeline plans. Must maintain high standards of quality and thoroughness. Should be able to monitor accuracy and quality of others' work. Ability to think in advance about potential risks and mitigation plans. o Logical Thinking – Able to think analytically, use a systematic and logical approach to analyze data, problems, and situations. Must be able to guide team members in analysis. o Handle Client Relationship – Manage client relationship and client expectations independently. Should be able to deliver results back to the Client independently. Should have excellent communication skills. Education BE/B.Tech Master of Computer Application Work Experience - Minimum of 5 years of relevant experience in Pharma domain. - Technical: Should have 10+ years of hands on experience in the following tools: Must have working knowledge of toolsAtleast 2 of the following – Qlikview, QlikSense, Tableau, Microstrategy, Spotfire/ (Informatica, SSIS, Talend & metallion)/ Big Data technologies - Hadoop ecosystem. Aware of techniques such asUI design, Report modeling, performance tuning and regression testing Basic expertise with MS excel Advanced expertise with SQL - Functional: Should have experience in following concepts and technologies: Specifics: Pharma data sources like IMS, Veeva, Symphony, Cegedim etc. Business processes like alignment, market definition, segmentation, sales crediting, activity metrics calculation Calculation of all sales, activity and managed care KPIs Behavioural Competencies Teamwork & Leadership Motivation to Learn and Grow Ownership Cultural Fit Talent Management Technical Competencies Problem Solving Lifescience Knowledge Communication Project Management Attention to P&L Impact Capability Building / Thought Leadership Scale of revenues managed / delivered

Engineer, Digital Data Development Gurgaon/ Bangalore, India AXA XL offers risk transfer and risk management solutions to clients globally. We offer worldwide capacity, flexible underwriting solutions, a wide variety of client-focused loss prevention services, and a team-based account management approach. AXA XL recognizes data and information as critical business assets, both in terms of managing risk and enabling new business opportunities. This data should not only be high quality, but also actionable - enabling AXA XL’s executive leadership team to maximize benefits and facilitate sustained dynamic advantage. Our Innovation, Data, and Analytics (IDA) organization is focused on driving innovation by optimizing how we leverage data to drive strategy and create a new business model - disrupting the insurance market. This role is part of the Digital Data Dev Division within the Digital Transformation vertical of IDA. It will be responsible for different aspects of Data Product development lifecycle activities, including but not limited to Data Production Support, business stakeholders’ engagement for usage & problem resolutions, Product migrations, and platform/data product rollouts, performance stability & reliability. What You’ll Be DOING What will your essential responsibilities include? Hands-on experience with CI/CD tools: Harness, Azure DevOps. Implement and manage DevSecOps tools and CI/CD pipelines with security controls. Automate security scanning and compliance checks (SAST, DAST, container scanning, etc.). Collaborate with development, operations, and security teams to embed security best practices. Conduct threat modeling, vulnerability assessments, and risk. Build, Release Management & DevSecOps support for various data solutions owned and managed by IDA organization. Experience with cloud platforms like Azure is preferred. Proficiency in scripting languages: Python, Bash, PowerShell. Familiarity with containerization and orchestration: Docker, Kubernetes, OpenShift. Experience using Tools like Git, JIRA, Confluence etc. Knowledge of Artifactory like JFrog / X-Ray. Experience of working with Agile methodologies. Good knowledge of OOP concepts & Microservice-based architecture. Analyze and mitigate risks (technical or otherwise) about Data Solution build & release delivery timelines. Provide top-class DevSecOps functionalities and support. Partner with the Product & Production Support team(s) as a Data/DevSecOps/Technical SME for migration of re-architected Product/Product functionalities to the new Cloud Platform. Demonstrate proactive communication with Business users, Development, Technology, Production Support, and Delivery Teams, and Senior Management. Provide day-to-day management of the DevSecOps services and ensure smooth operation of the Release pipelines to various Environments. Work in the “Follow the Sun” support model providing cross-team support coverage across Digital Data Dev division responsibilities. Build/Setup/Maintain various critical monitoring processes, alerts, and overall health reports (performance and functional) of production, and pre-production environments to be used by the Production Support Teams. Work with Product Teams to build deployment pipelines for various Data Science Products used within IDA/Pricing & Analytics Teams. Oversee the development and maintenance of Build & Release Management processes and their documentation. Ensure that all policies, standards, and best practices are followed and kept up to date. Timely and accurate completion of emergency Release pipelines/processes in a manner that is auditable, testable, and maintainable. Ensure any builds are consistent with Solution design, Security recommendations and business specifications. Achieve & maintain the highest business customer confidence and net promoter score (NPS). Good grasp of Azure fundamentals (Microsoft AZ-900). Robust understanding of Designing and Implementing DevOps/DevSecOps Solutions (Microsoft AZ-400). Knowledge of Python or R Programming Language is a plus. You will report to Senior Delivery Lead. What You Will BRING We’re looking for someone who has these abilities and skills: Required Skills And Abilities Excellent understanding of DevOps principles with integrated security practices. A minimum of an Undergraduate University Degree in Computer Science or related fields. Extensive experience in data-focused roles (analytics, specialist, or engineer) and one or more areas of Build, Release & Data Management. Distinctive problem-solving and analytical skills combined with robust business acumen. Experience/knowledge of Microservices, .NET, R Programming Language, Python, Azure, and Kibana. Experience with SQL, HIVE, ADLS, and Document Databases like Cosmos, SQL Databases & SQL DW Analytics. Experience/Understanding of systems integration, and developer support tools Azure DevOps/DevSecOps, CI/CD pipelines, Release Management, Configuration Management, and Automation. Data Engineering background or working experience with ETL and big data platforms (HDInsight / ADLS / Data Bricks) a plus. Desired Skills And Abilities Demonstrates a level of experience/ability to influence and understand business problems in technical terminology and able to liaise with staff at all levels in the organization. Excellent writing skills, with the ability to create clear requirements, specifications, and documentation for data systems. Experience with multiple software delivery models (Waterfall, Agile, etc.) is a plus. Previous experience leading small teams with a mix of onsite/offshore developers. Who WE are AXA XL, the P&C and specialty risk division of AXA, is known for solving complex risks. For mid-sized companies, multinationals and even some inspirational individuals we don’t just provide re/insurance, we reinvent it. How? By combining a comprehensive and efficient capital platform, data-driven insights, leading technology, and the best talent in an agile and inclusive workspace, empowered to deliver top client service across all our lines of business − property, casualty, professional, financial lines and specialty. With an innovative and flexible approach to risk solutions, we partner with those who move the world forward. Learn more at axaxl.com What we OFFER Inclusion AXA XL is committed to equal employment opportunity and will consider applicants regardless of gender, sexual orientation, age, ethnicity and origins, marital status, religion, disability, or any other protected characteristic. At AXA XL, we know that an inclusive culture and enables business growth and is critical to our success. That’s why we have made a strategic commitment to attract, develop, advance and retain the most inclusive workforce possible, and create a culture where everyone can bring their full selves to work and reach their highest potential. It’s about helping one another — and our business — to move forward and succeed. Five Business Resource Groups focused on gender, LGBTQ+, ethnicity and origins, disability and inclusion with 20 Chapters around the globe. Robust support for Flexible Working Arrangements Enhanced family-friendly leave benefits Named to the Diversity Best Practices Index Signatory to the UK Women in Finance Charter Learn more at axaxl.com/about-us/inclusion-and-diversity. AXA XL is an Equal Opportunity Employer. Total Rewards AXA XL’s Reward program is designed to take care of what matters most to you, covering the full picture of your health, wellbeing, lifestyle and financial security. It provides competitive compensation and personalized, inclusive benefits that evolve as you do. We’re committed to rewarding your contribution for the long term, so you can be your best self today and look forward to the future with confidence. Sustainability At AXA XL, Sustainability is integral to our business strategy. In an ever-changing world, AXA XL protects what matters most for our clients and communities. We know that sustainability is at the root of a more resilient future. Our 2023-26 Sustainability strategy, called “Roots of resilience”, focuses on protecting natural ecosystems, addressing climate change, and embedding sustainable practices across our operations. Our Pillars Valuing nature: How we impact nature affects how nature impacts us. Resilient ecosystems - the foundation of a sustainable planet and society - are essential to our future. We’re committed to protecting and restoring nature - from mangrove forests to the bees in our backyard - by increasing biodiversity awareness and inspiring clients and colleagues to put nature at the heart of their plans. Addressing climate change: The effects of a changing climate are far-reaching and significant. Unpredictable weather, increasing temperatures, and rising sea levels cause both social inequalities and environmental disruption. We're building a net zero strategy, developing insurance products and services, and mobilizing to advance thought leadership and investment in societal-led solutions. Integrating ESG: All companies have a role to play in building a more resilient future. Incorporating ESG considerations into our internal processes and practices builds resilience from the roots of our business. We’re training our colleagues, engaging our external partners, and evolving our sustainability governance and reporting. AXA Hearts in Action: We have established volunteering and charitable giving programs to help colleagues support causes that matter most to them, known as AXA XL’s “Hearts in Action” programs. These include our Matching Gifts program, Volunteering Leave, and our annual volunteering day - the Global Day of Giving. For more information, please see axaxl.com/sustainability. Show more Show less

We are seeking a skilled and motivated Data Engineer to join our dynamic team. The ideal candidate will have experience in designing, developing, and maintaining scalable data pipelines and architectures using Hadoop, PySpark, ETL processes , and Cloud technologies . Role: Senior Data Engineer Experience: 4-8 years Job locations: Coimbatore, Chennai, Bangalore, Hyderabad Responsibilities Design, develop, and maintain data pipelines for processing large-scale datasets. Build efficient ETL workflows to transform and integrate data from multiple sources. Develop and optimize Hadoop and PySpark applications for data processing. Ensure data quality, governance, and security standards are met across systems. Implement and manage Cloud-based data solutions (AWS, Azure, or GCP). Collaborate with data scientists and analysts to support business intelligence initiatives. Troubleshoot performance issues and optimize query executions in big data environments. Stay updated with industry trends and advancements in big data and cloud technologies. Required Skills Strong programming skills in Python, Scala, or Java. Hands-on experience with Hadoop ecosystem (HDFS, Hive, Spark, etc.). Expertise in PySpark for distributed data processing. Proficiency in ETL tools and workflows (SSIS, Apache Nifi, or custom pipelines). Experience with Cloud platforms (AWS, Azure, GCP) and their data-related services. Knowledge of SQL and NoSQL databases. Familiarity with data warehousing concepts and data modeling techniques. Strong analytical and problem-solving skills. Interested can contact us at +91 7305206696/ saranyadevib@talentien.com Skills: sql,data warehousing,aws,cloud,hadoop,scala,java,python,data engineering,azure,cloud technologies (aws, azure, gcp),etl processes,data modeling,nosql,pyspark,etl Show more Show less

Overview Data Science Team works in developing Machine Learning (ML) and Artificial Intelligence (AI) projects. Specific scope of this role is to develop ML solution in support of ML/AI projects using big analytics toolsets in a CI/CD environment. Analytics toolsets may include DS tools/Spark/Databricks, and other technologies offered by Microsoft Azure or open-source toolsets. This role will also help automate the end-to-end cycle with Azure Pipelines. You will be part of a collaborative interdisciplinary team around data, where you will be responsible of our continuous delivery of statistical/ML models. You will work closely with process owners, product owners and final business users. This will provide you the correct visibility and understanding of criticality of your developments. Responsibilities Delivery of key Advanced Analytics/Data Science projects within time and budget, particularly around DevOps/MLOps and Machine Learning models in scope Active contributor to code & development in projects and services Partner with data engineers to ensure data access for discovery and proper data is prepared for model consumption. Partner with ML engineers working on industrialization. Communicate with business stakeholders in the process of service design, training and knowledge transfer. Support large-scale experimentation and build data-driven models. Refine requirements into modelling problems. Influence product teams through data-based recommendations. Research in state-of-the-art methodologies. Create documentation for learnings and knowledge transfer. Create reusable packages or libraries. Ensure on time and on budget delivery which satisfies project requirements, while adhering to enterprise architecture standards Leverage big data technologies to help process data and build scaled data pipelines (batch to real time) Implement end-to-end ML lifecycle with Azure Databricks and Azure Pipelines Automate ML models deployments Qualifications BE/B.Tech in Computer Science, Maths, technical fields. Overall 2-4 years of experience working as a Data Scientist. 2+ years’ experience building solutions in the commercial or in the supply chain space. 2+ years working in a team to deliver production level analytic solutions. Fluent in git (version control). Understanding of Jenkins, Docker are a plus. Fluent in SQL syntaxis. 2+ years’ experience in Statistical/ML techniques to solve supervised (regression, classification) and unsupervised problems. 2+ years’ experience in developing business problem related statistical/ML modeling with industry tools with primary focus on Python or Pyspark development. Data Science - Hands on experience and strong knowledge of building machine learning models - supervised and unsupervised models. Knowledge of Time series/Demand Forecast models is a plus Programming Skills - Hands-on experience in statistical programming languages like Python, Pyspark and database query languages like SQL Statistics - Good applied statistical skills, including knowledge of statistical tests, distributions, regression, maximum likelihood estimators Cloud (Azure) - Experience in Databricks and ADF is desirable Familiarity with Spark, Hive, Pig is an added advantage Business storytelling and communicating data insights in business consumable format. Fluent in one Visualization tool. Strong communications and organizational skills with the ability to deal with ambiguity while juggling multiple priorities Experience with Agile methodology for team work and analytics ‘product’ creation. Experience in Reinforcement Learning is a plus. Experience in Simulation and Optimization problems in any space is a plus. Experience with Bayesian methods is a plus. Experience with Causal inference is a plus. Experience with NLP is a plus. Experience with Responsible AI is a plus. Experience with distributed machine learning is a plus Experience in DevOps, hands-on experience with one or more cloud service providers AWS, GCP, Azure(preferred) Model deployment experience is a plus Experience with version control systems like GitHub and CI/CD tools Experience in Exploratory data Analysis Knowledge of ML Ops / DevOps and deploying ML models is preferred Experience using MLFlow, Kubeflow etc. will be preferred Experience executing and contributing to ML OPS automation infrastructure is good to have Exceptional analytical and problem-solving skills Stakeholder engagement-BU, Vendors. Experience building statistical models in the Retail or Supply chain space is a plus Show more Show less

Introduction A career in IBM Consulting is rooted by long-term relationships and close collaboration with clients across the globe. You'll work with visionaries across multiple industries to improve the hybrid cloud and AI journey for the most innovative and valuable companies in the world. Your ability to accelerate impact and make meaningful change for your clients is enabled by our strategic partner ecosystem and our robust technology platforms across the IBM portfolio; including Software and Red Hat. Curiosity and a constant quest for knowledge serve as the foundation to success in IBM Consulting. In your role, you'll be encouraged to challenge the norm, investigate ideas outside of your role, and come up with creative solutions resulting in ground breaking impact for a wide network of clients. Our culture of evolution and empathy centers on long-term career growth and development opportunities in an environment that embraces your unique skills and experience Your Role And Responsibilities Location : Mumbai Role Overview As a Big Data Engineer, you'll design and build robust data pipelines on Cloudera using Spark (Scala/PySpark) for ingestion, transformation, and processing of high-volume data from banking systems. Key Responsibilities Build scalable batch and real-time ETL pipelines using Spark and Hive Integrate structured and unstructured data sources Perform performance tuning and code optimization Support orchestration and job scheduling (NiFi, Airflow) Preferred Education Master's Degree Required Technical And Professional Expertise Experience: 3–15 years Proficiency in PySpark/Scala with Hive/Impala Experience with data partitioning, bucketing, and optimization Familiarity with Kafka, Iceberg, NiFi is a must Knowledge of banking or financial datasets is a plus Show more Show less

Our Purpose Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential. Title And Summary Lead ML Engineer- Spark, Python. SQL Our Purpose We work to connect and power an inclusive, digital economy that benefits everyone, everywhere by making transactions safe, simple, smart and accessible. Using secure data and networks, partnerships and passion, our innovations and solutions help individuals, financial institutions, governments and businesses realize their greatest potential. Our decency quotient, or DQ, drives our culture and everything we do inside and outside of our company. We cultivate a culture of inclusion for all employees that respects their individual strengths, views, and experiences. We believe that our differences enable us to be a better team – one that makes better decisions, drives innovation and delivers better business results. Title and Summary Lead, ML Engineering Overview: As a Lead Machine Learning Engineer of the Data Science & AI team, you will develop analytical products and solutions that sit atop vast datasets gathered by retail stores, restaurants, banks, and other consumer-focused companies. The challenge will be to create high-performance algorithms built on data sets measured in the billions of transactions that allow our users to derive insights from big data that in turn drive their businesses with a keen eye for data privacy and governance. Role: Leads talent acquisition efforts and initiatives, facilitates training programs and conducts performance management for team of direct reports Lead teams in the creation of portfolio robust ML solutions through effective use of Mastercard’s global data assets and software platform Build, productionize and maintain data driver AI/ML application and data processing workflows or pipelines Consult with clients/ stakeholders to understand and translate their needs into a data analyses and/or solution, ensuring that their requirements are accurately captured and technically feasible Guide others in comprehensive technical analyses and allocates work across teams to ensure the delivery of high quality and effective solutions Liaise with internal stakeholders (e.g., MA TECH, Data Strategy Management, AI governance) to identify and elaborate on opportunities as they relate to analytical solution development, feasibility, and other technical offerings Lead development of presentations and technical documentation Identify and recommend opportunities to standardize and automate efforts to ensure quality and enable scaling of ML products Meet project deadlines for accountable deliverables and anticipates delays or foreseeable barriers to progress and escalates issues when necessary Conduct due diligence quality assurance testing for prototypes and tools in stage and resolves reoccurring complex issues and bugs Ensure that all machine learning processes, from data preparation to model deployment, are well-documented for internal use and compliance. Mentor and guide junior developers All About You: Expertise in Big Data Technologies: Proficiency in big data frameworks and tools such as Hadoop, Spark, Hive Technical Proficiency: Strong programming skills in languages such as Python and SQL. Experience with data visualization tools (e.g., Tableau, Power BI) and understanding of cloud computing services (AWS, Azure, GCP) related to data processing and storage is a plus. Experience with testing frameworks and test-driven development (TDD) practices Advanced Analytical Skills: Strong applied knowledge and hands on experience in machine learning algorithms and deep learning frameworks. Familiarity with AI and machine learning platforms such as TensorFlow, PyTorch, or similar. Familiar with training and deploying models with large datasets including strategies for parallelizing and optimizing the training/deployment workflows. Experience in productionizing of GenAI products a plus. Leadership and Strategic Planning: Proven experience in leading engineering teams, defining vision and strategy for data-driven initiatives, and driving projects from conception to implementation. Ability to mentor and develop talent within the team. Problem-Solving Skills: Strong analytical and critical thinking abilities to solve complex problems, along with the creativity to find innovative solutions. Communication and Collaboration: Excellent verbal and written communication skills, with the ability to explain complex analytical concepts to non-technical stakeholders. Experience in working cross-functionally with departments and flexibility to work as a member of a matrix based diverse and geographically distributed project teams. Project Management Skills: Proficiency in managing multiple projects simultaneously, with a focus on delivering results within tight deadlines. Responsible AI knowledge: Awareness of the principles and practices surrounding responsible AI, including fairness, transparency, accountability, and ethics in AI deployments. Innovation and Continuous Learning: A mindset geared towards innovation, staying abreast of industry trends, emerging technologies in big data and analytics, and continuously seeking opportunities for personal and professional growth. Corporate Security Responsibility All Activities Involving Access To Mastercard Assets, Information, And Networks Comes With An Inherent Risk To The Organization And, Therefore, It Is Expected That Every Person Working For, Or On Behalf Of, Mastercard Is Responsible For Information Security And Must: Abide by Mastercard’s security policies and practices; Ensure the confidentiality and integrity of the information being accessed; Report any suspected information security violation or breach, and Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines. Corporate Security Responsibility All Activities Involving Access To Mastercard Assets, Information, And Networks Comes With An Inherent Risk To The Organization And, Therefore, It Is Expected That Every Person Working For, Or On Behalf Of, Mastercard Is Responsible For Information Security And Must: Abide by Mastercard’s security policies and practices; Ensure the confidentiality and integrity of the information being accessed; Report any suspected information security violation or breach, and Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines. R-249208 Show more Show less