Home
Jobs

4958 Hadoop Jobs - Page 43

Filter
Filter Interviews
Min: 0 years
Max: 25 years
Min: ₹0
Max: ₹10000000
Setup a job Alert
JobPe aggregates results for easy application access, but you actually apply on the job portal directly.

5.0 years

0 Lacs

Bengaluru

On-site

Job Description Position Overview We are seeking a highly skilled and experienced Data Architect with expertise in cloud-based solutions. The ideal candidate will design, implement, and optimize our data architecture to meet the organization's current and future needs. This role requires a strong background in data modeling, transformation, and governance, along with hands-on experience with modern cloud platforms and tools such as Snowflake, Spark, Data Lakes, and Data Warehouses. The successful candidate will also establish and enforce standards and guidelines across data platforms to ensure consistency, scalability, and best practices. Exceptional communication skills are essential to collaborate across cross-functional teams and stakeholders. Key Responsibilities Design and Implementation: Architect and implement scalable, secure, and high-performance cloud data platforms, integrating data lakes, data warehouses, and databases. Develop comprehensive data models to support analytics, reporting, and operational needs. Data Integration and Transformation: Lead the design and execution of ETL/ELT pipelines using tools like, Talend / Matillion, SQL, BigData, Hadoop, AWS EMR, Apache Spark to process and transform data efficiently. Integrate diverse data sources into cohesive and reusable datasets for business intelligence and machine learning purposes. Standards and Guidelines: Establish, document, and enforce standards and guidelines for data architecture, Data modeling, transformation, and governance across all data platforms. Ensure consistency and best practices in data storage, integration, and security throughout the organization. Data Governance: Establish and enforce data governance standards, ensuring data quality, security, and compliance with regulatory requirements. Implement processes and tools to manage metadata, lineage, and data access controls. Cloud Expertise: Utilize Snowflake for advanced analytics and data storage needs, ensuring optimized performance and cost efficiency. Leverage modern cloud platforms to manage data lakes and ensure seamless integration with other services. Collaboration and Communication: Partner with business stakeholders, data engineers, and analysts to gather requirements and translate them into technical designs. Clearly communicate architectural decisions, trade-offs, and progress to both technical and non-technical audiences. Continuous Improvement: Stay updated on emerging trends in cloud and data technologies, recommending innovations to enhance the organization’s data capabilities. Optimize existing architectures to improve scalability, performance, and maintainability. Qualifications Technical Skills: Strong expertise in data modeling (conceptual, logical, physical) and data architecture design principles. Proficiency in Talend / Matillion, SQL, BigData, Hadoop, AWS EMR, Apache Spark, Snowflake, and cloud-based data platforms. Experience with data lakes, data warehouses, and relational and NoSQL databases. Experience with relational(PGSQL/Oracle) / NoSQL(Couchbase/Cassandra) databases Solid understanding of data transformation techniques and ETL/ELT pipelines. Proficiency in DevOps / DataOps / MLOps tools. Standards and Governance: Experience establishing and enforcing data platform standards, guidelines, and governance frameworks. Proven ability to align data practices with business goals and regulatory compliance. Communication: Exceptional written and verbal communication skills to interact effectively with technical teams and business stakeholders. Experience: 5+ years of experience in data architecture, with a focus on cloud technologies. Proven track record of delivering scalable, cloud-based data solutions. Education: Bachelor's or Master's degree in Computer Science, Information Systems, or a related field. Preferred Qualifications Certification in Snowflake, AWS data services, Any RDBMS / NoSQL, AI/ML, Data Governance. Familiarity with machine learning workflows and data pipelines. Experience working in Agile development environments. Job Type: Full-time Schedule: Monday to Friday Night shift Rotational shift Work Location: In person

Posted 1 week ago

Apply

15.0 years

4 - 9 Lacs

Bengaluru

On-site

Principal S/W Architect (B4) (Gen AI) Position Description: Honeywell's Value Engineering (VE) and Component Engineering (CE) Center of Excellence (COE) is a dynamic collective of professionals dedicated to refining product development through innovative engineering and strategic component selection. You will be part of Honeywell's VE/CE CoE Advanced Tech team driving Tech Charter & Strategy with focus on how AI/ML can enable us meet VE/CE revenue targets. You will be responsible to drive VE/CE COE Gen AI strategy & roadmap; perform market research; partner with strategic vendors and incubate new technologies, ideas and techniques that will ensure VE/CE CoE is able to leverage the latest and the best in the industry; and continues to be more efficient. You will also need to leverage existing Honeywell Technology ecosystem; Experts, CoPs and other Tech Forums to ensure adoption of best practices & ideas . YOU MUST HAVE: Overall 15+ years of experience, out of which 5+ years of experience in leading and managing AI/ML teams and projects , with demonstrated success in delivering impactful and actionable insights A master's degree or PhD in computer science, statistics, engineering, mathematics, or related fields. AI/ML Skills: Proficiency in Python, R, SQL, and common AI/ML frameworks and tools such as TensorFlow, PyTorch, Scikit-learn, etc. Solid understanding of machine learning concepts and algorithms, including supervised and unsupervised learning, deep learning etc. Working knowledge in LLM models like Gemini, Llama, GPT, DALL-E and frameworks like LangChain & Vector databases like Faiss, Pinecone etc. Experience in fine-tuning LLM using LoRA/QLoRA techniques Develop APIs and microservices to support Gen AI/ML applications Data Skills: Expertise in data integration, ETL processes, and working with SQL/NoSQL databases. Familiarity with Big Data technologies (Hadoop, Spark) and data warehousing solutions. Cloud & MLOps Skills: Strong understanding of cloud platforms, particularly GCP, for deploying AI applications Demonstrated experience in scaling AI/ML projects using MLOps practices for model lifecycle management Ability to analyse complex data sets, derive insights, and solve sophisticated problems with innovative solutions. Strong communication and presentation skills, with the ability to explain complex AI/ML concepts and results to both technical and non-technical audiences. Strong understanding of Business Process, IT Industry Standards Stay up-to-date with the latest developments in Gen AI/ML and cloud technologies. Track record of innovation and leadership in productizing AI research

Posted 1 week ago

Apply

7.0 years

0 Lacs

Bengaluru

On-site

Company Description The Bosch Group is a leading global supplier of technology and services, in the areas of Automotive Technology, Industrial Technology, Consumer Goods, Energy and Building Technology. In India, the Group operates through nine companies with a combined strength of over 30,000 associates which includes around 14,000 research and development associates. Bosch Automotive Electronics India Pvt. Ltd. (RBAI) is a 100% subsidiary of Robert Bosch GmbH. RBAI was established at the right time to cater to the demands of future Indian market. Established in 2009, started out with manufacturing Electronic Control Units. On an average adding one new product every year, Antenna and Immobilizer in 2011, wide range of BCM's since 2012, Electronic power steering control units from 2013, and Voltage regulator in 2014. Over the last 7 years of its existence, the company has grown over 44% CAGR, which is remarkable considering it was established during the peak of recession. The product portfolio of Bosch Automotive Electronics Pvt. Ltd. is into both Automotive and Non-Automotive Business catering to local as well as global demands. The products from RBAI fulfils 94% of the local demand. Apart from this, 72% of our sales are towards exports covering most of the global market. We invite promising and dynamic professionals for a long-term and rewarding career with Bosch. Job Description As a Data engineer in Operations, you will work on the operational management, monitoring, and support of scalable data pipelines running in Azure Databricks, Hadoop and Radium. You will ensure the reliability, performance, and availability of data workflows and maintain production environments. You will collaborate closely with data engineers, architects, and platform teams to implement best practices in data pipeline operations and incident management to ensure data availability and data completeness. Primary responsibilities: Operational support and incident management for Azure Databricks, Hadoop, Radium data pipelines. Collaborating with data engineering and platform teams to define and enforce operational standards, SLAs, and best practices. Designing and implementing monitoring, alerting, and logging solutions for Azure Databricks pipelines. Coordinating with central teams to ensure compliance with organizational operational standards and security policies. Developing and maintaining runbooks, SOPs, and troubleshooting guides for pipeline issues. Managing the end-to-end lifecycle of data pipeline incidents, including root cause analysis and remediation. Overseeing pipeline deployments, rollbacks, and change management using CI/CD tools such as Azure DevOps. Ensuring data quality and validation checks are effectively monitored in production. Working closely with platform and infrastructure teams to address pipeline and environment-related issues. Providing technical feedback and mentoring junior operations engineers. Conducting peer reviews of operational scripts and automation code. Automating manual operational tasks using Scala and Python scripts. Managing escalations and coordinating critical production issue resolution. Participating in post-mortem reviews and continuous improvement initiatives for data pipeline operations. · Qualifications Bachelor’s degree in Computer Science, Computer Engineering, or a relevant technical field 3+ years’ experience in data engineering, ETL tools, and working with large-scale data sets in Operations. Proven experience with cloud platforms, particularly Azure Databricks. Minimum 3 years of hands-on experience working with distributed cluster environments (e.g., Spark clusters). Strong operational experience in managing and supporting data pipelines in production environments. Additional Information Key Competencies: Experience in Azure Databricks operations or data pipeline support. Understanding of Scala/ Python programming for troubleshooting in Spark environments. Hands-on experience with Delta Lake, Azure Data Lake Storage (ADLS), DBFS, Azure Data Factory (ADF). Solid understanding of distributed data processing frameworks and streaming data operations. Understanding and hands-on usage of Kafka as message broker Experience with Azure SQL Database and cloud-based data services. Strong skills in monitoring tools like Splunk, ELK and Grafana, alerting frameworks, and incident management. Experience working with CI/CD pipelines using Azure DevOps or equivalent. Excellent problem-solving, investigative, and troubleshooting skills in large-scale data environments. Experience defining operational SLAs and implementing proactive monitoring solutions. Familiarity with data governance, security, and compliance best practices in cloud data platforms. Strong communication skills and ability to work independently under pressure. Soft Skills: Good Communication Skills, extensive usage of MS-Teams Experience in using Azure board and JIRA Decent Level in English as Business Language

Posted 1 week ago

Apply

8.0 years

0 Lacs

Bengaluru

On-site

Imagine what you could do here. At Apple, we believe new insights have a way of becoming excellent products, services, and customer experiences very quickly. Bring passion and dedication to your job and there's no telling what you could accomplish. The people here at Apple don’t just build products - they build the kind of wonder that’s revolutionized entire industries. It’s the diversity of those people and their ideas that inspires the innovation that runs through everything we do, from amazing technology to industry-leading environmental efforts. Join Apple, and help us leave the world better than we found it. Apple's Manufacturing Systems and Infrastructure (MSI) team is responsible for capturing, consolidating and tracking all manufacturing data for Apple’s products and modules worldwide. Our tools enable teams to confidently use data to shape the next generation of product manufacturing at Apple. We seek a practitioner with experience building large-scale data platforms, analytic tools, and solutions. If you are passionate about making data easily accessible, trusted, and available across the entire business at scale, we'd love to hear from you. As a Software Engineering Manager, you are an integral part of a data-centric team driving large- scale data infrastructure and processes development, implementation, and improvement. Our organization thrives on collaborative partnerships. Join and play a key role in developing and driving the adoption of Agentic AI, LLMs, Data Mesh and data-centric micro-services. Description As an Engineering Manager, you will lead a team of engineers responsible for the development and implementation of our cloud-based data infrastructure. You will work closely with cross-functional teams to understand data requirements, design scalable solutions, and ensure the integrity and availability of our data. The ideal candidate will have a deep understanding of cloud technologies, data engineering best practices, and a proven track record of successfully delivering complex data projects. Key Responsibilities include: - Hire, develop, and retain top engineering talent - Build and nurture self-sustained, high-performing teams - Provide mentorship and technical guidance to engineers, fostering continuous learning and development - Lead the design, development, and deployment of scalable cloud-based data infrastructure and applications - Drive end-to-end execution of complex data engineering projects - Partner with Data Scientists, ML Engineers, and business stakeholders to understand data needs and translate them into scalable engineering solutions - Align technical strategy with business goals through effective communication and collaboration - Implement and enforce best practices for data security, privacy, and compliance with regulatory standards - Optimize data storage, processing, and retrieval for improved performance and cost efficiency. - Continuously evaluate and improve the system architecture and workflows - Stay current with emerging trends and technologies in cloud data engineering - Recommend and adopt tools, frameworks, and platforms that enhance productivity and reliability Minimum Qualifications Bachelor’s degree in Computer Science or a related field Minimum 8 years of experience in software development with at least 2 years in a technical leadership or management role. Proven experience as a Full stack developer, with a focus on cloud platforms. Proficient in programming languages such as Python. Strong hands-on expertise with Python frameworks (Django, Flask, or FastAPI, RESTful APIs), React.js and modern JavaScript Experience with authentication and authorization (OAuth, JWT) Strong understanding of cloud services, preferably AWS & Experience in building cloud native platforms using containerization technologies like Kubernetes, docker, helm Preferred Qualifications Knowledge of data warehouse solutions (BigQuery, Snowflake, Druid) and Big Data technologies such as Spark, Kafka, Hive, Iceberg, Trino, Flink. Experience with big data technologies (Hadoop, Spark, etc.). Experience with streaming data technologies (Kafka, Kinesis). Experience building data streaming solutions using Apache Spark / Apache Storm / Flink / Flume. Familiarity with machine learning pipelines is an added advantage. Proven ability to deliver complex, high-scale systems in a production environment. Strong people management and cross-functional collaboration skills. Submit CV

Posted 1 week ago

Apply

0 years

3 - 10 Lacs

Bengaluru

On-site

Employment Type Permanent Closing Date 13 June 2025 11:59pm Job Title IT Domain Specialist Job Summary As the IT Domain Specialist, your role is key in improving the stability and reliability of our cloud offerings and solutions to ensure continuity of service for our customers. You will be responsible for supporting the end-to-end development of key cloud platform and solutions which includes technical design, integration requirements, delivery and lifecycle management. You are a specialist across and/or within a technology domain and viewed as the go-to person in the business to provide technical support in the development and delivery of cloud infrastructure platforms and solutions. Job Description Who We Are Telstra is Australia’s leading telecommunications and technology company spanning over a century with a footprint in over 20+ countries. In India, we’re building a platform for innovative delivery and engagement that will strengthen our position as an industry leader. We’ve grown quickly since our inception in 2019, now with offices in Pune, Hyderabad and Bangalore. Focus of the Role Event Data Engineer role is to plan, coordinate, and execute all activities related to the requirements interpretation, design and implementation of Business intelligence capability. This individual will apply proven industry and technology experience as well as communication skills, problem-solving skills, and knowledge of best practices to issues related to design, development, and deployment of mission-critical business systems with a focus on quality application development and delivery. What We Offer Performance-related pay Access to thousands of learning programs so you can level-up Global presence across 22 countries; opportunities to work where we do business. Up to 26 weeks maternity leave provided to the birth mother with benefits for all child births Employees are entitled to 12 paid holidays per calendar year Eligible employees are entitled to 12 days of paid sick / casual leave per calendar year Relocation support options across India, from junior to senior positions within the company Receive insurance benefits such as medical, accidental and life insurances What You’ll Do Experience in Analysis, Design, and Development in the fields of Business Intelligence, Databases and Web-based Applications. Experience in NiFi, Kafka, Spark, and Cloudera Platforms design and development. Experience in Alteryx Workflow development and Data Visualization development using Tableau to create complex, intuitive dashboards. In-depth understanding and experience in Cloudera framework includes CDP (Cloudera Data Platform). Experience in Cloudera manager to monitor Hadoop cluster and critical services . Hadoop administration ( Hive, Kafka, zookeeper etc.). Experience in data management including data integration, modeling, optimization and data quality. Strong knowledge in writing SQL and database management. Working experience in tools like Alteryx , KNIME will be added advantage. Implementing Data security and access control compliant to Telstra Security Standards Ability to review vendor designs and recommended solutions based on industry best practises Understand overall business operations and develops innovative solutions to help improve productivity Ability to understand and design provisioning solutions at Telstra and how Data lakes Monitor process of software configuration/development/testing to assure quality deliverable. Ensure standards of QA are being met Review deliverables to verify that they meet client and contract expectations; Implement and enforce high standards for quality deliverables Analyses performance and capacity issues of the highest complexity with Data applications. Assists leadership with development and management of new application capabilities to improve productivity Provide training and educate other team members around core capabilities and helps them deliver high quality solutions and deliverables/documentation Self-Motivator to perform Design / Develop user requirements, test and deploy the changes into production. About You Experience in data flow development and Data Visualization development to create complex, intuitive dashboards. Experience with Hortonworks Data Flow (HDF) this includes NiFi and Kafka experience with Cloudera Edge Big Data & Data Lake Experience Cloudera Hadoop with project implementation experience Data Analytics experience Data Analyst and Data Science exposure Exposure to various data management architectures like data warehouse, data lake and data hub, and supporting processes like data integration, data modeling. Working experience with large, heterogeneous datasets in building and optimizing data pipelines, pipeline architectures and integrated datasets using data integration technologies Experience in supporting operations and knowledge of standard operation procedures: OS Patches, Security Scan, Log Onboarding, Agent Onboarding, Log Extraction etc. Development and deployment and scaling of containerised applications with Docker preferred. A good understanding of enterprise application integration, including SOA, ESB, EAI, ETL environments and an understanding of integration considerations such as process orchestration, customer data integration and master data management A good understanding of the security processes, standards & issues involved in multi-tier, multi-tenant web applications We're amongst the top 2% of companies globally in the CDP Global Climate Change Index 2023, being awarded an 'A' rating. If you want to work for a company that cares about sustainability, we want to hear from you. As part of your application with Telstra, you may receive communications from us on +61 440 135 548 (for job applications in Australia) and +1 (623) 400-7726 (for job applications in the Philippines and India). When you join our team, you become part of a welcoming and inclusive community where everyone is respected, valued and celebrated. We actively seek individuals from various backgrounds, ethnicities, genders and disabilities because we know that diversity not only strengthens our team but also enriches our work. We have zero tolerance for harassment of any kind, and we prioritise creating a workplace culture where everyone is safe and can thrive. As part of the hiring process, all identified candidates will undergo a background check, and the results will play a role in the final decision regarding your application. We work flexibly at Telstra. Talk to us about what flexibility means to you. When you apply, you can share your pronouns and / or any reasonable adjustments needed to take part equitably during the recruitment process. We are aware of current limitations with our website accessibility and are working towards improving this. Should you experience any issues accessing information or the application form, and require this in an alternate format, please contact our Talent Acquisition team on DisabilityandAccessibility@team.telstra.com.

Posted 1 week ago

Apply

0 years

3 - 10 Lacs

Bengaluru

On-site

Employment Type Permanent Closing Date 13 June 2025 11:59pm Job Title IT Domain Specialist Job Summary As the IT Domain Specialist, your role is key in improving the stability and reliability of our cloud offerings and solutions to ensure continuity of service for our customers. You will be responsible for supporting the end-to-end development of key cloud platform and solutions which includes technical design, integration requirements, delivery and lifecycle management. You are a specialist across and/or within a technology domain and viewed as the go-to person in the business to provide technical support in the development and delivery of cloud infrastructure platforms and solutions. Job Description Who We Are Telstra is Australia’s leading telecommunications and technology company spanning over a century with a footprint in over 20+ countries. In India, we’re building a platform for innovative delivery and engagement that will strengthen our position as an industry leader. We’ve grown quickly since our inception in 2019, now with offices in Pune, Hyderabad and Bangalore. Focus of the Role Event Data Engineer role is to plan, coordinate, and execute all activities related to the requirements interpretation, design and implementation of Business intelligence capability. This individual will apply proven industry and technology experience as well as communication skills, problem-solving skills, and knowledge of best practices to issues related to design, development, and deployment of mission-critical business systems with a focus on quality application development and delivery. What We Offer Performance-related pay Access to thousands of learning programs so you can level-up Global presence across 22 countries; opportunities to work where we do business. Up to 26 weeks maternity leave provided to the birth mother with benefits for all child births Employees are entitled to 12 paid holidays per calendar year Eligible employees are entitled to 12 days of paid sick / casual leave per calendar year Relocation support options across India, from junior to senior positions within the company Receive insurance benefits such as medical, accidental and life insurances What You’ll Do Experience in Analysis, Design, and Development in the fields of Business Intelligence, Databases and Web-based Applications. Experience in NiFi, Kafka, Spark, and Cloudera Platforms design and development. Experience in Alteryx Workflow development and Data Visualization development using Tableau to create complex, intuitive dashboards. In-depth understanding and experience in Cloudera framework includes CDP (Cloudera Data Platform). Experience in Cloudera manager to monitor Hadoop cluster and critical services . Hadoop administration ( Hive, Kafka, zookeeper etc.). Experience in data management including data integration, modeling, optimization and data quality. Strong knowledge in writing SQL and database management. Working experience in tools like Alteryx , KNIME will be added advantage. Implementing Data security and access control compliant to Telstra Security Standards Ability to review vendor designs and recommended solutions based on industry best practises Understand overall business operations and develops innovative solutions to help improve productivity Ability to understand and design provisioning solutions at Telstra and how Data lakes Monitor process of software configuration/development/testing to assure quality deliverable. Ensure standards of QA are being met Review deliverables to verify that they meet client and contract expectations; Implement and enforce high standards for quality deliverables Analyses performance and capacity issues of the highest complexity with Data applications. Assists leadership with development and management of new application capabilities to improve productivity Provide training and educate other team members around core capabilities and helps them deliver high quality solutions and deliverables/documentation Self-Motivator to perform Design / Develop user requirements, test and deploy the changes into production. About You Experience in data flow development and Data Visualization development to create complex, intuitive dashboards. Experience with Hortonworks Data Flow (HDF) this includes NiFi and Kafka experience with Cloudera Edge Big Data & Data Lake Experience Cloudera Hadoop with project implementation experience Data Analytics experience Data Analyst and Data Science exposure Exposure to various data management architectures like data warehouse, data lake and data hub, and supporting processes like data integration, data modeling. Working experience with large, heterogeneous datasets in building and optimizing data pipelines, pipeline architectures and integrated datasets using data integration technologies Experience in supporting operations and knowledge of standard operation procedures: OS Patches, Security Scan, Log Onboarding, Agent Onboarding, Log Extraction etc. Development and deployment and scaling of containerised applications with Docker preferred. A good understanding of enterprise application integration, including SOA, ESB, EAI, ETL environments and an understanding of integration considerations such as process orchestration, customer data integration and master data management A good understanding of the security processes, standards & issues involved in multi-tier, multi-tenant web applications We're amongst the top 2% of companies globally in the CDP Global Climate Change Index 2023, being awarded an 'A' rating. If you want to work for a company that cares about sustainability, we want to hear from you. As part of your application with Telstra, you may receive communications from us on +61 440 135 548 (for job applications in Australia) and +1 (623) 400-7726 (for job applications in the Philippines and India). When you join our team, you become part of a welcoming and inclusive community where everyone is respected, valued and celebrated. We actively seek individuals from various backgrounds, ethnicities, genders and disabilities because we know that diversity not only strengthens our team but also enriches our work. We have zero tolerance for harassment of any kind, and we prioritise creating a workplace culture where everyone is safe and can thrive. As part of the hiring process, all identified candidates will undergo a background check, and the results will play a role in the final decision regarding your application. We work flexibly at Telstra. Talk to us about what flexibility means to you. When you apply, you can share your pronouns and / or any reasonable adjustments needed to take part equitably during the recruitment process. We are aware of current limitations with our website accessibility and are working towards improving this. Should you experience any issues accessing information or the application form, and require this in an alternate format, please contact our Talent Acquisition team on DisabilityandAccessibility@team.telstra.com.

Posted 1 week ago

Apply

5.0 years

6 - 7 Lacs

Bengaluru

On-site

About Tarento: Tarento is a fast-growing technology consulting company headquartered in Stockholm, with a strong presence in India and clients across the globe. We specialize in digital transformation, product engineering, and enterprise solutions, working across diverse industries including retail, manufacturing, and healthcare. Our teams combine Nordic values with Indian expertise to deliver innovative, scalable, and high-impact solutions. We're proud to be recognized as a Great Place to Work , a testament to our inclusive culture, strong leadership, and commitment to employee well-being and growth. At Tarento, you’ll be part of a collaborative environment where ideas are valued, learning is continuous, and careers are built on passion and purpose. Role Overview An Azure Data Engineer specializing in Databricks is responsible for designing, building, and maintaining scalable data solutions on the Azure cloud platform, with a focus on leveraging Databricks and related big data technologies. The role involves close collaboration with data scientists, analysts, and software engineers to ensure efficient data processing, integration, and delivery for analytics and business intelligence needshttps://en.wizbii.com/company/dxc-technology/job/azure-data-bricks-developerhttps://www.expertia.ai/blogs/jd/data-engineer-databricks-job-description-12901ghttps://www.lafosse.com/job-description/azure-data-engineer-job-description/. Key Responsibilities Design, develop, and maintain robust and scalable data pipelines using Azure Databricks, Azure Data Factory, and other Azure services. Build and optimize data architectures to support large-scale data processing and analytics. Collaborate with cross-functional teams to gather requirements and deliver data solutions tailored to business needs. Ensure data quality, integrity, and security across various data sources and pipelines. Implement data governance, compliance, and best practices for data security (e.g., encryption, RBAC). Monitor, troubleshoot, and optimize data pipeline performance, ensuring reliability and scalability. Document technical specifications, data pipeline processes, and architectural decisions Support and troubleshoot data workflows, ensuring consistent data delivery and availability for analytics and reporting Automate data tasks and deploy production-ready code using CI/CD practices Stay updated with the latest Azure and Databricks features, recommending improvements and adopting new tools as appropriate Required Skills and Qualifications Bachelor’s degree in Computer Science, Engineering, or a related field 5+ years of experience in data engineering, with hands-on expertise in Azure and Databricks environments Proficiency in Databricks, Apache Spark, and Spark SQL Strong programming skills in Python and/or Scala Advanced SQL skills and experience with relational and NoSQL databases Experience with ETL processes, data warehousing concepts, and big data technologies (e.g., Hadoop, Kafka) Familiarity with Azure services: Azure Data Lake Storage (ADLS), Azure Data Factory, Azure SQL Data Warehouse, Cosmos DB, Azure Stream Analytics, Azure Functions Understanding of data modeling, schema design, and data integration best practices Strong analytical, problem-solving, and troubleshooting abilities Experience with source code control systems (e.g., GIT) and technical documentation tools Excellent communication and collaboration skills; ability to work both independently and as part of a team Preferred Skills Experience with automation, unit testing, and CI/CD pipelines Certifications in Azure Data Engineering or Databricks are advantageous Soft Skills Flexible, self-starter, and proactive in learning and adopting new technologies Ability to manage multiple priorities and work to tight deadlines Strong stakeholder management and teamwork capabilities

Posted 1 week ago

Apply

2.0 years

6 - 7 Lacs

Bengaluru

On-site

Company Description Visa is a world leader in payments and technology, with over 259 billion payments transactions flowing safely between consumers, merchants, financial institutions, and government entities in more than 200 countries and territories each year. Our mission is to connect the world through the most innovative, convenient, reliable, and secure payments network, enabling individuals, businesses, and economies to thrive while driven by a common purpose – to uplift everyone, everywhere by being the best way to pay and be paid. Make an impact with a purpose-driven industry leader. Join us today and experience Life at Visa. Job Description Functional Summary The GTM Optimization and Business Health team has a simple mission: we turn massive amounts of data into robust tools and actionable insights that drive business value, ensure ecosystem integrity, and provide best in class experience to our money movement clients. Our team is working to build consolidated, strategic and scalable analytics and monitoring infrastructure for commercial and money movement products. Responsibilities The Process Optimization Analyst will create risk, rules, and performance monitoring dashboards and alerting tools and will use these to monitor transactions in near real time, investigate alerts and anomalous events, and partner with internal teams to investigate and manage incidents from end-to-end. Specific activities may include: Develop monitoring and alerting tools from real-time data feeds to monitor for performance drops, risk and fraud events, and rules violations Monitor near real time alerting tools and investigate and generate incidents for risk events and out of pattern activity Manage a caseload to ensure appropriate investigation and resolution of identified risk and performance events Drive to understand the root problems, define analytical objectives and formalize data requirements for various types of dashboards and analyses Design and launch robust and intuitive dashboards supporting best in class money movement client experience Create and present analytic deliverables to colleagues in the analytics team, other internal stakeholders with varying degrees of analytical and technical expertise Distill massive amounts of data across disparate data sources into efficient functional data repositories in a Big Data environment Independently perform analysis to derive insights and render robust, thoughtful results Partner with Visa Direct and money movement teams across multiple areas of the business to understand their data and reporting needs Compare client performance against industry best practices with a shrewd eye toward identifying performance and/or profitability improvement opportunity Develop presentations of complex data and content for clients in an accurate, understandable, and engaging manner This is a hybrid position. Expectation of days in office will be confirmed by your Hiring Manager. Qualifications Basic Qualifications: 3 or more years of relevant work experience with a Bachelor’s Degree or at least 2 years of work experience with an Advanced degree (e.g. Masters, MBA, JD, MD) or 0 years of work experience with a PhD Preferred Qualifications: 3 or more years of work experience with a Bachelor’s Degree or 2 or more years of relevant experience with an Advanced Degree (e.g. Masters, MBA, JD, MD) or up to 1 years of relevant experience with a PhD Experience monitoring real-time data and following incident management workflows Familiarity with Microsoft Dynamics or other ERP/CRM tools Proficiency in Tableau and experience with best in class data visualization Experience with Elasticsearch and Kibana dashboard and alerting High level of proficiency manipulating data from a variety of sources - Big data skills (Hadoop, Hive, Spark) and/or SQL skills required Strong verbal, written, and interpersonal skills Proficient in all MS Office applications with advanced Excel spreadsheet skills Functional knowledge of programming languages such as Python, Java, and/or Shell Scripting Strong strategic thinking, problem-solving, and decision-making abilities, with the ability to translate complex data into actionable insights Visa experience or knowledge of the payments industry Additional Information Visa is an EEO Employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability or protected veteran status. Visa will also consider for employment qualified applicants with criminal histories in a manner consistent with EEOC guidelines and applicable local law.

Posted 1 week ago

Apply

4.0 years

10 - 17 Lacs

India

On-site

We are looking for an Only immediate joiner and e*xperienced Big Data Developer with a strong background in PySpark, Python/Scala, Spark, SQL, and the Hadoop ecosystem. The ideal candidate should have over 4 years of experience and be ready to join immediately.* This role requires hands-on expertise in big data technologies and the ability to design and implement robust data processing solutions. Key Responsibilities: Design, develop, and optimize large-scale data processing pipelines using PySpark. Work with various Apache tools and frameworks (like Hadoop, Hive, HDFS, etc.) to ingest, transform, and manage large datasets. Ensure high performance and reliability of ETL jobs in production. Collaborate with Data Scientists, Analysts, and other stakeholders to understand data needs and deliver robust data solutions. Implement data quality checks and data lineage tracking for transparency and auditability. Work on data ingestion, transformation, and integration from multiple structured and unstructured sources. Leverage Apache NiFi for automated and repeatable data flow management (if applicable). Write clean, efficient, and maintainable code in Python and Java. Contribute to architectural decisions, performance tuning, and scalability planning. Required Skills: 5–7 years of experience. Strong hands-on experience with PySpark for distributed data processing. Deep understanding of Apache ecosystem (Hadoop, Hive, Spark, HDFS, etc.). Solid grasp of data warehousing, ETL principles, and data modeling. Experience working with large-scale datasets and performance optimization. Familiarity with SQL and NoSQL databases. Proficiency in Python and basic to intermediate knowledge of Java. Experience in using version control tools like Git and CI/CD pipelines. Nice-to-Have Skills: Working experience with Apache NiFi for data flow orchestration. Experience in building real-time streaming data pipelines. Knowledge of cloud platforms like AWS, Azure, or GCP. Familiarity with containerization tools like Docker or orchestration tools like Kubernetes. Soft Skills: Strong analytical and problem-solving skills. Excellent communication and collaboration abilities. Self-driven with the ability to work independently and as part of a team. Education: Bachelor’s or Master’s degree in Computer Science, Information Systems, or a related field. Job Type: Full-time Pay: ₹1,000,000.00 - ₹1,700,000.00 per year Benefits: Health insurance Schedule: Day shift Supplemental Pay: Performance bonus Yearly bonus Ability to commute/relocate: Basavanagudi, Bengaluru, Karnataka: Reliably commute or planning to relocate before starting work (Preferred) Application Question(s): Are you ready to join within 15 days? What is your Current CTC ? Experience: Python: 4 years (Preferred) Pyspark: 4 years (Required) Data warehouse: 4 years (Required) Work Location: In person Application Deadline: 12/06/2025

Posted 1 week ago

Apply

5.0 years

5 - 9 Lacs

Chennai

On-site

Job ID: 18927 Location: Chennai, IN Area of interest: Technology Job type: Regular Employee Work style: Office Working Opening date: 15 May 2025 Job Summary Responsible for Building and maintaining high-performance data systems that enable deeper insights for all parts of our organization Responsible for Developing ETL/ELT pipelines for both batch and streaming data Responsible for Data flow for the real-time and analytics Improving data pipelines performance by implementing the industry’s best practices and different techniques for data parallel processing Responsible for the documentation, design, development and testing of Hadoop reporting and analytical application. Responsible for Technical discussion and finalization of the requirement by communicating effectively with Stakeholder. Responsible for converting functional requirements into the detailed technical design Responsible for adhering to SCRUM timelines and deliver accordingly Responsible for preparing the Unit/SIT/UAT test cases and log the results Responsible for Planning and tracking the implementation to closure Ability to drive enterprise-wide initiatives for usage of external data Envision enterprise-wide Entitlement’s platform and align it with Bank’s NextGen technology vision. Continually looking for process improvements Coordinate between various technical teams for various systems for smooth project execution starting from technical requirements discussion, overall architecture design, technical solution discussions, build, unit testing, regression testing, system integration testing, user acceptance testing, go live, user verification testing and rollback [if required] Prepare technical plan with clear milestone dates for technical tasks which will be input to the PM’s overall project plan. Coordinate with technical teams across technology on need basis who are not directly involved in the project example: Firewall network teams, DataPower teams, EDMP , OAM, OIM, ITSC , GIS teams etc. Responsible to support change management process Responsible to work alongside PSS teams and ensure proper KT sessions are provided to the support teams. Ensure to identify any risks within the project and get that recorded in Risk wise after discussion with business and manager. Ensure the project delivery is seamless with zero to negligible defects. Key Responsibilities Hands on experience with C++, .Net, SQL Language, jQuery, Web API & Service, Postgres SQL & MS SQL server, Azure Dev Ops & related, GitHub, ADO CI/CD Pipeline Should be transversal to handle Linux, PowerShell, Unix shell scripting, Kafka, Spark streaming Hadoop – Hive, Spark, Python, PYSpark Hands on experience of workflow/schedulers like NIFI/Ctrl-m Experience with Data loading tools like sqoop Experience and understanding of Object-oriented programming Motivation to learn innovative trade of programming, debugging, and deploying Self-starter, with excellent self-study skills and growth aspirations, capable of working without direction and able to deliver technical projects from scratch Excellent written and verbal communication skills. Flexible attitude, perform under pressure Ability to lead and influence direction and strategy of technology organization Test driven development, commitment to quality and a thorough approach to work A good team player with ability to meet tight deadlines in a fast-paced environment Guide junior’s developers and share the best practices Having Cloud certification will be an added advantage: any one of Azure/Aws/GCP Must have Knowledge & understanding of Agile principles Must have good understanding of project life cycle Must have Sound problem analysis and resolution abilities Good understanding of External & Internal Data Management & implications of Cloud usage in context of external data Strategy Develop the strategic direction and roadmap for CRES TTO, aligning with Business Strategy, ITO Strategy and investment priorities. Business Work hand in hand with Product Owners, Business Stakeholders, Squad Leads, CRES TTO partners taking product programs from investment decisions into design, specifications, solutioning, development, implementation and hand-over to operations, securing support and collaboration from other SCB teams Ensure delivery to business meeting time, cost and high quality constraints Support respective businesses in growing Return on investment, commercialisation of capabilities, bid teams, monitoring of usage, improving client experience, enhancing operations and addressing defects & continuous improvement of systems Thrive an ecosystem of innovation and enabling business through technology Governance Promote an environment where compliance with internal control functions and the external regulatory framework People & Talent Ability to work with other developers and assist junior team members. Identify training needs and take action to ensure company-wide compliance. Pursue continuing education on new solutions, technology, and skills. Problem solving with other team members in the project. Risk Management Interpreting briefs to create high-quality coding that functions according to specifications. Key stakeholders CRES Domain Clients Functions MT members, Operations and COO ITO engineering, build and run teams Architecture and Technology Support teams Supply Chain Management, Risk, Legal, Compliance and Audit teams External vendors Regulatory & Business Conduct Display exemplary conduct and live by the Group’s Values and Code of Conduct. Take personal responsibility for embedding the highest standards of ethics, including regulatory and business conduct, across Standard Chartered Bank. This includes understanding and ensuring compliance with, in letter and spirit, all applicable laws, regulations, guidelines and the Group Code of Conduct. Lead the team to achieve the outcomes set out in the Bank’s Conduct Principles: [Fair Outcomes for Clients; Effective Financial Markets; Financial Crime Compliance; The Right Environment.] * Effectively and collaboratively identify, escalate, mitigate and resolve risk, conduct and compliance matters. Serve as a Director of the Board Exercise authorities delegated by the Board of Directors and act in accordance with Articles of Association (or equivalent) Other Responsibilities Embed Here for good and Group’s brand and values in team Perform other responsibilities assigned under Group, Country, Business or Functional policies and procedures Multiple functions (double hats) Skills and Experience Technical Project Delivery (Agile & Classic) Vendor Management Stakeholder Management Qualifications 5+ years of lead development role Should have managed a team of minimum 5 members Should have delivered multiple projects end to end Experience in Property Technology products (eg. Lenel, CBRE, Milestone etc) Strong analytical, numerical and problem-solving skills Should be able to understand and communicate technical details of the project Good communication skills – oral and written. Very good exposure to technical projects Eg: server maintenance, system administrator or development or implementation experience Effective interpersonal, relational skills to be able to coach and develop the team to deliver their best Certified Scrum Master About Standard Chartered We're an international bank, nimble enough to act, big enough for impact. For more than 170 years, we've worked to make a positive difference for our clients, communities, and each other. We question the status quo, love a challenge and enjoy finding new opportunities to grow and do better than before. If you're looking for a career with purpose and you want to work for a bank making a difference, we want to hear from you. You can count on us to celebrate your unique talents and we can't wait to see the talents you can bring us. Our purpose, to drive commerce and prosperity through our unique diversity, together with our brand promise, to be here for good are achieved by how we each live our valued behaviours. When you work with us, you'll see how we value difference and advocate inclusion. Together we: Do the right thing and are assertive, challenge one another, and live with integrity, while putting the client at the heart of what we do Never settle, continuously striving to improve and innovate, keeping things simple and learning from doing well, and not so well Are better together, we can be ourselves, be inclusive, see more good in others, and work collectively to build for the long term What we offer In line with our Fair Pay Charter, we offer a competitive salary and benefits to support your mental, physical, financial and social wellbeing. Core bank funding for retirement savings, medical and life insurance, with flexible and voluntary benefits available in some locations. Time-off including annual leave, parental/maternity (20 weeks), sabbatical (12 months maximum) and volunteering leave (3 days), along with minimum global standards for annual and public holiday, which is combined to 30 days minimum. Flexible working options based around home and office locations, with flexible working patterns. Proactive wellbeing support through Unmind, a market-leading digital wellbeing platform, development courses for resilience and other human skills, global Employee Assistance Programme, sick leave, mental health first-aiders and all sorts of self-help toolkits A continuous learning culture to support your growth, with opportunities to reskill and upskill and access to physical, virtual and digital learning. Being part of an inclusive and values driven organisation, one that embraces and celebrates our unique diversity, across our teams, business functions and geographies - everyone feels respected and can realise their full potential. www.sc.com/careers

Posted 1 week ago

Apply

8.0 years

7 - 8 Lacs

Chennai

On-site

Comcast brings together the best in media and technology. We drive innovation to create the world's best entertainment and online experiences. As a Fortune 50 leader, we set the pace in a variety of innovative and fascinating businesses and create career opportunities across a wide range of locations and disciplines. We are at the forefront of change and move at an amazing pace, thanks to our remarkable people, who bring cutting-edge products and services to life for millions of customers every day. If you share in our passion for teamwork, our vision to revolutionize industries and our goal to lead the future in media and technology, we want you to fast-forward your career at Comcast. Job Summary Responsible for planning and designing new software and web applications. Analyzes, tests and assists with the integration of new applications. Oversees the documentation of all development activity. Trains non-technical personnel. Assists with tracking performance metrics. Integrates knowledge of business and functional priorities. Acts as a key contributor in a complex and crucial environment. May lead teams or projects and shares expertise. Job Description Core Responsibilities 8+ years development and data engineering experience. Proficiency in programming languages such as Python, Bash, PySpark Experience with ETL frameworks like Apache Spark, Airflow, or similar tools. Data pipeline experience with common pipeline and management tools Familiarity with data lake architectures and big data technologies (AWS Security Lake, Data Bricks, Snowflake, Hadoop) in a large, complex deployment. Strong knowledge of Data modeling, SQL and relational databases. Knowledge of data processing frameworks and data manipulation libraries Experience with cloud computing platforms (e.g. AWS)• Direct experience building systems in AWS and using devops toolchains including (Git, GitHub Actions, Jenkins, CodePipeline, Azure devops, etc.). Familiarity with Serverless services like AWS Lambda. Knowledge of microservices architecture and containerization technologies. Highly collaborative; personally, and professionally self-aware; able to and interested in interacting with employees at all levels; embody integrity; and represent and inspire the highest ethical standards. A thirst for improvement and an inclination to thoughtfully challenge the status quo. Desire to try things and iterate on them, fail fast, and focus on functionality that matters. Eagerness to learn new security tools/services to support broadening our portfolio. Disclaimer: This information has been designed to indicate the general nature and level of work performed by employees in this role. It is not designed to contain or be interpreted as a comprehensive inventory of all duties, responsibilities and qualifications. Comcast is proud to be an equal opportunity workplace. We will consider all qualified applicants for employment without regard to race, color, religion, age, sex, sexual orientation, gender identity, national origin, disability, veteran status, genetic information, or any other basis protected by applicable law. Base pay is one part of the Total Rewards that Comcast provides to compensate and recognize employees for their work. Most sales positions are eligible for a Commission under the terms of an applicable plan, while most non-sales positions are eligible for a Bonus. Additionally, Comcast provides best-in-class Benefits to eligible employees. We believe that benefits should connect you to the support you need when it matters most, and should help you care for those who matter most. That’s why we provide an array of options, expert guidance and always-on tools, that are personalized to meet the needs of your reality – to help support you physically, financially and emotionally through the big milestones and in your everyday life. Please visit the compensation and benefits summary on our careers site for more details. Education Bachelor's Degree While possessing the stated degree is preferred, Comcast also may consider applicants who hold some combination of coursework and experience, or who have extensive related professional experience. Relevant Work Experience 7-10 Years

Posted 1 week ago

Apply

2.0 - 9.0 years

4 - 5 Lacs

Coimbatore

On-site

At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. EY_ Consulting _Data Analytics Testing_Staff The opportunity As a Data Analytics Test Engineer, you will be responsible for testing Business Intelligence & Data warehousing Solutions both in on premise and cloud platform and should ensure Quality of deliverables. You will work closely with Test Lead for the Projects under Test. Testing proficiency in ETL, data-warehousing, and Business Intelligence area are required for this position. Added advantage to have experience in testing Big Data/unstructured data using Hadoop/Spark framework, cloud platform knowledge either in AWS/Azure, knowledge in predictive analytics, Machine Learning and Artificial intelligence. Skills and attributes for success Delivery of Testing needs for BI & DWH Projects. Ability to effectively communicate with team members across geographies effectively Perform unstructured data / big data testing both in on-premise and cloud platform. Thorough understanding of Requirements and provide feedback on the requirements. Develop Test Strategy for Testing BI & DWH Projects for various aspects like ETL testing & Reports testing (Front end and Backend Testing), Integration Testing and UAT as needed. Provide inputs for Test Planning aligned with Test Strategy. Perform Test Case design, identify opportunity for Test Automation. Develop Test Cases both Manual and Automation Scripts as required. Ensure Test readiness (Test Environment, Test Data, Tools Licenses etc) Perform Test execution and report the progress. Report defects and liaise with development & other relevant team for defect resolution. Prepare Test Report and provide inputs to Test Lead for Test Sign off/ Closure Provide support in Project meetings/ calls with Client for status reporting. Provide inputs on Test Metrics to Test Lead. Support in Analysis of Metric trends and implementing improvement actions as necessary. Handling changes and conducting Regression Testing Generate Test Summary Reports Co-coordinating Test team members and Development team Interacting with client-side people to solve issues and update status Actively take part in providing Analytics and Advanced Analytics Testing trainings in the company To qualify for the role, you must have BE/BTech/MCA/M.Sc Overall 2 to 9 years of experience in Testing Data warehousing / Business Intelligence solutions, minimum 2 years of experience in Testing BI & DWH technologies and Analytics applications. Experience in Bigdata testing with Hadoop/Spark framework and exposure to predictive analytics testing. Very good understanding of business intelligence concepts, architecture & building blocks in areas ETL processing, Datawarehouse, dashboards and analytics. Experience in cloud AWS/Azure infrastructure testing is desirable. Knowledge on python data processing is desirable. Testing experience in more than one of these areas- Data Quality, ETL, OLAP, Reports Good working experience with SQL server or Oracle database and proficiency with SQL scripting. Experience in backend Testing of Enterprise Applications/ Systems built on different platforms including Microsoft .Net and Sharepoint technologies Experience in ETL Testing using commercial ETL tools is desirable. Knowledge/ experience in SSRS, Spotfire (SQL Server Reporting Services) and SSIS is desirable. Experience/ Knowledge in Data Transformation Projects, database design concepts & white-box testing is desirable. Ideally, you’ll also have Able to contribute as an individual contributor and when required Lead a small Team Able to create Test Strategy & Test Plan for Testing BI & DWH applications/ solutions that are moderate to complex / high risk Systems Design Test Cases, Test Data and perform Test Execution & Reporting. Should be able to perform Test Management for small Projects as and when required Participate in Defect Triaging and track the defects for resolution/ conclusion Experience/ exposure to Test Automation and scripting experience in perl & shell is desirable Experience with Test Management and Defect Management tools preferably HP ALM Good communication skills (both written & verbal) Good understanding of SDLC, test process in particular Good analytical & problem solving or troubleshooting skills Good understanding of Project Life Cycle and Test Life Cycle. Exposure to CMMi and Process improvement Frameworks is a plus. Should have excellent communication skills & should be able to articulate concisely & clearly Should be ready to do an individual contributor as well as Team Leader role What working at EY offers At EY, we’re dedicated to helping our clients, from start–ups to Fortune 500 companies — and the work we do with them is as varied as they are. You get to work with inspiring and meaningful projects. Our focus is education and coaching alongside practical experience to ensure your personal development. We value our employees and you will be able to control your own development with an individual progression plan. You will quickly grow into a responsible role with challenging and stimulating assignments. Moreover, you will be part of an interdisciplinary environment that emphasizes high quality and knowledge exchange. Plus, we offer: Support, coaching and feedback from some of the most engaging colleagues around Opportunities to develop new skills and progress your career The freedom and flexibility to handle your role in a way that’s right for you EY | Building a better working world EY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets. Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate. Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.

Posted 1 week ago

Apply

2.0 years

0 Lacs

Trivandrum, Kerala, India

On-site

Linkedin logo

What you’ll do Perform general application development activities, including unit testing, code deployment to development environment and technical documentation. Works on one or more projects, making contributions to unfamiliar code written by team members. Participates in estimation process, use case specifications, reviews of test plans and test cases, requirements, and project planning. Diagnose and resolve performance issues. Documents code/processes so that any other developer is able to dive in with minimal effort. Develop, and operate high scale applications from the backend to UI layer, focusing on operational excellence, security and scalability. Apply modern software development practices (serverless computing, microservices architecture, CI/CD, infrastructure-as-code, etc.). Work across teams to integrate our systems with existing internal systems, Data Fabric, CSA Toolset. Participate in technology roadmap and architecture discussions to turn business requirements and vision into reality. Participate in a tight-knit engineering team employing agile software development practices. Triage product or system issues and debug/track/resolve by analyzing the sources of issues and the impact on network, or service operations and quality. Able to write, debug, and troubleshoot code in mainstream open source technologies. Lead effort for Sprint deliverables, and solve problems with medium complexity What Experience You Need Bachelor's degree or equivalent experience 2+ years experience working with software design and Java, SQL and Javascript programming languages 2+ years experience with software testing, performance, and quality engineering techniques and strategies 2+ years experience with Cloud technology: GCP, AWS, or Azure What Could Set You Apart: Knowledge or experience with Apache Beam for stream and batch data processing. Familiarity with big data tools and technologies like Apache Kafka, Hadoop, or Spark. Experience with containerization and orchestration tools (e.g., Docker, Kubernetes). Exposure to data visualization tools or platforms. Show more Show less

Posted 1 week ago

Apply

3.0 years

0 Lacs

Andhra Pradesh

On-site

We are looking for a PySpark solutions developer and data engineer who can design and build solutions for one of our Fortune 500 Client programs, which aims towards building a data standardized and curation needs on Hadoop cluster. This is high visibility, fast-paced key initiative will integrate data across internal and external sources, provide analytical insights, and integrate with the customers critical systems. Key Responsibilities Ability to design, build and unit test applications on Spark framework on Python. Build PySpark based applications for both batch and streaming requirements, which will require in-depth knowledge on majority of Hadoop and NoSQL databases as well. Develop and execute data pipeline testing processes and validate business rules and policies. Optimize performance of the built Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame, and Pair RDD's. Optimize performance for data access requirements by choosing the appropriate native Hadoop file formats (Avro, Parquet, ORC etc) and compression codec respectively. Build integrated solutions leveraging Unix shell scripting, RDBMS, Hive, HDFS File System, HDFS File Types, HDFS compression codec. Build data tokenization libraries and integrate with Hive & Spark for column-level obfuscation. Experience in processing large amounts of structured and unstructured data, including integrating data from multiple sources. Create and maintain integration and regression testing framework on Jenkins integrated with Bit Bucket and/or GIT repositories. Participate in the agile development process, and document and communicate issues and bugs relative to data standards in scrum meetings. Work collaboratively with onsite and offshore team. Develop & review technical documentation for artifacts delivered. Ability to solve complex data-driven scenarios and triage towards defects and production issues. Ability to learn-unlearn-relearn concepts with an open and analytical mindset. Participate in code release and production deployment. Challenge and inspire team members to achieve business results in a fast paced and quickly changing environment. Preferred Qualifications BE/B.Tech/ B.Sc. in Computer Science/ Statistics from an accredited college or university. Minimum 3 years of extensive experience in design, build and deployment of PySpark-based applications. Expertise in handling complex large-scale Big Data environments preferably (20Tb+). Minimum 3 years of experience in the following: HIVE, YARN, HDFS. Hands-on experience writing complex SQL queries, exporting, and importing large amounts of data using utilities. Ability to build abstracted, modularized reusable code components. Prior experience on ETL tools preferably Informatica PowerCenter is advantageous. Able to quickly adapt and learn. Able to jump into an ambiguous situation and take the lead on resolution. Able to communicate and coordinate across various teams. Are comfortable tackling new challenges and new ways of working Are ready to move from traditional methods and adapt into agile ones Comfortable challenging your peers and leadership team. Can prove yourself quickly and decisively. Excellent communication skills and Good Customer Centricity. Strong Target & High Solution Orientation. About Virtusa Teamwork, quality of life, professional and personal development: values that Virtusa is proud to embody. When you join us, you join a team of 27,000 people globally that cares about your growth — one that seeks to provide you with exciting projects, opportunities and work with state of the art technologies throughout your career with us. Great minds, great potential: it all comes together at Virtusa. We value collaboration and the team environment of our company, and seek to provide great minds with a dynamic place to nurture new ideas and foster excellence. Virtusa was founded on principles of equal opportunity for all, and so does not discriminate on the basis of race, religion, color, sex, gender identity, sexual orientation, age, non-disqualifying physical or mental disability, national origin, veteran status or any other basis covered by appropriate law. All employment is decided on the basis of qualifications, merit, and business need.

Posted 1 week ago

Apply

0 years

0 Lacs

Chennai, Tamil Nadu, India

On-site

Linkedin logo

Job Description Enthusiastic and self-motivated, with ability to execute Supply Chain Analytics projects proactively Meticulous attention to detail, with an overall passion for continuous improvement Innovative and creative, with a logical and methodical approach to problem solving Credible and articulate, with excellent communication, presentation, and interpersonal skills Responsibilities Execute high impact business projects with time bound and effective project management leveraging tools like Rally, Jira Gather business requirements and convert them into analytical problems and identify relevant tools, techniques, and an overall framework to provide solutions Use statistical methodologies leveraging analytical tools to support different business initiatives Continual enhancement of statistical techniques and their applications in solving business objectives Compile and analyze the results from modeling output and translate into actionable insights through dashboards Acquire and share deep knowledge of data utilized by the team and its business partners Participate in global conference calls and meetings as needed and manage multiple customer interfaces Execute analytics special studies and ad hoc analyses in a quick turn around time Evaluate new tools and technologies to improve analytical processes Efforts will focus on the following key areas: Domain – Supply Chain Analytics Hands on with Machine Learning Good understanding of various classical Statistical techniques such as Regression, Multivariate Analysis Data & Text Mining, NLP, Gen AI, Large Language Models, Time Series based forecasting modeling Experience with SQL and data warehousing (e.g. GCP/Hadoop/Teradata/Oracle/DB2) Experience using tools in BI, ETL, Reporting /Visualization/Dashboards - Qliksense/Power BI etc. Programming experience in languages like Python Exposure to Bigdata based analytical solutions Good Soft Skills - able to communicate clearly with stakeholders Good Analysis and problem-solving skills. Ability to get Insights from Data, provide visualization, and storytelling. Flexibility to explore and work with newer technologies and tools. Ability to learn quickly, adapt, and set direction when faced with ambiguous scenarios. Excellent collaborative communication and Team skills Qualifications Bachelors/Masters Candidates should have significant hands-on experience with analytics projects or related quantitative techniques across various functions Candidates will be expected to successfully prioritize and manage multiple analytical projects Good technical depth with strong analytical/programing skills and ability to apply technical knowledge Experience in Python, SQL, GCP or any other Cloud Platforms highly desired Show more Show less

Posted 1 week ago

Apply

5.0 years

0 Lacs

Bangalore Urban, Karnataka, India

On-site

Linkedin logo

Job Title: Data Analyst / Technical Business Analyst Job Summary We are looking for a skilled Data Analyst to support a large-scale data migration initiative within the banking and insurance domain. The role involves analyzing, validating, and transforming data from legacy systems to modern platforms, ensuring regulatory compliance, data integrity, and business continuity. Key Responsibilities Collaborate with business stakeholders, data architects, and IT teams to gather and understand data migration requirements. Analyze legacy banking and insurance systems (e.g., core banking, policy admin, claims, CRM) to identify data structures and dependencies. Work with large-scale datasets and understand big data architectures (e.g., Hadoop, Spark, Hive) to support scalable data migration and transformation. Perform data profiling, cleansing, and transformation using SQL and ETL tools, with the ability to understand and write complex SQL queries and interpret the logic implemented in ETL workflows. Develop and maintain data mapping documents and transformation logic specific to financial and insurance data (e.g., customer KYC, transactions, policies, claims). Validate migrated data against business rules, regulatory standards, and reconciliation reports. Support UAT by preparing test cases and validating migrated data with business users. Ensure data privacy and security compliance throughout the migration process. Document issues, risks, and resolutions related to data quality and migration. Required Skills & Qualifications Bachelor’s degree in Computer Science, Information Systems, Finance, or a related field. 5+ years of experience in data analysis or data migration projects in banking or insurance. Strong SQL skills and experience with data profiling and cleansing. Familiarity with ETL tools (e.g., Informatica, Talend, SSIS) and data visualization tools (e.g., Power BI, Tableau). Experience working with big data platforms (e.g., Hadoop, Spark, Hive) and handling large volumes of structured and unstructured data. Understanding of banking and insurance data domains (e.g., customer data, transactions, policies, claims, underwriting). Knowledge of regulatory and compliance requirements (e.g., AML, KYC, GDPR, IRDAI guidelines). Excellent analytical, documentation, and communication skills. Preferred Qualifications Experience with core banking systems (e.g., Finacle, Flexcube) or insurance platforms Exposure to cloud data platforms (e.g.,AWS, Azure, GCP). Experience working in Agile/Scrum environments. Certification in Business Analysis (e.g., CBAP, CCBA) or Data Analytics. Show more Show less

Posted 1 week ago

Apply

0 years

0 Lacs

Thiruvananthapuram, Kerala, India

Remote

Linkedin logo

Brief Description The Cloud Data Engineer will play a critical implementation role on the Data Engineering and Data Products team and be responsible for data pipeline solutions design and development, troubleshooting, and optimization tuning on the next generation data and analytics platform being developed with leading edge big data technologies in a highly secure cloud infrastructure. The Cloud Data Engineer will serve as a liaison to platform user groups ensuring successful implementation of capabilities on the new platform. Data Engineer Responsibilities : Deliver end-to-end data and analytics capabilities, including data ingest, data transformation, data science, and data visualization in collaboration with Data and Analytics stakeholder groups Design and deploy databases and data pipelines to support analytics projects Develop scalable and fault-tolerant workflows Clearly document issues, solutions, findings and recommendations to be shared internally & externally Learn and apply tools and technologies proficiently, including: Languages: Python, PySpark, ANSI SQL, Python ML libraries Frameworks/Platform: Spark, Snowflake, Airflow, Hadoop , Kafka Cloud Computing: AWS Tools/Products: PyCharm, Jupyter, Tableau, PowerBI Performance optimization for queries and dashboards Develop and deliver clear, compelling briefings to internal and external stakeholders on findings, recommendations, and solutions Analyze client data & systems to determine whether requirements can be met Test and validate data pipelines, transformations, datasets, reports, and dashboards built by team Develop and communicate solutions architectures and present solutions to both business and technical stakeholders Provide end user support to other data engineers and analysts Candidate Requirements : Expert experience in the following[ Should have / Good to have ]: SQL, Python, PySpark, Python ML libraries. Other programming languages (R, Scala, SAS, Java, etc.) are a plus Data and analytics technologies including SQL/NoSQL/Graph databases, ETL, and BI Knowledge of CI/CD and related tools such as Gitlab, AWS CodeCommit etc. AWS services including EMR, Glue, Athena, Batch, Lambda CloudWatch, DynamoDB, EC2, CloudFormation, IAM and EDS Exposure to Snowflake and Airflow. Solid scripting skills (e.g., bash/shell scripts, Python) Proven work experience in the following: Data streaming technologies Big Data technologies including, Hadoop, Spark, Hive, Teradata, etc. Linux command-line operations Networking knowledge (OSI network layers, TCP/IP, virtualization) Candidate should be able to lead the team, communicate with business, gather and interpret business requirements Experience with agile delivery methodologies using Jira or similar tools Experience working with remote teams AWS Solutions Architect / Developer / Data Analytics Specialty certifications, Professional certification is a plus Bachelor Degree in Computer Science relevant field, Masters Degree is a plus Show more Show less

Posted 1 week ago

Apply

4.0 - 6.0 years

0 Lacs

Mumbai Metropolitan Region

On-site

Linkedin logo

Responsible for developing, optimize, and maintaining business intelligence and data warehouse systems, ensuring secure, efficient data storage and retrieval, enabling self-service data exploration, and supporting stakeholders with insightful reporting and analysis. Grade - T5 Please note that the Job will close at 12am on Posting Close date, so please submit your application prior to the Close Date Accountabilities What your main responsibilities are: Data Pipeline - Develop and maintain scalable data pipelines and builds out new API integrations to support continuing increases in data volume and complexity Data Integration - Connect offline and online data to continuously improve overall understanding of customer behavior and journeys for personalization. Data pre-processing including collecting, parsing, managing, analyzing and visualizing large sets of data Data Quality Management - Cleanse the data and improve data quality and readiness for analysis. Drive standards, define and implement/improve data governance strategies and enforce best practices to scale data analysis across platforms Data Transformation - Processes data by cleansing data and transforming them to proper storage structure for the purpose of querying and analysis using ETL and ELT process Data Enablement - Ensure data is accessible and useable to wider enterprise to enable a deeper and more timely understanding of operation. Qualifications & Specifications Masters /Bachelor’s degree in Engineering /Computer Science/ Math/ Statistics or equivalent. Strong programming skills in Python/Pyspark/SAS. Proven experience with large data sets and related technologies – Hadoop, Hive, Distributed computing systems, Spark optimization. Experience on cloud platforms (preferably Azure) and it's services Azure Data Factory (ADF), ADLS Storage, Azure DevOps. Hands-on experience on Databricks, Delta Lake, Workflows. Should have knowledge of DevOps process and tools like Docker, CI/CD, Kubernetes, Terraform, Octopus. Hands-on experience with SQL and data modeling to support the organization's data storage and analysis needs. Experience on any BI tool like Power BI (Good to have). Cloud migration experience (Good to have) Cloud and Data Engineering certification (Good to have) Working in an Agile environment 4-6 Years Of Relevant Work Experience Is Required. Experience with stakeholder management is an added advantage. What We Are Looking For Education: Bachelor's degree or equivalent in Computer Science, MIS, Mathematics, Statistics, or similar discipline. Master's degree or PhD preferred. Knowledge, Skills And Abilities Fluency in English Analytical Skills Accuracy & Attention to Detail Numerical Skills Planning & Organizing Skills Presentation Skills Data Modeling and Database Design ETL (Extract, Transform, Load) Skills Programming Skills FedEx was built on a philosophy that puts people first, one we take seriously. We are an equal opportunity/affirmative action employer and we are committed to a diverse, equitable, and inclusive workforce in which we enforce fair treatment, and provide growth opportunities for everyone. All qualified applicants will receive consideration for employment regardless of age, race, color, national origin, genetics, religion, gender, marital status, pregnancy (including childbirth or a related medical condition), physical or mental disability, or any other characteristic protected by applicable laws, regulations, and ordinances. Our Company FedEx is one of the world's largest express transportation companies and has consistently been selected as one of the top 10 World’s Most Admired Companies by "Fortune" magazine. Every day FedEx delivers for its customers with transportation and business solutions, serving more than 220 countries and territories around the globe. We can serve this global network due to our outstanding team of FedEx team members, who are tasked with making every FedEx experience outstanding. Our Philosophy The People-Service-Profit philosophy (P-S-P) describes the principles that govern every FedEx decision, policy, or activity. FedEx takes care of our people; they, in turn, deliver the impeccable service demanded by our customers, who reward us with the profitability necessary to secure our future. The essential element in making the People-Service-Profit philosophy such a positive force for the company is where we close the circle, and return these profits back into the business, and invest back in our people. Our success in the industry is attributed to our people. Through our P-S-P philosophy, we have a work environment that encourages team members to be innovative in delivering the highest possible quality of service to our customers. We care for their well-being, and value their contributions to the company. Our Culture Our culture is important for many reasons, and we intentionally bring it to life through our behaviors, actions, and activities in every part of the world. The FedEx culture and values have been a cornerstone of our success and growth since we began in the early 1970’s. While other companies can copy our systems, infrastructure, and processes, our culture makes us unique and is often a differentiating factor as we compete and grow in today’s global marketplace. Show more Show less

Posted 1 week ago

Apply

1.0 years

0 Lacs

Hyderabad, Telangana, India

On-site

Linkedin logo

Description Amazon is a place where data drives most of our decision-making. Analytics, Operations & Programs (AOP) team is looking for a dynamic data engineer who can be innovative, strong problem solver and can lead the implementation of the analytical data infrastructure that will guide the decision making. As a Data Engineer, you think like an entrepreneur, constantly innovating and driving positive change, but more importantly, you consistently deliver mind-boggling results. You're a leader, who uses both quantitative and qualitative methods to get things done. And on top of it all, you're someone who wonders "What if?" and then seeks out the solution. This position offers exceptional opportunities to grow their technical and non-technical skills. You have the opportunity to really make a difference to our business by inventing, enhancing and building world class systems, delivering results, working on exciting and challenging projects. As a Data Engineer, you are responsible for analyzing large amounts of business data, solve real world problems, and develop metrics and business cases that will enable us to continually delight our customers worldwide. This is done by leveraging data from various platforms such as Jira, Portal, Salesforce. You will work with a team of Product Managers, Software Engineers and Business Intelligence Engineers to automate and scale the analysis, and to make the data more actionable to manage business at scale. You will own many large datasets, implement new data pipelines that feed into or from critical data systems at Amazon. You must be able to prioritize and work well in an environment with competing demands. Successful candidates will bring strong technical abilities combined with a passion for delivering results for customers, internal and external. This role requires a high degree of ownership and a drive to solve some of the most challenging data and analytic problems in retail. Candidates must have demonstrated ability to manage large-scale data modeling projects, identify requirements and tools, build data warehousing solutions that are explainable and scalable. In addition to the technical skills, a successful candidate will possess strong written and verbal communication skills and a high intellectual curiosity with ability to learn new concepts/frameworks and technology rapidly as changes arise. Key job responsibilities Design, implement and support an analytical data infrastructure Managing AWS resources including EC2, EMR, S3, Glue, Redshift, etc. Interface with other technology teams to extract, transform, and load data from a wide variety of data sources using SQL and AWS big data technologies Explore and learn the latest AWS technologies to provide new capabilities and increase efficiency Collaborate with Data Scientists and Business Intelligence Engineers (BIEs) to recognize and help adopt best practices in reporting and analysis Help continually improve ongoing reporting and analysis processes, automating or simplifying self-service support for customers Maintain internal reporting platforms/tools including troubleshooting and development. Interact with internal users to establish and clarify requirements in order to develop report specifications. Work with Engineering partners to help shape and implement the development of BI infrastructure including Data Warehousing, reporting and analytics platforms. Contribute to the development of the BI tools, skills, culture and impact. Write advanced SQL queries and Python code to develop solutions A day in the life This role requires you to live at the intersection of data, software, and analytics. We leverage a comprehensive suite of AWS technologies, with key tools including S3, Redshift, DynamoDB, Lambda, API's, Glue. You will drive the development process from design to release. Managing data ingestion from heterogeneous data sources, with automated data quality checks. Creating scalable data models for effective data processing, storage, retrieval, and archiving. Using scripting for automation and tool development, which is scalable, reusable, and maintainable. Providing infrastructure for self serve analytics and science use cases. Using industry best practices in building CI/CD pipelines About The Team AOP (Analytics Operations and Programs) team is missioned to standardize BI and analytics capabilities, and reduce repeat analytics/reporting/BI workload for operations across IN, AU, BR, MX, SG, AE, EG, SA marketplace. AOP is responsible to provide visibility on operations performance and implement programs to improve network efficiency and defect reduction. The team has a diverse mix of strong engineers, Analysts and Scientists who champion customer obsession. We enable operations to make data-driven decisions through developing near real-time dashboards, self-serve dive-deep capabilities and building advanced analytics capabilities. We identify and implement data-driven metric improvement programs in collaboration (co-owning) with Operations teams Basic Qualifications 1+ years of data engineering experience Experience with data modeling, warehousing and building ETL pipelines Experience with one or more query language (e.g., SQL, PL/SQL, DDL, MDX, HiveQL, SparkSQL, Scala) Experience with one or more scripting language (e.g., Python, KornShell) Preferred Qualifications Experience with big data technologies such as: Hadoop, Hive, Spark, EMR Experience with any ETL tool like, Informatica, ODI, SSIS, BODI, Datastage, etc. Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit https://amazon.jobs/content/en/how-we-hire/accommodations for more information. If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner. Company - ASSPL - Karnataka Job ID: A2904529 Show more Show less

Posted 1 week ago

Apply

1.0 years

0 Lacs

Hyderabad, Telangana, India

On-site

Linkedin logo

About The Job Job Title: AI/ML Developer About The Role Duration: 12 Months Location: PAN INDIA Timings: Full Time (As per company timings) Notice Period: within 15 days or immediate joiner Experience: 1- 3 years Key Responsibilities Design and deploy ML models focused on NLP and Computer Vision. Handle data labelling, preprocessing, and model validation. Assist in API development to integrate ML models into apps. Fine-tune and train models to improve performance. Collaborate with teams to deliver practical AI-driven solutions. Maintain documentation of model processes and outcomes. Required Skills & Qualification 1- 3 years in AI/ML development with hands-on work in NLP or CV. Strong Python skills with libraries like TensorFlow, PyTorch, scikit-learn. Experience in data preprocessing and model deployment. Exposure to cloud platforms (AWS/GCP/Azure) is a plus. Familiarity with MLOps and API integration is desirable. Degree in Computer Science, Data Science, or related field. Knowledge of big data tools (Spark/Hadoop) or other languages (R/Java/C++) is an advantage. Show more Show less

Posted 1 week ago

Apply

3.0 years

0 Lacs

Andhra Pradesh, India

On-site

Linkedin logo

We are looking for a PySpark solutions developer and data engineer who can design and build solutions for one of our Fortune 500 Client programs, which aims towards building a data standardized and curation needs on Hadoop cluster. This is high visibility, fast-paced key initiative will integrate data across internal and external sources, provide analytical insights, and integrate with the customers critical systems. Key Responsibilities Ability to design, build and unit test applications on Spark framework on Python. Build PySpark based applications for both batch and streaming requirements, which will require in-depth knowledge on majority of Hadoop and NoSQL databases as well. Develop and execute data pipeline testing processes and validate business rules and policies. Build integrated solutions leveraging Unix shell scripting, RDBMS, Hive, HDFS File System, HDFS File Types, HDFS compression codec. Create and maintain integration and regression testing framework on Jenkins integrated with Bit Bucket and/or GIT repositories. Participate in the agile development process, and document and communicate issues and bugs relative to data standards in scrum meetings. Work collaboratively with onsite and offshore team. Develop & review technical documentation for artifacts delivered. Ability to solve complex data-driven scenarios and triage towards defects and production issues. Ability to learn-unlearn-relearn concepts with an open and analytical mindset. Participate in code release and production deployment. Preferred Qualifications BE/B.Tech/ B.Sc. in Computer Science/ Statistics from an accredited college or university. Minimum 3 years of extensive experience in design, build and deployment of PySpark-based applications. Expertise in handling complex large-scale Big Data environments preferably (20Tb+). Minimum 3 years of experience in the following: HIVE, YARN, HDFS. Hands-on experience writing complex SQL queries, exporting, and importing large amounts of data using utilities. Ability to build abstracted, modularized reusable code components. Prior experience on ETL tools preferably Informatica PowerCenter is advantageous. Able to quickly adapt and learn. Able to jump into an ambiguous situation and take the lead on resolution. Able to communicate and coordinate across various teams. Are comfortable tackling new challenges and new ways of working Are ready to move from traditional methods and adapt into agile ones Comfortable challenging your peers and leadership team. Can prove yourself quickly and decisively. Excellent communication skills and Good Customer Centricity. Strong Target & High Solution Orientation. Show more Show less

Posted 1 week ago

Apply

3.0 years

0 Lacs

Andhra Pradesh, India

On-site

Linkedin logo

We are looking for a PySpark solutions developer and data engineer who can design and build solutions for one of our Fortune 500 Client programs, which aims towards building a data standardized and curation needs on Hadoop cluster. This is high visibility, fast-paced key initiative will integrate data across internal and external sources, provide analytical insights, and integrate with the customers critical systems. Key Responsibilities Ability to design, build and unit test applications on Spark framework on Python. Build PySpark based applications for both batch and streaming requirements, which will require in-depth knowledge on majority of Hadoop and NoSQL databases as well. Develop and execute data pipeline testing processes and validate business rules and policies. Optimize performance of the built Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame, and Pair RDD's. Optimize performance for data access requirements by choosing the appropriate native Hadoop file formats (Avro, Parquet, ORC etc) and compression codec respectively. Build integrated solutions leveraging Unix shell scripting, RDBMS, Hive, HDFS File System, HDFS File Types, HDFS compression codec. Build data tokenization libraries and integrate with Hive & Spark for column-level obfuscation. Experience in processing large amounts of structured and unstructured data, including integrating data from multiple sources. Create and maintain integration and regression testing framework on Jenkins integrated with Bit Bucket and/or GIT repositories. Participate in the agile development process, and document and communicate issues and bugs relative to data standards in scrum meetings. Work collaboratively with onsite and offshore team. Develop & review technical documentation for artifacts delivered. Ability to solve complex data-driven scenarios and triage towards defects and production issues. Ability to learn-unlearn-relearn concepts with an open and analytical mindset. Participate in code release and production deployment. Challenge and inspire team members to achieve business results in a fast paced and quickly changing environment. Preferred Qualifications BE/B.Tech/ B.Sc. in Computer Science/ Statistics from an accredited college or university. Minimum 3 years of extensive experience in design, build and deployment of PySpark-based applications. Expertise in handling complex large-scale Big Data environments preferably (20Tb+). Minimum 3 years of experience in the following: HIVE, YARN, HDFS. Hands-on experience writing complex SQL queries, exporting, and importing large amounts of data using utilities. Ability to build abstracted, modularized reusable code components. Prior experience on ETL tools preferably Informatica PowerCenter is advantageous. Able to quickly adapt and learn. Able to jump into an ambiguous situation and take the lead on resolution. Able to communicate and coordinate across various teams. Are comfortable tackling new challenges and new ways of working Are ready to move from traditional methods and adapt into agile ones Comfortable challenging your peers and leadership team. Can prove yourself quickly and decisively. Excellent communication skills and Good Customer Centricity. Strong Target & High Solution Orientation. Show more Show less

Posted 1 week ago

Apply

5.0 years

0 Lacs

Gurugram, Haryana, India

On-site

Linkedin logo

Description The AOP (Analytics Operations and Programs) team is responsible for creating core analytics, insight generation and science capabilities for ROW Ops. We develop scalable analytics applications, AI/ML products and research models to optimize operation processes. You will work with Product Managers, Data Engineers, Data Scientists, Research Scientists, Applied Scientists and Business Intelligence Engineers using rigorous quantitative approaches to ensure high quality data/science products for our customers around the world. We are looking for a Sr.Data Scientist to join our growing Science Team. As Data Scientist, you are able to use a range of science methodologies to solve challenging business problems when the solution is unclear. You will be responsible for building ML models to solve complex business problems and test them in production environment. The scope of role includes defining the charter for the project and proposing solutions which align with org's priorities and production constraints but still create impact. You will achieve this by leveraging strong leadership and communication skills, data science skills and by acquiring domain knowledge pertaining to the delivery operations systems. You will provide ML thought leadership to technical and business leaders, and possess ability to think strategically about business, product, and technical challenges. You will also be expected to contribute to the science community by participating in science reviews and publishing in internal or external ML conferences. Our Team Solves a Broad Range Of Problems That Can Be Scaled Across ROW (Rest Of The World Including Countries Like India, Australia, Singapore, MENA And LATAM). Here Is a Glimpse Of The Problems That This Team Deals With On a Regular Basis Using live package and truck signals to adjust truck capacities in real-time HOTW models for Last Mile Channel Allocation Using LLMs to automate analytical processes and insight generation Ops research to optimize middle mile truck routes Working with global partner science teams to affect Reinforcement Learning based pricing models and estimating Shipments Per Route for $MM savings Deep Learning models to synthesize attributes of addresses Abuse detection models to reduce network losses Key job responsibilities Use machine learning and analytical techniques to create scalable solutions for business problems Analyze and extract relevant information from large amounts of Amazon’s historical business data to help automate and optimize key processes Design, develop, evaluate and deploy, innovative and highly scalable ML/OR models Work closely with other science and engineering teams to drive real-time model implementations Work closely with Ops/Product partners to identify problems and propose machine learning solutions Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model maintenance Work proactively with engineering teams and product managers to evangelize new algorithms and drive the implementation of large-scale complex ML models in production Leading projects and mentoring other scientists, engineers in the use of ML techniques Basic Qualifications 5+ years of data scientist experience Experience with data scripting languages (e.g. SQL, Python, R etc.) or statistical/mathematical software (e.g. R, SAS, or Matlab) Experience with statistical models e.g. multinomial logistic regression Experience in data applications using large scale distributed systems (e.g., EMR, Spark, Elasticsearch, Hadoop, Pig, and Hive) Experience working with data engineers and business intelligence engineers collaboratively Demonstrated expertise in a wide range of ML techniques Preferred Qualifications Experience as a leader and mentor on a data science team Master's degree in a quantitative field such as statistics, mathematics, data science, business analytics, economics, finance, engineering, or computer science Expertise in Reinforcement Learning and Gen AI is preferred Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit https://amazon.jobs/content/en/how-we-hire/accommodations for more information. If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner. Company - Amazon Development Centre (India) Private Limited Job ID: A3003385 Show more Show less

Posted 1 week ago

Apply

0 years

0 Lacs

Anupgarh, Rajasthan, India

On-site

Linkedin logo

글로벌 콘텐츠 플랫폼 기업, SpoonLabs와 함께 하실 Data Engineer 를 지금 찾고 있어요 | 오디오(Spoon)와 비디오(Vigloo)를 아우르는 콘텐츠 플랫폼으로의 더 큰 도약을 위해, 스푼라디오에서 스푼랩스로 사명을 변경하였습니다 | 🧑‍🤝‍🧑 [SpoonLabs Data Tech 팀을 소개합니다] Data Tech 팀은 이런 일을 해요! 스푼과 비글루에서 발생하는 음성, 텍스트, 이미지, 영상 등 모든 데이터를 활용하는 팀입니다. 한국, 일본, 대만, 영미국, 아랍권 국가에서 발생하는 전세계 데이터를 활용한 다양한 유형의 규모가 큰 데이터 처리 뿐만 아니라 데이터 기반의 의사 결정 지원을 위한 시각화 서비스 및 ML / AI 를 활용한 추천 시스템 및 서비스 품질을 개선하기 위한 어뷰징 시스템 개발에 이르기까지 데이터가 필요한 다양한 영역에서 활용하고 있습니다. Data Tech 팀은 이런 문화로 일해요! 다양한 아이디어 및 신기술에 대해 적극 공유하고 의견을 나눕니다. 이렇게 모인 아이디어를 기반으로 도전적인 자세로서 PoC를 진행합니다. 처음 마주한 문제점 혹은 질문이 있다면 언제든 질문하고 빠르게 답변 받을 수 있습니다. 설계된 Architecture 혹은 개발된 코드는 팀 리뷰 혹은 온라인 리뷰를 통해 더욱 더 건설적인 피드백을 받을 수 있습니다. 여기에 팀의 생산성을 높이기 위한 상용 AI 서비스를 아낌없이 지원합니다. Data Tech 팀은 문제 해결 중심의 사고방식을 지향합니다. 다만, 개인 보다는 팀으로서 서로가 잘하는 부분을 함께 활용하여 더 큰 시너지를 낼 수 있는 방법을 찾으려고 합니다. 그 해결방법이 굳이 ML이 아니어도 됩니다. 빠른 문제 해결이 가능하다면 ML 을 비롯한 다양한 방법을 함께 찾고, 해결하려고 노력합니다. 또한, 다양한 부서와의 커뮤니케이션을 통해 활용 가능한 신기술 혹은 데이터를 선제적으로 사용해보고 실제 구현된 Demo를 제시합니다. Data Tech 팀은 이런 기술 스택을 사용해요! Hadoop Ecosystem Hive Spark Airflow Metabase Jupyter PostgreSQL / MariaDB Grafana / Alert Manager Ranger [AWS] GLUE EC2 S3 EKS EMR Open Search (Elastic Search) [Language] Python Java or Kotlin SQL 스푼랩스가 만드는 글로벌 콘텐츠 플랫폼 Spoon & Vigloo! 오디오부터 비디오까지, 스푼랩스는 전 세계 사람들의 일상을 재밌는 콘텐츠로 가득 채우고 있습니다. 크리에이터의 오디오 라이브 콘텐츠 플랫폼, Spoon - https://www.spooncast.net/kr 2분의 몰입을 선사하는 숏폼 드라마 플랫폼, Vigloo - https://www.vigloo.com/ko 💼 [주요 업무 - 주로 이런 업무를 수행해요] Big-data 기반 OpenSource(s)가 설치된 On-Premise Cluster를 관리 및 운용합니다 AWS Cloud 기반의 환경에서 Glue EC2 EKS 등을 활용하여 데이터 수집 및 처리 서비스를 운용하고 관리합니다 서비스에서 발생한 Database Log 및 3rd-Party Service에서 수집합니다 수집된 다양한 유형의 데이터(정형 비정형 반정형)를 전처리하여 정형화 및 표준화합니다 정형화된 데이터를 기반으로 SQL Query를 활용하여 타 부서에 데이터를 제공 및 시각화된 대시보드를 제공합니다 📌 [자격 요건 - 이런 분과 함께하고 싶어요] 5년 이상 데이터 엔지니어 업무 경력 Hadoop ecosystem 기반의 Open Source 설치 및 운영 경험 Spark 기반의 ETL 처리 경험 Airflow 기반의 스케쥴링 및 DAG 구성 경험 Ansi-SQL 기반의 쿼리 작성 및 고급 함수 사용 경험 Python Java Kotlin 등 프로그래밍 언어 개발 경험 데이터 파이프라인 운영 및 모니터링 등의 경험 ➕ [우대 사항 - 이런 분은 더욱 반가워요] Glue EC2 EKS 등 AWS 서비스 경험이 있으신 분 쿼리 실행 계획 및 로그 등을 통한 ETL 성능 최적화 경험 ML Engineer와의 개발 및 협업 경험 개발 비개발 직군을 모두 포함하여 문제점 논의 및 해결 방안에 대한 설명 등 원활한 커뮤니케이션 경험 스푼 서비스를 적극적으로 사용해보신 분 📑 [제출 서류 - 지원자님을 알기 위해서는 다음 서류가 필요해요] 이력서(필수 제출) 이력서 외 추가로 공유하고 싶은 자료가 있다면 함께 제출하셔도 무방합니다. 🎯 [채용 전형 - 다음 과정을 거쳐 스푼랩스에 합류하게 돼요] 서류 전형 > 1차 직무 인터뷰 전형 > 2차 컬처핏 & 3차 경영진 인터뷰 전형 > 레퍼런스 체크 전형 > 처우 협의 > 최종 합격 및 입사 1차 직무 인터뷰 전형 스푼랩스 Development Group 실무진과의 직무 인터뷰를 진행합니다. 대면으로 진행되며, 예상 소요시간은 약 1시간 30분입니다. 2차 컬처핏 인터뷰 전형 스푼랩스 EX팀(인사팀)과 컬처핏 인터뷰를 진행합니다. 대면으로 진행되며, 예상 소요시간은 약 1시간입니다. 2차 인터뷰가 끝나고 잠깐 휴식을 하신 이후, 3차 인터뷰가 바로 진행됩니다. (2, 3차 인터뷰는 하루에 연이어 진행됩니다.) 3차 경영진 인터뷰 전형 스푼랩스 개발 그룹 리드와 경영진 인터뷰를 진행합니다. 대면으로 진행되며, 예상 소요시간은 약 1시간입니다. 레퍼런스 체크 전형 > 처우 협의 > 최종 합격 및 입사  상황에 따라 채용 절차가 생략 혹은 추가될 수 있습니다. (과제 전형/코딩 테스트/커피챗/추가 인터뷰 등) 이력서 및 제출 서류에 허위 사실이 발견되거나 근무 이력 중 징계사항이 확인될 경우, 채용이 취소될 수 있습니다. 스푼랩스 취업규칙 제10조(채용결격)에 따라 결격사유에 해당하는 자는 채용이 취소될 수 있습니다. 👀 [스푼랩스는 어떻게 일하나요? 여기에 답이 있어요] 우리는 더 빠르게, 더 치열하게, 더 단단하게. 완벽보다 속도, 완성보다 실행. 스푼랩스는 빠르게 시도하고, 실패는 안고 다시 달립니다. 결국 답에 닿을 때까지, 우리는 계속 몰입합니다. 반짝이는 아이디어, 밤을 새워 몰입했던 순간, 깨달음에서 전율을 느끼던 날들. 속도는 빠르고, 기준은 높고, 불확실성은 큽니다. 누군가에게 이곳은 버거울 수 있지만, 우리는 그 안에서 치열하게 성장하며 매일 조금씩 더 나아갑니다. 더 멀리, 더 빠르게 나아가기 위해 어깨를 맞대되 각자의 빛을 잃지 않고, 앞서가되 독주하지 않습니다. 함께할 때 더 단단하기에, 우리는 서로를 믿고 배우며, 겸손하게 성장합니다. 원하는 게 ‘편한 일’인지, ‘치열한 성장’인지 스스로에게 물어보세요. 우리는 이미 ‘치열한 성장’을 선택했습니다. 단순한 일이 아닌, 내 인생과 세상을 바꾸는 전력질주를 위한 무대. 몰입과 끈기로 성장할 준비가 된 사람을 환영합니다. 스푼랩스 문화 블로그 스푼랩스 테크 블로그 스푼랩스 링크드인 스푼랩스 채용 사이트 🌱 [몰입하고, 성장할 수 있도록 이런 제도가 준비되어 있어요] [성장을 위한 제도] 끊임없이 도전하고 더 나은 결과를 만들어내는 구성원을 위해 월 10만원 한도의 자기계발비 지원 월 20만원 한도의 일본어, 영어, 한국어 외국어 학습비 지원 AWS re:Invent, Digital Marketing Summit, MAU Conference 등 업무 관련 국내외 교육 및 세미나 참석 지원 우리 같이 공부해요! 사내 스터디 모임 지원 사내 도서관 운영 및 신청 도서 구매 입사자와 추천자 모두 후한 보상을! 사내 직원 추천 제도 우리는 빠르게 배우고, 더 나은 방향으로 스스로 성장하는 사람들이 함께 모인 팀입니다. [일하는 방식] 몰입과 실행의 밀도를 높이기 위해 해외 법인에서 근무하는 오피스 익스체인지 프로그램 창의적 몰입을 위한 워케이션 프로그램 오전 8시~10시 30분 사이 자유롭게 출근하는 자율출근제 열심히 일하고 안전하게 퇴근해야죠. 야근 식비 및 택시비 지원 월요일은 4시간의 몰입을! 더욱 집중해서 몰입하는 주 4.5일제 우리는 더 깊이, 함께 몰입할 수 있는 환경을 선택합니다. [함께하는 팀을 위한 환경] 치열하게 일한 만큼, 서로를 챙기기 위해 함께해 주셔서 감사해요! 근속 기간별 리프레시 휴가 및 휴가비 지원 생일을 진심으로 축하합니다. 생일자를 위한 반반차 휴가 개개인의 일상에도 진심을 담아, 경조 휴가 및 경조비 지원 개인 근무 일정에 맞게 알아서 사용하는 휴가/반반차 제도 든든히 드세요. 아침 식사 제공 및 점심 식비 지원 무엇보다 건강이 우선이죠. 연 1회 종합건강검진 제공 강남역 역세권의 깔끔하고 세련된 사무실 에너지는 항상 충전되어야 하니까. 무제한 카페테리아 운영 힘들 땐 잠시 쉬어요. 고급 안마의자, 게임기, 다트, 탁구대 구비 치열하게 일한 만큼, 재충전도 중요하니까. 우리는 일하는 순간뿐 아니라, 그 사이사이도 함께 고민합니다. 채용에 대해 궁금한 점이 있다면? 아래 메일로 문의 주세요! 스푼랩스 채용: recruit@spoonlabs.com 주식회사 스푼랩스는 채용 ATS 그리팅의 개인정보 처리방침에 따라 개인정보를 수집 및 이용하고 있습니다. Show more Show less

Posted 1 week ago

Apply

3.0 - 5.0 years

12 - 15 Lacs

Chennai

Work from Office

Naukri logo

Role & responsibilities Design ML design and Ops stack considering the various trade-offs. Statistical Analysis and fundamentals MLOPS frameworks design and implementation Model Evaluation best practices -Train and retrain systems when necessary. Extend existing ML libraries and frameworks -Keep abreast of developments in the field. Act as a SME and tech lead / veteran for any data engineering question and manage data scientists and influence DS development across the company. Promote services, contribute to the identification of innovative initiatives within the Group, share information on new technologies in dedicated internal communities. Ensure compliance with policies related to Data Management and Data Protection Preferred candidate profile Strong experience (2+ years) with Building statistical models, applying machine learning techniques Experience (2+ years) on Big Data technologies such as Hadoop, Spark, Airflow/Databricks Proven experience (2+ years) in solving complex problems with multi-layered data sets, as well as optimizing existing machine learning libraries and frameworks. Proven experience (2+ years) on innovation implementation from exploration to production: these may include containerization (i.e. Docker/Kubernetes) , Big data (Hadoop, Spark) and MLOps platforms. Deep understanding of E2E software development in a team, and a track record of shipping software on time Ensure high-quality data and understand how data, which is generated out experimental design can produce actionable, trustworthy conclusions. Proficiency with SQL and NoSQL databases, data warehousing concepts, and cloud-based analytics database (e.g. Snowflake , Databricks or Redshift) administration

Posted 1 week ago

Apply

Exploring Hadoop Jobs in India

The demand for Hadoop professionals in India has been on the rise in recent years, with many companies leveraging big data technologies to drive business decisions. As a job seeker exploring opportunities in the Hadoop field, it is important to understand the job market, salary expectations, career progression, related skills, and common interview questions.

Top Hiring Locations in India

  1. Bangalore
  2. Mumbai
  3. Pune
  4. Hyderabad
  5. Chennai

These cities are known for their thriving IT industry and have a high demand for Hadoop professionals.

Average Salary Range

The average salary range for Hadoop professionals in India varies based on experience levels. Entry-level Hadoop developers can expect to earn between INR 4-6 lakhs per annum, while experienced professionals with specialized skills can earn upwards of INR 15 lakhs per annum.

Career Path

In the Hadoop field, a typical career path may include roles such as Junior Developer, Senior Developer, Tech Lead, and eventually progressing to roles like Data Architect or Big Data Engineer.

Related Skills

In addition to Hadoop expertise, professionals in this field are often expected to have knowledge of related technologies such as Apache Spark, HBase, Hive, and Pig. Strong programming skills in languages like Java, Python, or Scala are also beneficial.

Interview Questions

  • What is Hadoop and how does it work? (basic)
  • Explain the difference between HDFS and MapReduce. (medium)
  • How do you handle data skew in Hadoop? (medium)
  • What is YARN in Hadoop? (basic)
  • Describe the concept of NameNode and DataNode in HDFS. (medium)
  • What are the different types of join operations in Hive? (medium)
  • Explain the role of the ResourceManager in YARN. (medium)
  • What is the significance of the shuffle phase in MapReduce? (medium)
  • How does speculative execution work in Hadoop? (advanced)
  • What is the purpose of the Secondary NameNode in HDFS? (medium)
  • How do you optimize a MapReduce job in Hadoop? (medium)
  • Explain the concept of data locality in Hadoop. (basic)
  • What are the differences between Hadoop 1 and Hadoop 2? (medium)
  • How do you troubleshoot performance issues in a Hadoop cluster? (advanced)
  • Describe the advantages of using HBase over traditional RDBMS. (medium)
  • What is the role of the JobTracker in Hadoop? (medium)
  • How do you handle unstructured data in Hadoop? (medium)
  • Explain the concept of partitioning in Hive. (medium)
  • What is Apache ZooKeeper and how is it used in Hadoop? (advanced)
  • Describe the process of data serialization and deserialization in Hadoop. (medium)
  • How do you secure a Hadoop cluster? (advanced)
  • What is the CAP theorem and how does it relate to distributed systems like Hadoop? (advanced)
  • How do you monitor the health of a Hadoop cluster? (medium)
  • Explain the differences between Hadoop and traditional relational databases. (medium)
  • How do you handle data ingestion in Hadoop? (medium)

Closing Remark

As you navigate the Hadoop job market in India, remember to stay updated on the latest trends and technologies in the field. By honing your skills and preparing diligently for interviews, you can position yourself as a strong candidate for lucrative opportunities in the big data industry. Good luck on your job search!

cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

Featured Companies