876 Apache Spark Jobs

Responsible for design, development of scalable data pipelines, integrations to support continual increases in data volume, complexity. Work with analytics & business teams create pipelines, improve data models that feed BI and visualization tools. Required Candidate profile 8+ Yrs Exp in Data Pipeline Development,Data Modeling,Data Integration, Performance Optimization, Data Quality &Governance, Collaboration, Documentation and Best Practices, Continuous Improvement,

As a Data Scientist at InMobi Advertising, you will play a crucial role in impacting company metrics through the development of high-performance models in the fast-paced world of AI and machine learning. Your work will directly contribute to real-time business outcomes, leveraging InMobi's advanced advertising platform that reaches over 2 billion people across 150+ countries. **Role Overview:** - You will work alongside engineering colleagues to deploy models at scale and speed, contributing to the continuous experimentation, refinement, and impact assessment of your solutions. - Your primary responsibility will involve developing deep expertise in Ad Tech, AI/ML, and Data Science, focusing ...

As a Technical Manager (Java & Big Data) at our company based in Bangalore, you will be part of the dynamic R&D team. Your role will involve spearheading key technical initiatives, managing cross-functional teams, and driving innovative telecom solutions using modern Java and Big Data technologies. **Key Responsibilities:** - Possess 8+ years of software development experience. - Hold a minimum of 3 years in a people management role. - Lead and mentor a high-performing development team. - Drive the design, development, and deployment of Java-based applications. - Oversee Big Data projects and ensure smooth integration with cloud platforms. - Ensure code quality, best practices, and continuou...

You will be responsible for architecting and implementing scalable offline data pipelines for manufacturing systems, including AMHS, MES, SCADA, PLCs, vision systems, and sensor data. You will design and optimize ETL/ELT workflows using Python, Spark, SQL, and orchestration tools like Airflow to transform raw data into actionable insights. Leading database design and performance tuning across SQL and NoSQL systems will be part of your responsibilities, optimizing schema design, queries, and indexing strategies for manufacturing data. It will be essential for you to enforce robust data governance by implementing data quality checks, lineage tracking, access controls, security measures, and re...

As a Data Platform developer at Barclays, you'll play a crucial role in the evolution of the digital landscape, driving innovation and excellence. By harnessing cutting-edge technology, you will revolutionize digital offerings to ensure unparalleled customer experiences. You'll work collaboratively with fellow engineers, business analysts, and stakeholders to deliver technology stacks that meet business requirements and deliver quality solutions. Your role will involve solving complex technical problems that require strong analytical skills and detailed analysis. **Key Responsibilities:** - Hands-on coding experience, preferably in Python - Strong knowledge and hands-on experience of AWS dev...

As an Azure Data Engineer, your role involves architecting, configuring, and optimizing Databricks Pipelines for large-scale data processing within an Azure Data Lakehouse environment. Your key responsibilities include: - Architect, configure, and optimize Databricks Pipelines for large-scale data processing within an Azure Data Lakehouse environment - Set up and manage Azure infrastructure components including Databricks Workspaces, AKS/ACI, Storage Accounts, and Networking - Design and implement monitoring and observability frameworks using Azure Monitor, Log Analytics, Prometheus, and Grafana - Collaborate with platform and data engineering teams to enable microservices-based architecture...

Role Overview: As a skilled Cloud Specialist with expertise in Machine Learning, you will be responsible for designing, implementing, and managing cloud-based machine learning solutions to drive business value. Collaborating with data scientists and engineers, your primary focus will be on optimizing machine learning workflows, enhancing data pipelines, and ensuring the reliability and performance of machine learning applications. Additionally, you will provide insights and guidance on best practices for leveraging cloud services to maximize the potential of machine learning initiatives. Key Responsibilities: - Design and implement scalable architecture for machine learning applications in c...

As a Software Engineer specializing in Distributed Computing & Database Internals at Zettabolt, you will be working on engineering high-performance databases and distributed systems that drive large-scale applications. Your role will involve deep-diving into execution engines, query optimizations, and database internals, while understanding their interactions at scale. **Key Responsibilities:** - Work at the source code level of distributed computing frameworks to optimize query execution, indexing, storage engines, and concurrency mechanisms. - Improve execution efficiency, optimize computation overhead, and fine-tune scheduling and resource management in distributed environments. - Enhance...

Educational Requirements Bachelor of Engineering,BTech,BCA,MTech,MCA Service Line Data & Analytics Unit Responsibilities A day in the life of an Infoscion As part of the Infosys delivery team, your primary role would be to provide best fit architectural solutions for one or more projects. You would also provide technology consultation and assist in defining scope and sizing of work You would implement solutions, create technology differentiation and leverage partner technologies. Additionally, you would participate in competency development with the objective of ensuring the best-fit and high quality technical solutions. You would be a key contributor in creating thought leadership within th...

Job description Hiring for PySpark Specialist | Data Pipeline Mandatory Skills: PySpark, Python, SQL Responsibilities A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to actively aid the consulting team in different phases of the project including problem definition, effort estimation, diagnosis, solution generation and design and deployment You will explore the alternatives to the recommended solutions based on research that includes literature surveys, information available in public domains, vendor evaluation information, etc. and build POCs You will create requirement specifications from the business needs, define the to-be-processes an...

Job description Hiring for PySpark Specialist | Data Pipeline Mandatory Skills: PySpark, Python, SQL Responsibilities A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to actively aid the consulting team in different phases of the project including problem definition, effort estimation, diagnosis, solution generation and design and deployment You will explore the alternatives to the recommended solutions based on research that includes literature surveys, information available in public domains, vendor evaluation information, etc. and build POCs You will create requirement specifications from the business needs, define the to-be-processes an...

Job description Hiring for PySpark Specialist | Data Pipeline Mandatory Skills: PySpark, Python, SQL Responsibilities A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to actively aid the consulting team in different phases of the project including problem definition, effort estimation, diagnosis, solution generation and design and deployment You will explore the alternatives to the recommended solutions based on research that includes literature surveys, information available in public domains, vendor evaluation information, etc. and build POCs You will create requirement specifications from the business needs, define the to-be-processes an...

As a Data Engineer at Crowe, you will play a crucial role in providing integration infrastructure for analytical support and solution development for the broader Enterprise. Your expertise in API integration, pipelines or notebooks, programming languages (Python, Spark, T-SQL), dimensional modeling, and advanced data engineering techniques will be essential in creating and delivering robust solutions and data products. You will be responsible for designing, developing, and maintaining the Enterprise Analytics Platform to facilitate data-driven decision-making across the organization. Your success in this role will rely on your strong interest and passion in data analytics, ETL/ELT best pract...

As a Principal Engineer/Architect at our company, you will be a key player in building and scaling our Data & AI practice. Your role will involve leading complex data migration and transformation projects, defining best practices and reusable frameworks, and influencing decision-making at Fortune 500 companies and innovative startups. You will work with modern cloud technologies such as AWS, Azure, GCP, Databricks, and advanced Data & AI tools. This is a senior leadership position that requires technical expertise, strategic vision, and client-facing leadership. **Key Responsibilities:** - Design and architect scalable, high-performance data solutions on cloud platforms (AWS, Azure, GCP). - ...

Role Overview: KPMG in India is a professional services firm affiliated with KPMG International Limited, offering services to national and international clients across various sectors. Their offices are located in multiple cities across India, and they aim to provide industry-focused, technology-enabled services that align with global and local industries and the Indian business environment. Key Responsibilities: - 2 years of experience in software development or a related field - 2 years of experience in database technologies - 2 years of experience working on project(s) involving the implementation of solutions applying development life cycles (SDLC) - 2 years of experience in Java, JEE ap...

Role Overview: You will be joining Kainskep Solutions as a Data Scientist where you will be responsible for utilizing programming languages such as Python, R, and SQL for data manipulation, statistical analysis, and machine learning tasks. Your role will involve developing supervised and unsupervised machine learning models, conducting data manipulation and visualization, and performing exploratory data analysis. Additionally, you will be expected to familiarize yourself with big data technologies and manage relational and NoSQL databases for data storage and retrieval. Being part of a diverse and inclusive team, you will collaborate with colleagues using version control systems for collabor...

Role Overview: As a Technical Specialist - Data Engineer, you will be responsible for designing, developing, and delivering significant components of engineering solutions to achieve business goals. Your key responsibilities will include actively participating in the design and development of new features for application enhancement, ensuring solutions are maintainable and integrated successfully, assisting junior team members, and controlling their work when necessary. Key Responsibilities: - Develop and deploy source code for all software components in alignment with detailed software requirements specification. - Provide development activities for projects and technical infrastructure com...

As a Data Engineer, you will be responsible for managing and optimizing data pipelines for efficient data processing. Your role will involve working with a variety of technologies and platforms to ensure smooth data workflows. Key Responsibilities: - Utilize your 5+ years of experience in data engineering, including at least 2+ years of hands-on experience with Databricks. - Demonstrate strong expertise in Apache Spark using PySpark, Scala, and SQL for efficient distributed data processing. - Implement and maintain Delta Lake and Lakehouse architecture for effective data modeling. - Work with cloud platforms such as Azure Data Lake, AWS S3, or GCP BigQuery/Storage to manage data infrastructu...

Role Overview: As a Technology Practice Leader at our company, you will play a crucial role in enabling delivery scalability and leading the successful delivery of complex projects in the Application Performance Engineering, Application Performance Management, and Quality Assurance domain. You will be at the forefront of providing thought leadership and innovation to build a High-end Technology company that aims to be a Global leader in the P-A-S-S (performance, availability, scalability, and security) Assurance space. Your role will involve leading from the front, supporting the team, managing stakeholders, and ensuring successful account entry and downstream business with new customers. Ke...

Role Overview: You will be responsible for designing and implementing scalable and secure cloud architecture on cloud platforms like AWS, Azure, and GCP. Your expertise in building enterprise-grade data platforms using technologies such as Databricks, Snowflake, Spark, and Data Lake/Warehouse solutions will be crucial in supporting advanced analytics and AI/ML integrations. Additionally, you will conduct research and development for custom architecture PoCs, including AI/ML model deployment, security enhancements, and scalable data workflows to meet evolving business needs. Key Responsibilities: - Design and deploy data pipelines with technologies like AWS (S3, Lambda, Glue, Data Pipeline), ...

Responsible for design, development of scalable data pipelines, integrations to support continual increases in data volume, complexity. Work with analytics & business teams create pipelines, improve data models that feed BI and visualization tools. Required Candidate profile 8+ Yrs Exp in Data Pipeline Development,Data Modeling,Data Integration, Performance Optimization, Data Quality &Governance, Collaboration, Documentation and Best Practices, Continuous Improvement,

8+ years of relevant IT experience in the BI/DW domain with hands-on experience on the Azure modern data platform . Strong knowledge and experience with the Python programming language Experience using version control systems like Git, bitbucket . Experience in data analysis and transformation using PySpark on Azure Databricks or Apache Spark . Good knowledge of Distributed Processing using Databricks or Apache Spark. Experience in creating data structures optimized for storage and various query patterns. Experience of integrating Unity Data Catalog with the Databricks and registering data assets etc. Understanding of CI/CD pipelines. Knowledge of Agile/Scrum methodologies.

Skill: Databricks Production Support - Senior Level Experience: 8-12 Years Location: Hyderabad & Bangalore Notice Period: Immediate - 15 Days Shift Timings: 1PM - 10PM Detailed job description - Skill Set: We are seeking a highly skilled Databricks Platform Operations to join our team, responsible for daily monitoring and resolution of data load issues, platform optimization, capacity planning, and governance management. This role is pivotal in ensuring the stability, scalability, and security of our Databricks environment while acting as a technical architect for platform best practices. The ideal candidate will bring a strong operational background, potentially with earlier experience as a...

Skill: Databricks Production Support - Senior Level Experience: 8-12 Years Location: Hyderabad & Bangalore Notice Period: Immediate - 15 Days Shift Timings: 1PM - 10PM Detailed job description - Skill Set: We are seeking a highly skilled Databricks Platform Operations to join our team, responsible for daily monitoring and resolution of data load issues, platform optimization, capacity planning, and governance management. This role is pivotal in ensuring the stability, scalability, and security of our Databricks environment while acting as a technical architect for platform best practices. The ideal candidate will bring a strong operational background, potentially with earlier experience as a...

Skill:-Apache Iceberg Exp- 3 -12 Yrs Job Description: Must Have Skills: • Good experience in Apache Iceberg, Apache Spark, Trino • Proficiency in SQL and data modeling Nice To Have Skills: Technical/Functional Skills Key Qualifications • Experience with open Data Lakehouse using Apache Iceberg • Experience with Data Lakehouse architecture with Apache Iceberg and Trino • Experience with building/managing Lakehouse for large scale data platforms storage, computing aspects, data ingestion/governance and data management and performance. • Experience with distributed computing technology(s) like Spark. • Experience with data platforms, data lakes and ingestion systems that work at scale. • Skille...