1537 Apache Spark Jobs

Role Description: We are looking for highly motivated expert Senior Data Engineer who can own the design & development of complex data pipelines, solutions and frameworks. The ideal candidate will be responsible to design, develop, and optimize data pipelines, data integration frameworks, and metadata-driven architectures that enable seamless data access and analytics. This role prefers deep expertise in big data processing, distributed computing, data modeling, and governance frameworks to support self-service analytics, AI-driven insights, and enterprise-wide data management. Roles & Responsibilities: Design, develop, and maintain scalable ETL/ELT pipelines to support structured, semi-stru...

What you will do In this vital role you will be responsible for designing, building, maintaining, analyzing, and interpreting data to provide actionable insights that drive business decisions. This role involves working with large datasets, developing reports, supporting and performing data governance initiatives and, visualizing data to ensure data is accessible, reliable, and efficiently managed. The ideal candidate has deep technical skills, experience with database technologies, and a deep understanding of data architecture and ETL processes. Roles & Responsibilities: Design, develop, and maintain data solutions for data generation, collection, and processing Be a crucial team member tha...

Role Description: This role acts as t echnical architect and hands-on lead for Data Engineering practices across the Smart Supply Chain initiative within Amgen. Additionally, responsible for designing, building, maintaining , analyzing, and interpreting data to provide actionable insights that drive business decisions. This role involves working with large datasets, developing reports, supporting and executing data governance initiatives and , visualizing data to ensure data is accessible, reliable, and efficiently managed. The ideal candidate has strong technical skills, experience with big data technologies, and a deep understanding of data architecture and ETL processes and will architect...

Overview Key Responsibilities Architectural Vision & Strategy: Design and own the end-to-end architecture for our data platform, encompassing data ingestion, processing, storage, and consumption layers System Design: Create detailed architectural diagrams, design documents, and technical specifications for new platform capabilities Define the standards for how services interact Technology Evaluation & Governance: Research, evaluate, and prototype new technologies (e. g,table formats, query engines, streaming platforms) Establish and govern technical standards and bestpractices across all data engineering teams Cross-Functional Leadership: Collaborate with engineering leaders, product manager...

What you will do Role Description: We are seeking a Data Solutions Architect with deep R&D expertise in Biotech/Pharma to design, implement, and optimize scalable and high-performance data solutions that support enterprise analytics, AI-driven insights, and digital transformation initiatives. This role will focus on data strategy, architecture, governance, security, and operational efficiency, ensuring seamless data integration across modern cloud platforms. The ideal candidate will work closely with R&D and engineering teams, business stakeholders, and leadership to establish a future-ready data ecosystem, balancing performance, cost-efficiency, security, and usability. This position requir...

About The Role Project Role : Custom Software Engineer Project Role Description : Design, build and configure applications to meet business process and application requirements. Must have skills : PySpark Good to have skills : NA Minimum 5 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As an Application Developer, you will engage in the design, construction, and configuration of applications tailored to fulfill specific business processes and application requirements. Your typical day will involve collaborating with team members to understand project needs, developing innovative solutions, and ensuring that applications are optimized for p...

About The Role Project Role : Custom Software Engineer Project Role Description : Develop custom software solutions to design, code, and enhance components across systems or applications. Use modern frameworks and agile practices to deliver scalable, high-performing solutions tailored to specific business needs. Must have skills : Apache Spark Good to have skills : NA Minimum 7.5 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a Custom Software Engineer, you will develop custom software solutions to design, code, and enhance components across systems or applications. Your typical day will involve collaborating with cross-functional teams...

About The Role Project Role : Custom Software Engineer Project Role Description : Develop custom software solutions to design, code, and enhance components across systems or applications. Use modern frameworks and agile practices to deliver scalable, high-performing solutions tailored to specific business needs. Must have skills : Apache Spark Good to have skills : NA Minimum 7.5 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a Custom Software Engineer, you will develop custom software solutions to design, code, and enhance components across systems or applications. Your typical day will involve collaborating with cross-functional teams...

About The Role Project Role : Analytics Practitioner Project Role Description : Develop business insights using predictive statistical modeling activities. Use analytical tools and techniques to generate and improve decision-making by modeling client, market, and key performance data. Must have skills : Scala Good to have skills : NA Minimum 5 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As an Analytics Practitioner, you will engage in developing business insights through predictive statistical modeling activities. Your typical day will involve utilizing various analytical tools and techniques to enhance decision-making processes by mode...

We are looking for a skilled Python developer with expertise in Airflow to join our team at IDESLABS PRIVATE LIMITED. The ideal candidate should have 6-10 years of experience. Roles and Responsibility Design, develop, and deploy scalable data pipelines using Apache Airflow. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop complex data transformations and scripts using Python. Troubleshoot and optimize existing data pipelines for improved efficiency. Ensure data quality and integrity by implementing robust testing and validation procedures. Participate in code reviews and contribute to the improvement of the overall codebase. Job Requirements St...

We are looking for a skilled Python developer with expertise in Airflow to join our team at IDESLABS PRIVATE LIMITED. The ideal candidate should have 6-10 years of experience. Roles and Responsibility Design, develop, and deploy scalable data pipelines using Apache Airflow. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop complex data transformations and scripts using Python. Troubleshoot and optimize existing data pipelines for improved efficiency. Ensure data quality and integrity by implementing robust testing and validation procedures. Participate in code reviews and contribute to the improvement of the overall codebase. Job Requirements St...

We are looking for a skilled Data Engineer with expertise in Apache Spark and Hive to join our team at IDESLABS PRIVATE LIMITED. The ideal candidate will have 4-6 years of experience. Roles and Responsibility Design, develop, and implement data pipelines using Apache Spark and Hive. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data processing systems using Python, Scala, or Java. Work with Kafka, Solace, and object stores such as S3 and MinIO. Ensure data quality and integrity by implementing robust testing and validation frameworks. Participate in code reviews and contribute to improving overall code quality. Job R...

Data Engineer - PySpark ( Bulk hiring) Strong expertise in Apache Spark (batch + streaming) and Hive Proficiency in Python, Scala, or Java Knowledge of orchestration tools (Airflow / Control-M) and SQL transformation frameworks (DBT preferred) Experience working with Kafka, Solace, and object stores (S3, MinIO) Exposure to Docker/Kubernetes for deployment Hands on experience of data Lakehouse formats (Iceberg, Delta Lake, Hudi)

Amgen is seeking a Associate Software Engineer (Data Tester) to contribute to designing, building, maintaining, analyzing, and interpreting data to provide actionable insights that drive business decisions. This role involves working with large datasets, developing data pipelines, supporting and executing data governance initiatives and visualizing data to ensure data is accessible, reliable, and efficiently managed. The ideal candidate has strong technical skills and experience with Databricks, Airflow and AWS. Roles & Responsibilities: Design, develop, and maintain data solutions for data generation, collection, and processing using Databricks and Airflow Design, build and maintain APIs an...

Role Description: We are seeking a Data Solutions Architect to design, implement, and optimize scalable and high-performance data solutions that support enterprise analytics, AI-driven insights, and digital transformation initiatives. This role will focus on data strategy, architecture, governance, security, and operational efficiency, ensuring seamless data integration across modern cloud platforms. The ideal candidate will work closely with engineering teams, business stakeholders, and leadership to establish a future-ready data ecosystem, balancing performance, cost-efficiency, security, and usability. This position requires expertise in modern cloud-based data architectures, data enginee...

Project Role : Application Developer Project Role Description : Design, build and configure applications to meet business process and application requirements. Must have skills : PySpark Good to have skills : NA Minimum 3 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As an Application Developer, you will design, build, and configure applications to meet business process and application requirements. A typical day involves collaborating with team members to understand project needs, developing application features, and ensuring that the applications function smoothly and efficiently. You will also engage in problem-solving discussions and ...

Project Role : BI Engineer Project Role Description : Develop, migrate, deploy, and maintain data-driven insights and knowledge engine interfaces that drive adoption and decision making. Integrate security and data privacy protection. Must have skills : Apache Spark Good to have skills : NA Minimum 7.5 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a BI Engineer, you will develop, migrate, deploy, and maintain data-driven insights and knowledge engine interfaces that drive adoption and decision making. Your typical day will involve collaborating with various teams to ensure the integration of security and data privacy protection while d...

Project Role : BI Engineer Project Role Description : Develop, migrate, deploy, and maintain data-driven insights and knowledge engine interfaces that drive adoption and decision making. Integrate security and data privacy protection. Must have skills : Apache Spark Good to have skills : NA Minimum 7.5 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a BI Engineer, you will develop, migrate, deploy, and maintain data-driven insights and knowledge engine interfaces that drive adoption and decision making. Your typical day will involve collaborating with various teams to ensure the integration of security and data privacy protection while d...

Project Role : BI Engineer Project Role Description : Develop, migrate, deploy, and maintain data-driven insights and knowledge engine interfaces that drive adoption and decision making. Integrate security and data privacy protection. Must have skills : Apache Spark Good to have skills : NA Minimum 5 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a BI Engineer, you will develop, migrate, deploy, and maintain data-driven insights and knowledge engine interfaces that drive adoption and decision making. Your typical day will involve collaborating with various teams to ensure the integration of security and data privacy protection while del...

Project Role : BI Engineer Project Role Description : Develop, migrate, deploy, and maintain data-driven insights and knowledge engine interfaces that drive adoption and decision making. Integrate security and data privacy protection. Must have skills : Apache Spark Good to have skills : NA Minimum 7.5 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a BI Engineer, you will develop, migrate, deploy, and maintain data-driven insights and knowledge engine interfaces that drive adoption and decision making. Your typical day will involve collaborating with various teams to ensure the integration of security and data privacy protection while d...

Project Role : BI Engineer Project Role Description : Develop, migrate, deploy, and maintain data-driven insights and knowledge engine interfaces that drive adoption and decision making. Integrate security and data privacy protection. Must have skills : Apache Spark Good to have skills : NA Minimum 7.5 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a BI Engineer, you will develop, migrate, deploy, and maintain data-driven insights and knowledge engine interfaces that drive adoption and decision making. Your typical day will involve collaborating with various teams to ensure the integration of security and data privacy protection while d...

Project Role : Custom Software Engineer Project Role Description : Develop custom software solutions to design, code, and enhance components across systems or applications. Use modern frameworks and agile practices to deliver scalable, high-performing solutions tailored to specific business needs. Must have skills : Apache Spark Good to have skills : Health Insurance Operations Minimum 3 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As an Application Developer, you will engage in the design, construction, and configuration of applications tailored to fulfill specific business processes and application requirements. Your typical day will i...

Project Role : BI Engineer Project Role Description : Develop, migrate, deploy, and maintain data-driven insights and knowledge engine interfaces that drive adoption and decision making. Integrate security and data privacy protection. Must have skills : Apache Spark Good to have skills : NA Minimum 3 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a BI Engineer, you will develop, migrate, deploy, and maintain data-driven insights and knowledge engine interfaces that drive adoption and decision making. Your typical day will involve collaborating with various teams to ensure the integration of security and data privacy protection while del...

Project Role : Custom Software Engineer Project Role Description : Design, build and configure applications to meet business process and application requirements. Must have skills : PySpark Good to have skills : Apache Spark Minimum 3 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As an Application Developer, you will engage in the design, construction, and configuration of applications tailored to fulfill specific business processes and application requirements. Your typical day will involve collaborating with team members to understand project needs, developing innovative solutions, and ensuring that applications are optimized for perfor...

What you will do In this vital role you will be responsible for designing, building, maintaining, analyzing, and interpreting data to provide actionable insights that drive business decisions. This role involves working with large datasets, developing reports, supporting and executing data governance initiatives and, visualizing data to ensure data is accessible, reliable, and efficiently managed. The ideal candidate has strong technical skills, experience with big data technologies, and a deep understanding of data architecture and ETL processes Roles & Responsibilities: Design, develop, and maintain data solutions for data generation, collection, and processing Be a key team member that as...