530 Apache Spark Jobs

Key Skills & Responsibilities Hands-on experience with AWS services: S3, Lambda, Glue, API Gateway, and SQS. Strong data engineering expertise on AWS, with proficiency in Python, PySpark, and SQL. Experience in batch job scheduling and managing data dependencies across pipelines. Familiarity with data processing tools such as Apache Spark and Airflow. Ability to automate repetitive tasks and build reusable frameworks for improved efficiency. Provide RunOps DevOps support, and manage the ongoing operation and monitoring of data services. Ensure high performance, scalability, and reliability of data workflows in cloud environments. Skills: aws,s3,glue,apache spark,lambda,airflow,sql,s3, lambda...

Key Skills & Responsibilities Hands-on experience with AWS services: S3, Lambda, Glue, API Gateway, and SQS. Strong data engineering expertise on AWS, with proficiency in Python, PySpark, and SQL. Experience in batch job scheduling and managing data dependencies across pipelines. Familiarity with data processing tools such as Apache Spark and Airflow. Ability to automate repetitive tasks and build reusable frameworks for improved efficiency. Provide RunOps DevOps support, and manage the ongoing operation and monitoring of data services. Ensure high performance, scalability, and reliability of data workflows in cloud environments. Skills: aws,s3,glue,apache spark,lambda,airflow,sql,s3, lambda...

Solugenix is a leader in IT services, delivering cutting-edge technology solutions, exceptional talent, and managed services to global enterprises. With extensive expertise in highly regulated and complex industries, we are a trusted partner for integrating advanced technologies with streamlined processes. Our solutions drive growth, foster innovation, and ensure complianceproviding clients with reliability and a strong competitive edge. Recognized as a 2024 Top Workplace, Solugenix is proud of its inclusive culture and unwavering commitment to excellence. Our recent expansion, with new offices in the Dominican Republic, Jakarta, and the Philippines, underscores our growing global presence a...

About the Role: We are looking for an experienced Java Developer to design, build, and maintain scalable data solutions. You will work on modern Java (21+) technologies, contribute to distributed cloud-native systems, and collaborate with cross-functional teams. Key Responsibilities: Develop and maintain scalable, efficient Java-based applications. Collaborate with product and data teams to translate requirements into solutions. Implement best practices: code reviews, unit testing, CI/CD. Troubleshoot performance issues and ensure system reliability. Participate in Agile ceremonies (sprints, stand-ups, retros). Mentor junior developers and support knowledge sharing. Required Skills: 6+ years...

Bachelors or Master’s degree in Computer Science, Engineering, or a related field with Data Engineering Experience. 5+ years of professional experience in software development, with at least 2 years in a lead or senior engineering role. Strong proficiency in Python programming and experience building data processing applications. Hands-on expertise with Apache Spark and Hadoop for big data processing. Solid experience with workflow orchestration tools like Apache Airflow. Experience designing and managing data ingestion and integration pipelines with Apache NiFi. Understanding on Data Quality automation, CI/CD, Jenkins, Oracle, Power BI, Splunk Deep understanding of data quality concepts, da...

You're ready to gain the skills and experience needed to grow within your role and advance your career and there is a perfect software engineering opportunity for you at JPMorgan Chase within the Commercial & Investment Bank Team. As a Software Engineer II - Databricks Engineer + Python programming, you will serve as a seasoned member of an agile team to deliver trusted market-leading Data Engineering Solutions and Data products in a secure, stable, and scalable way. Your responsibilities will include implementing Databricks integration and interoperability with other cloud providers, collaborating with data scientists, analysts, and business stakeholders, developing and maintaining data pro...

Role Overview: As a Software Engineer (A2) at [Company Name], you will be responsible for designing and developing AI-driven data ingestion frameworks and real-time processing solutions to enhance data analysis and machine learning capabilities. Your role will involve deploying, maintaining, and supporting application codes and machine learning models in production environments, as well as creating and enhancing AI solutions that facilitate seamless integration and flow of data across the data ecosystem. Additionally, you will be conducting business analysis, developing ETL processes, scripts, and machine learning pipelines, and ensuring compliance with data governance policies. Key Responsi...

Design, develop, and deploy scalable and reliable event processing pipelines using technologies such as Apache Kafka, Confluent Platform, Apache Flink, and Apache Spark./databricks Develop and maintain infrastructure-as-code (IaC) solutions for event processing infrastructure using tools like Terraform or CloudFormation within the AWS ecosystem. Write clean, efficient, and well-documented code primarily in Python and PySpark. Monitor and optimize the performance of event processing pipelines. Development and maintenance of CI/CD pipelines. Utilise Databricks for data processing and analysis tasks. S3 for data storage and retrieval. Mandatory Skillset - Apache Kafka, Confluent Platform, Apach...

AWS Infrastructure Engineer with strong expertise in designing, implementing, and optimizing AWS cloud infrastructure along with hands-on experience in deploying ML/AI workloads on AWS

**Role Overview:** Join Amgen's mission of serving patients by becoming a vital part of the Data Platform Engineering team. You will play a crucial role in using Cloud and Big Data technologies to design, develop, and maintain solutions supporting various functions like Manufacturing, Commercial, Research, and Development. **Key Responsibilities:** - Collaborate with Lead Architect, Business SMEs, and Data Scientists to design data solutions. - Act as a Lead Engineer for technical implementation, including planning, architecture, design, development, testing, and deployment following agile methodologies. - Design and develop API services for managing Databricks resources, services & features...

As a Software Engineer (A2) at our company, you will be responsible for designing and developing AI-driven data ingestion frameworks and real-time processing solutions to enhance data analysis and machine learning capabilities across the full technology stack. Your key responsibilities will include: - Deploying, maintaining, and supporting application codes and machine learning models in production environments, ensuring seamless integration with front-end and back-end systems. - Creating and enhancing AI solutions that facilitate the seamless integration and flow of data across the data ecosystem, enabling advanced analytics and insights for end users. - Conducting business analysis to gath...

Cloud native application with AWS Lambda ECS Fargate DynamoDB SNS SQS Automate infrastructure using AWS CDK CloudFormation VPC IAM Subnets Security Groups from network to application layer ETL pipelines SQL Apache Spark AWS Redshift Athena S3 Iceberg

As the Vice President of Engineering at Teradata, you will lead the India-based software development organization for the AI Platform Group. Your primary responsibility will be to execute the product roadmap for key technologies such as Vector Store, Agent platform, Apps, user experience, and AI/ML-driven use-cases at scale. Success in this role will involve building a world-class engineering culture, attracting and retaining top technical talent, accelerating product delivery, and driving innovation to bring measurable value to customers. Key Responsibilities: - Lead a team of 150+ engineers to help customers achieve outcomes with Data and AI - Partner closely with Product Management, Produ...

Role Overview: As an AWS Senior Developer at PwC - AC, you will collaborate with the Offshore Manager and Onsite Business Analyst to comprehend the requirements and take charge of implementing end-to-end Cloud data engineering solutions such as Enterprise Data Lake and Data hub in AWS. Your role will involve showcasing strong proficiency in AWS cloud technology, exceptional planning and organizational skills, and the ability to work as a cloud developer/lead in an agile team to deliver automated cloud solutions. Key Responsibilities: - Proficient in Azure Data Services, Databricks, and Snowflake - Deep understanding of traditional and modern data architecture and processing concepts - Profic...

Role Overview: As a backend and data engineer at Agoda, you will be responsible for developing robust backend systems and architecting scalable data pipelines and storage solutions. Your expertise in backend and data engineering will be crucial in optimizing financial technology solutions and driving informed business decisions through reliable data insights. Key Responsibilities: - Own the full life cycle of products, from business requirements to coding standards, agile development, testing, and monitoring - Design, develop, and maintain platforms and data pipelines across fintech - Build stable, scalable, and highly performant systems to meet the dynamic demands of the financial landscape...

Role Overview: Our engineering team at Adobe develops the Adobe Experience Platform, focusing on innovative data management and analytics. Developing a reliable and resilient system at a large scale is crucial for us. We utilize Big Data and open-source technologies for Adobe's services, supporting large enterprise products across geographies. The data needs to be easily accessible at very low latency to cater to various scenarios and use cases. We are looking for candidates with deep expertise in building low latency services at high scales to lead us in accomplishing our vision. Key Responsibilities: - Lead the technical design and implementation strategy for major systems and components o...

Role Overview: YASH Technologies is seeking AWS Professionals with expertise in AWS services such as Glue, Pyspark, SQL, Databricks, Python, and more. As an AWS Data Engineer, you will be responsible for designing, developing, testing, and supporting data pipelines and applications. This role requires a degree in computer science, engineering, or related fields along with strong experience in data integration and pipeline development. Key Responsibilities: - Design, develop, test, and support data pipelines and applications using AWS services like Glue, Pyspark, SQL, Databricks, Python, etc. - Work with a mix of Apache Spark, Glue, Kafka, Kinesis, and Lambda in S3 Redshift, RDS, MongoDB/Dyna...

Role Overview: At WRD, we are redefining the world of IT consulting by merging human expertise with cutting-edge AI technology. Join us to work on the forefront of AI-powered innovation, where your skills drive real impact, creativity meets advanced technology, and each project sets a new standard in IT excellence. Key Responsibilities: - Develop & Maintain data pipelines to support business and analytical needs. - Work with structured and unstructured data to ensure high data quality and availability. - Build scalable and efficient data workflows using SQL, Python, and Java/Scala. - Manage and process large datasets on AWS EMR and other cloud platforms. - Develop and maintain batch and stre...

You are being sought after to take on the role of a Senior Data Engineer, where your primary responsibility will be leading the development of a scalable data ingestion framework. In this capacity, you will play a crucial role in ensuring high data quality and validation, as well as designing and implementing robust APIs for seamless data integration. Your expertise in building and managing big data pipelines using modern AWS-based technologies will be put to good use, making sure that quality and efficiency are at the forefront of data processing systems. Key Responsibilities: - **Data Ingestion Framework**: - Design & Development: You will architect, develop, and maintain an end-to-end dat...

We are looking for a skilled Data Engineer with experience in Python, SQL, Databricks, Apache Spark , and Microsoft Azure . You will be responsible for building scalable data pipelines, optimizing data workflows, and supporting analytics initiatives on the Azure cloud platform. Key Skills: Strong Python and SQL programming Databricks & Apache Spark for big data processing Azure Data Lake, Data Factory, and other Azure services Data pipeline development and optimization Experience with ETL/ELT processes and large-scale data systems

Position Overview We are seeking a highly skilled and motivated Big Data Engineer to join our dynamic team in either Mumbai City or Bangalore . As a Big Data Engineer, you will play a crucial role in designing, developing, and maintaining scalable data processing systems that enable our organization to harness the power of big data. With an annual salary of 26,00,000 , this full-time position offers an exciting opportunity to work with cutting-edge technologies in a collaborative environment. Key Responsibilities Design and implement robust big data solutions using technologies such as Hadoop, Spark, and Apache Spark. Develop and maintain data pipelines to ensure efficient data processing an...

Role & responsibilities Data Engineer- Z2 category Experience:- 5Yrs • Good Communication & presentation skills. • Strong Knowledge of Azure Components Azure Data Lake, Azure Data Factory, Azure SQL, Azure Databricks • Must have strong knowledge & hands on in Apache Spark and Scala / Python programming, working with Delta Tables, Experience in Databricks is must • Strong SQL Skills: Data modelling, Developing SQL Store Procedures, Functions, Dynamic SQL queries, Joins • Strong knowledge of Data Warehousing Concepts. • Hands on experience in ingesting data from various data sources and data types & file types • Should have good knowledge on development lifecycle , best practice, and coding st...

As a Senior Data Engineer at Amgen, you will play a crucial role in driving the development and implementation of the company's data strategy. You will be responsible for designing, building, and optimizing data pipelines and platforms while also mentoring junior engineers. Your technical expertise and data-driven problem-solving skills will be essential in ensuring data quality, integrity, and reliability. **Key Responsibilities:** - Contribute to the design, development, and implementation of automations to monitor data and cloud platforms for reliability, cost, and maintenance. - Take ownership of platform management, managing scope, timelines, and risks. - Ensure data quality and integri...

As a Snowflake Engineer - Digital Solution Consultant Sr. Analyst at NTT DATA in Hyderabad, Telangana (IN-TG), India, you will be responsible for the following: - Experience with other cloud data warehousing solutions. - Knowledge of big data technologies such as Apache Spark and Hadoop. - Proficiency in working with CI/CD pipelines and DevOps practices. - Familiarity with data visualization tools. About NTT DATA: NTT DATA is a $30 billion global innovator of business and technology services. Serving 75% of the Fortune Global 100, we are dedicated to assisting clients in innovating, optimizing, and transforming for long-term success. With a diverse team of experts in over 50 countries and a ...

At Rearc, we are dedicated to empowering engineers like you to create exceptional products and experiences by providing you with the best tools possible. We value individuals who think freely, challenge the norm, and embrace alternative problem-solving approaches. If you are driven by the desire to make a difference and solve complex problems, you'll feel right at home with us. As a Data Engineer at Rearc, you will be an integral part of our data engineering team, contributing to the optimization of data workflows for efficiency, scalability, and reliability. Your role will involve designing and implementing robust data solutions in collaboration with cross-functional teams to meet business ...