908 Apache Spark Jobs - Page 26

As an ideal candidate for this role, you should possess in-depth knowledge of Python and have good experience in creating APIs using FastAPI. You should also have exposure to data libraries like Pandas, DataFrame, NumPy, as well as knowledge in Apache open-source components and Apache Spark. Familiarity with Lakehouse architecture and Open table formats is also desirable. Additionally, you should be well-versed in automated unit testing, preferably using PyTest, and have exposure to distributed computing. Experience working in a Linux environment is a must, while working knowledge in Kubernetes would be considered an added advantage. Basic exposure to ML and MLOps would also be advantageous ...

As a Custom Software Engineer, you will be responsible for developing custom software solutions to design, code, and enhance components across systems or applications. Your role will involve using modern frameworks and agile practices to deliver scalable, high-performing solutions tailored to specific business needs. On a typical day, you will collaborate with cross-functional teams to understand business requirements and work towards aligning the software solutions with project goals. You are expected to be subject matter expert (SME) within the team, make team decisions, and engage in problem-solving activities to contribute to the success of the organization. Additionally, you will be men...

The Content and Data Analytics team is an integral part of Global Operations at Elsevier, within the DataOps division. The team primarily provides data analysis services using Databricks, catering to product owners and data scientists of Elsevier's Research Data Platform. Your work in this team will directly contribute to the development of cutting-edge data analytics products for the scientific research sector, including renowned products like Scopus and SciVal. As a Data Analyst II, you are expected to possess a foundational understanding of best practices and project execution, with supervision from senior team members. Your responsibilities will include generating basic insights and reco...

As a Data Engineer at our IT Services Organization, you will be responsible for developing and maintaining scalable data processing systems using Apache Spark and Python. Your role will involve designing and implementing Big Data solutions that integrate data from various sources, including RDBMS, NoSQL databases, and cloud services. Additionally, you will lead a team of data engineers to ensure efficient project execution and adherence to best practices. Your key responsibilities will include optimizing Spark jobs for performance and scalability, collaborating with cross-functional teams to gather requirements, and delivering data solutions that meet business needs. You will also be involve...

As a Senior Engineer at Impetus Technologies, you will play a crucial role in designing, developing, and deploying scalable data processing applications using Java and Big Data technologies. Your responsibilities will include collaborating with cross-functional teams, mentoring junior engineers, and contributing to architectural decisions to enhance system performance and scalability. Your key responsibilities will revolve around designing and maintaining high-performance applications, implementing data ingestion and processing workflows using frameworks like Hadoop and Spark, and optimizing existing applications for improved performance and reliability. You will also be actively involved in...

You should have familiarity with modern storage formats like Parquet and ORC. Your responsibilities will include designing and developing conceptual, logical, and physical data models to support enterprise data initiatives. You will build, maintain, and optimize data models within Databricks Unity Catalog, developing efficient data structures using Delta Lake to optimize performance, scalability, and reusability. Collaboration with data engineers, architects, analysts, and stakeholders is essential to ensure data model alignment with ingestion pipelines and business goals. You will translate business and reporting requirements into a robust data architecture using best practices in data ware...

You will be joining our team as a Senior Data Scientist with expertise in Artificial Intelligence (AI) and Machine Learning (ML). The ideal candidate should possess a minimum of 5-7 years of experience in data science, focusing on AI/ML applications. You are expected to have a strong background in various ML algorithms, programming languages such as Python, R, or Scala, and data processing frameworks like Apache Spark. Proficiency in data visualization tools and experience in model deployment using Docker, Kubernetes, and cloud services will be essential for this role. Your responsibilities will include end-to-end AI/ML project delivery, from data processing to model deployment. You should h...

Genpact (NYSE: G) is a global professional services and solutions firm committed to delivering outcomes that shape the future. With over 125,000 employees spread across more than 30 countries, we are fueled by our innate curiosity, entrepreneurial agility, and the aspiration to create lasting value for our clients. Driven by our purpose - the relentless pursuit of a world that works better for people - we cater to and transform leading enterprises, including the Fortune Global 500, leveraging our profound business and industry expertise, digital operations services, and proficiency in data, technology, and AI. We are currently seeking applications for the position of Lead Consultant-Data Bri...

The Lead QA Engineer is responsible for ensuring the quality and functionality of Big Data systems through automation and rigorous testing. You will lead a team, develop and execute automated test scripts, and work with cross-functional teams to ensure that testing strategies are aligned with product goals. Expertise in Python, Pytest, SQL, Apache Spark, and cloud platforms is crucial for maintaining data integrity and quality across all environments. Key Responsibilities: Test Automation & Framework Development: - Develop and automate processes for gathering expected results from data sources and comparing them with actual testing outcomes. - Own the test automation framework, ensuring its ...

We are seeking experienced and talented engineers to join our team. Your main responsibilities will include designing, building, and maintaining the software that drives the global logistics industry. WiseTech Global is a leading provider of software for the logistics sector, facilitating connectivity for major companies like DHL and FedEx within their supply chains. Our organization is product and engineer-focused, with a strong commitment to enhancing the functionality and quality of our software through continuous innovation. Our primary Research and Development center in Bangalore plays a pivotal role in our growth strategies and product development roadmap. As a Lead Software Engineer, ...

At EY, you'll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. We are counting on your unique voice and perspective to help EY become even better. Join us and build an exceptional experience for yourself, and a better working world for all. We are seeking a highly skilled and motivated Data Analyst with experience in ETL services to join our dynamic team. As a Data Analyst, you will be responsible for data requirement gathering, preparing data requirement artefacts, data integration strategies, data quality, data cleansing, optimizing data pipelines, and solutions that support business ...

The AM3 Group is looking for a highly skilled Senior Java Developer with a strong background in AWS cloud services to be a part of our dynamic team. In this role, you will have the opportunity to create and manage modern, scalable, and cloud-native applications using Java (up to Java 17), Spring Boot, Angular, and a comprehensive range of AWS tools. As a Senior Java Developer at AM3 Group, your responsibilities will include developing full-stack applications utilizing Java, Spring Boot, Angular, and RESTful APIs. You will be involved in building and deploying cloud-native solutions with AWS services such as EC2, S3, Lambda, RDS, DynamoDB, and API Gateway. Additionally, you will be tasked wit...

YASH Technologies is a leading technology integrator specializing in helping clients reimagine operating models, enhance competitiveness, optimize costs, foster exceptional stakeholder experiences, and drive business transformation. At YASH, you will be part of a team of innovative professionals working with cutting-edge technologies. Our purpose is anchored in bringing real positive changes in an increasingly virtual world, transcending generational gaps and future disruptions. We are currently seeking SQL Professionals for the role of Data Engineer with 4-6 years of experience. The ideal candidate must have a strong academic background. As a Data Engineer at BNY Mellon in Pune, you will be...

As an AWS Data Engineer, you should have at least 3 years of experience in AWS Data Engineering. Your main responsibilities will include designing and building ETL pipelines and Data lakes to automate the ingestion of both structured and unstructured data. You will need to be proficient in working with AWS big data technologies such as Redshift, S3, AWS Glue, Kinesis, Athena, DMS, EMR, and Lambda for Serverless ETL processes. Knowledge in SQL and NoSQL programming languages is essential, along with experience in batch and real-time pipelines. Your role will require excellent programming and debugging skills in either Scala or Python, as well as expertise in Spark. You should have a good unde...

As a Pyspark Developer_VIS, your primary responsibility will be to develop high-performance Pyspark applications for large-scale data processing. You will collaborate with data engineers and analysts to integrate data pipelines and design ETL processes using Pyspark. Optimizing existing data models and workflows to enhance overall performance is also a key aspect of your role. Additionally, you will need to analyze large datasets to derive actionable insights and ensure data quality and integrity throughout the data processing lifecycle. Utilizing SQL for querying databases and validating data is essential, along with working with cloud technologies to deploy and maintain data solutions. You...

We are seeking a skilled and seasoned Senior Data Engineer to become a valued member of our innovative team. The ideal candidate should possess a solid foundation in data engineering and demonstrate proficiency in Azure, particularly Azure Data Factory (ADF), Azure Fabric, Databricks, and Snowflake. In this role, you will be responsible for the design, construction, and upkeep of data pipelines, ensuring data quality and accessibility, as well as collaborating with various teams to support our data-centric initiatives. Your responsibilities will include crafting, enhancing, and sustaining robust data pipelines utilizing tools such as Azure Data Factory, Azure Fabric, Databricks, and Snowflak...

You will be responsible for exploring and visualizing data to gain insights and identify differences in data distribution that may impact model performance in real-world deployment. You will verify data quality through data cleaning and work efficiently to enhance decision-making for stakeholders using Machine Learning. Your role will involve designing and developing Machine Learning systems, conducting statistical analysis, and fine-tuning models based on test results. Additionally, you will be required to train and retrain ML systems as needed, deploy models in production, and manage cloud infrastructure costs. Developing Machine Learning applications tailored to client and data scientist ...

You should be a skilled and experienced Spark Scala Developer with a strong expertise in AWS cloud services and SQL to join our data engineering team. Your primary responsibility will be to design, build, and optimize scalable data processing systems that support our data platform. Your key responsibilities will include developing and maintaining large-scale distributed data processing pipelines using Apache Spark with Scala, working with AWS services (S3, EMR, Lambda, Glue, Redshift, etc.) to build and manage data solutions in the cloud, writing complex SQL queries for data extraction, transformation, and analysis, optimizing Spark jobs for performance and cost-efficiency, collaborating wit...

As a Lead Azure Data Engineer at CGI, you will have the opportunity to be part of a dynamic team of builders who are dedicated to helping clients succeed. With our global resources, expertise, and stability, we aim to achieve results for our clients and members. If you are looking for a challenging role that offers professional growth and development, this is the perfect opportunity for you. In this role, you will be responsible for supporting the development and maintenance of our trading and risk data platform. Your main focus will be on designing and building data foundations and end-to-end solutions to maximize the value from data. You will collaborate with other data professionals to in...

BizViz is a company that offers a comprehensive view of a business's data, catering to various industries and meeting the diverse needs of business executives. With a dedicated team of over 50 professionals working on the BizViz platform for several years, the company aims to develop technological solutions that provide our clients with a competitive advantage. At BizViz, we are committed to the success of our customers, striving to create applications that align with their unique visions and requirements. We steer clear of generic ERP templates, offering businesses a more tailored solution. As a Big Data Engineer at BizViz, you will join a small, agile team of data engineers focused on buil...

As a Databricks Engineer specializing in the Azure Data Platform, you will be responsible for designing, developing, and optimizing scalable data pipelines within the Azure ecosystem. You should have hands-on experience with Python-based ETL development, Lakehouse architecture, and building Databricks workflows utilizing the bronze-silver-gold data modeling approach. Your key responsibilities will include developing and maintaining ETL pipelines using Python and Apache Spark in Azure Databricks, implementing and managing bronze-silver-gold data lake layers using Delta Lake, and working with various Azure services such as Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), and Azure Syn...

Job Description 15 to 18 years. with at-least 3 to 4 years as expertise in ETL, data engineering and Cloud Technologies, with a proven ability to orchestrate cutting-edge technology to connect various applications within the cloud environment in a large development project. Primary Technical Skills: ETL, Apache Spark, AWS EMR, EKS, Serverless, Data Engineering, Distributed Computing, Data Lineage, Apache Airflow, Java 17+, Springboot/ Quarkus, Hibernate ORM, REST, Postgres or any RDBMS, Microservices, Cloud-native development, Secondary Technical Skills Devops : Docker, Kubernetes, CI / CD stack Jenkins or Gitlab CI, Maven, GIT, SonarQube, Nexus, AWS, expertise in at least one Data Engineeri...

about our diversity, equity, and inclusion efforts and the networks ZS supports to assist our ZSers in cultivating community spaces, obtaining the resources they need to thrive, and sharing the messages they are passionate about. ZSs Platform Development team designs, implements, tests and supports ZSs ZAIDYN Platform which helps drive superior customer experiences and revenue outcomes through integrated products analytics. Whether writing distributed optimization algorithms or advanced mapping and visualization interfaces, you will have an opportunity to solve challenging problems, make an immediate impact and contribute to bring better health outcomes. What you'll do: As part of our full-s...

Job Title: Data bricks SME Engineer Work Timing: US EST Hours Experience: 8+ years Location: Remote Job Responsibilities: Databricks Hands on SME Engineer Architect, configure, and optimize Databricks Pipelines for large-scale data processing within an Azure Data Lakehouse environment. Set up and manage Azure infrastructure components including Databricks Workspaces, Azure Containers (AKS/ACI), Storage Accounts, and Networking. Design and implement a monitoring and observability framework using tools like Azure Monitor, Log Analytics, and Prometheus/Grafana. Collaborate with platform and data engineering teams to enable micro services-based architecture for scalable and modular data solution...

Greetings, We have an opening for one of our clients " Mphasis " for " Scala Developer " position. Role & responsibilities Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field. Minimum 5 years of professional experience in Data Engineering, with a strong focus on big data technologies. Proficiency in Scala for developing big data applications and transformations, especially with Apache Spark. Expert-level proficiency in SQL; ability to write complex queries, optimize performance, and understand database internals. Extensive hands-on experience with Apache Spark (Spark SQL, DataFrames, RDDs) for large-scale data processing and analytics. Solid unders...