905 Apache Spark Jobs - Page 19

Hiring for Immediate to 15 Days Joiner for Data Engineer Position Mandatory skillset- Java Apache Spark AWS (EMR, Lambda, EKS, Airflow) Serverless Microservices Leadership

As a Data Scientist at DataBeat.io, you will have the opportunity to work with leading companies globally, shaping the technology world with breakthrough solutions and cutting-edge tools. With a Bachelor's or Master's degree in Data Science, Computer Science, Statistics, Mathematics, or related field, along with 6+ years of proven experience in data science and successful project leadership, you will play a crucial role in the big data and analytics ecosystem. Your responsibilities will include a deep understanding of advanced machine learning techniques like deep learning and ensemble methods, as well as experience in projects involving LLMs and GenAI models. Proficiency in programming lang...

As a Data Engineer, you will be responsible for building highly scalable, fault-tolerant distributed data processing systems that handle hundreds of terabytes of data daily and manage petabyte-sized data warehouses and Elasticsearch clusters. Your role will involve developing quality data solutions and simplifying existing datasets into self-service models. Additionally, you will create data pipelines that enhance data quality and are resilient to unreliable data sources. You will take ownership of data mapping, business logic, transformations, and data quality, and engage in low-level systems debugging and performance optimization on large production clusters. Your responsibilities will als...

We are seeking a highly skilled and motivated Data Engineer to join our growing team. The ideal candidate will have a strong background in building and maintaining scalable data pipelines, with hands-on experience in ETL processes, Python programming, Apache Airflow, Apache Spark, and Neo4j graph database technology. You will play a crucial role in designing, implementing, and managing our data infrastructure to support our data-driven initiatives. Responsibilities: Design, build, and maintain robust and scalable ETL pipelines to ingest data from a wide variety of sources. Develop and optimize data processing workflows and data models for efficiency and reliability. Utilize Apache Spark for ...

We are seeking a highly skilled and motivated Engineering Lead to join our team as a Cloud and Data Engineer. The ideal candidate will possess extensive expertise in cloud technologies and data engineering, with a proven ability to orchestrate cutting-edge technology to connect various applications within the cloud environment. Key Responsibilities: Lead the design and implementation of cloud-based solutions for data engineering projects. Orchestrate the integration of diverse applications and data sources across the cloud. Coordinate with various teams to ensure smooth connectivity, data transfer, and overall functionality.. Provide guidance and mentorship to junior engineers in both cloud ...

We seek a Data Engineer (Mid-level) with 46 years of hands-on experience in designing, building, and optimizing data pipelines. You will work closely with AI/ML teams to ensure data availability, quality, and performance for analytics and GenAI use cases.Key Responsibilities 1. Data Pipeline Development Build and maintain scalable ETL/ELT pipelines for structured and unstructured data. Ingest data from diverse sources (APIs, streaming, batch systems). 2. Data Modeling & Warehousing Design efficient data models to support analytics and AI workloads. Develop and optimize data warehouses/lakes using Redshift, BigQuery, Snowflake, or Delta Lake. 3. Big Data & Streaming Work with distributed syst...

Your Responsibilities Design and develop scalable data pipelines using Apache Spark and Scala Optimize and troubleshoot Spark jobs for performance (e.g. memory management, shuffles, skew) Work with massive datasets in on-prem Hadoop clusters or cloud platforms like AWS/GCP/Azure Write clean, modular Scala code using functional programming principles Collaborate with data teams to integrate with platforms like Snowflake, Databricks, or data lakes Ensure code quality, documentation, and CI/CD practices are followed Must-Have Skills 3+ years of experience with Apache Spark in Scala Deep understanding of Spark internalsDAG, stages, tasks, caching, joins, partitioning Hands-on experience with per...

As a Fullstack Developer, you will be based in Bangalore and will be responsible for developing web applications using Java, Springboot, React, and PostgreSQL or any other Relational database. Additionally, you will work with Kafka for real-time data processing. Developers with additional skills in Apache Nifi or Apache Spark will be preferred for this role. If you are passionate about fullstack development and have experience with the mentioned technologies, we encourage you to apply for this exciting opportunity.,

You should have hands-on experience in big data processing, mining, and visualization. Design and develop deep learning systems using frameworks like TensorFlow and PyTorch. Efficiently plan and continuously improve existing systems. Provide team mentorship and excel in innovating, scaling, adapting, addressing data drift, and improving system performance. Research and apply advanced AI/ML techniques, including Generative AI models (e.g., GPT, LLaMA) and NLP architectures. Experience in on-premises/Hadoop ecosystem is essential. For Principal Data Engineering roles, a minimum of 13+ years of experience is required, while for Data Engineering roles, 5+ years of experience is necessary. Proven...

As a Data Engineer at Everstream Analytics, you will be a key player in the development and maintenance of our data infrastructure. Working alongside a team of skilled engineers, you will be responsible for designing, developing, and optimizing Python-based data pipelines and products that support our cloud-native data platform. Your expertise will be crucial in utilizing various AWS services such as MSK (Kafka), Lambda, Glue, Spark, Athena, Lake Formation, Redshift, S3, and RDS to ensure the scalability, reliability, and efficiency of our data infrastructure. Your responsibilities will include architecting, developing, and owning data pipelines that manage large volumes of data from multipl...

You will be joining our team as an experienced Databricks Engineer. Your primary responsibility will involve designing, developing, and implementing big data solutions using Databricks and Apache Spark. This role requires collaborating with cross-functional teams to gather requirements and design effective big data solutions. You will be tasked with developing and maintaining Databricks notebooks, jobs, and workflows, as well as troubleshooting and resolving any big data processing issues that may arise. Additionally, optimizing Databricks solutions for performance, scalability, and cost-effectiveness will be a key part of your role. Ensuring seamless integration of big data solutions with o...

As an Azure Cloud Data Engineer at our organization based in Pune, you will play a crucial role in designing, implementing, and managing cloud-based data solutions on Microsoft Azure. Your primary responsibilities will include developing data ingestion & enrichment processes, model development/execution, and building state-of-the-art solutions for monitoring & surveillance models using technologies like Scala, Apache Spark, Azure Data Lake, Databricks, SQL, Postgres, and more. We are looking for a seasoned professional with over 5 years of experience in roles such as Data Engineer or Cloud Data Engineer, with a strong focus on Azure services. Your expertise should include working with Azure ...

The role is a 6 Months contract based in Hyderabad and Bangalore with an immediate joiner requirement. You should have 5-7 years of experience and proficiency in Python, SQL, Databricks, Snowflake Data Modelling, Data Orchestration, ETL Process, Apache Spark, PySpark, Data Integration and workflow orchestration, real-time data processing experience and frameworks, as well as Azure cloud experience. You should have a minimum of 4 years of experience in a Data Engineering role focusing on analyzing and organizing raw data, and building data systems & pipelines in the Cloud Platform, specifically Azure. Experience in migrating data from on-premises to cloud-based solution architectures is also ...

The Applications Development Technology Senior Lead Analyst is a senior-level position responsible for establishing and implementing new or revised application systems and programs in coordination with the Technology Team. Your main objective will be to lead applications systems analysis and programming activities. To excel in this role, you should possess natural analytical skills and be an innovative thinker capable of providing efficient and effective solutions to complex Technology and Business problems. Your responsibilities will include leading the integration of functions to meet goals, deploying new products, and enhancing processes. You will analyze complex business processes, syste...

As a member of our team, your primary responsibility will be the development and training of foundational models across various modalities. You will be involved in the end-to-end lifecycle management of foundational model development, starting from data curation to model deployment, by collaborating closely with core team members. Your role will also entail conducting research to enhance model accuracy and efficiency, as well as implementing state-of-the-art AI techniques in Text/Speech and language processing. Collaboration with cross-functional teams will be essential as you work towards building robust AI stacks and seamlessly integrating them into production pipelines. You will be expect...

Job Title: Senior Data Engineer Basic Function The Senior Data Engineer should be an expert familiar with all areas of data warehousing technical components (e.g., ETL, Reporting, Data Model), connected infrastructure, and their integrations. The ideal candidate will be responsible for developing the overall architecture and high-level design of the data schema environment. The candidate must have extensive experience with Star Schemas, Dimensional Models, and Data Marts. The individual is expected to build efficient, flexible, extensible, and scalable ETL design and mappings. Excellent written and verbal communication skills are required as the candidate will work very closely with diverse ...

Job Description: We are looking for a skilledPySparkDeveloperwith experience in Azure Databricks (ADB) and Azure Data Factory (ADF) to join our team. The ideal candidate will play a crucial role in designing, developing, and implementing data solutions usingPySparkfor large-scale data processing and analytics. Responsibilities: Design, develop, and deployPySparkapplications and workflows on Azure Databricks for data transformation, cleansing, and aggregation. Implement data pipelines using Azure Data Factory (ADF) to orchestrate ETL/ELT processes across heterogeneous data sources. CollaboratewithData Engineers and Data Scientists to integrate and process structured and unstructured data sets...

Role Overview We are seeking an experienced PySpark Developer with strong expertise in Azure Databricks (ADB) and Azure Data Factory (ADF). The candidate will be responsible for building, optimizing, and managing large-scale data processing solutions to enable seamless data integration, transformation, and analytics. Key Responsibilities Design, develop, and deploy PySpark applications and workflows on Azure Databricks for data transformation, cleansing, and aggregation. Build and manage data pipelines using Azure Data Factory (ADF) to support ETL/ELT processes across diverse data sources. Collaborate with Data Engineers and Data Scientists to process structured and unstructured datasets and...

Your journey at Crowe starts here with an opportunity to build a meaningful and rewarding career. At Crowe, you will enjoy real flexibility to balance work with life moments while being trusted to deliver results and make a significant impact. You will be embraced for who you are, your well-being will be cared for, and your career will be nurtured. Equitable access to opportunities for career growth and leadership is available to everyone. With a history spanning over 80 years, delivering excellent service through innovation is ingrained in our DNA across audit, tax, and consulting groups. We continuously invest in innovative ideas like AI-enabled insights and technology-powered solutions to...

As a Principal Engineer/Architect at our company, you will play a crucial role in spearheading the development and expansion of our Data & AI practice. This senior leadership position requires a blend of technical proficiency, strategic acumen, and client-oriented leadership. You will be responsible for leading intricate data migration and transformation initiatives, establishing best practices and reusable frameworks, and influencing decision-making processes at both Fortune 500 companies and innovative startups. Your expertise will be utilized across modern cloud technologies such as AWS, Azure, GCP, Databricks, and advanced Data & AI tools. Your primary responsibilities will include desig...

As a Technical Specialist - Data Engineer at Pune, India, you will be responsible for designing, developing, and delivering significant components of engineering solutions to achieve business goals. Your key responsibilities include actively participating in the design and development of new application features, ensuring solutions are maintainable and integrated successfully, and assisting junior team members as needed. You will develop and deploy source code, provide development activities for projects and technical infrastructure components, debug and provide support to the L3 team, and contribute to quality assurance through unit testing. Additionally, you will ensure architectural chang...

The company StatusNeo is a global consulting firm that specializes in cutting-edge AI, automation, and cloud-first digital solutions. With a focus on product & platform engineering, the team at StatusNeo includes top-notch software engineers and product designers who are experts in their respective fields. Located in Gurugram, StatusNeo is a Great Place To Work certified organization that values innovation and collaboration. As a GCP Data Engineer at StatusNeo, your primary responsibility will be designing, developing, and maintaining scalable data pipelines and architectures on Google Cloud Platform. Working alongside data scientists, analysts, and other stakeholders, you will ensure that d...

At FYERS, the mission is to Empower New-age Traders & Investors with the best-in-class technology and platform. The company is led by passionate founders committed to creating value for customers, employees, and communities. If you are passionate and creative and wish to embark on a journey of growth, please submit your latest profile for review. Immerse yourself in the dynamic world of capital markets and shape the future of digital finance by contributing to groundbreaking projects. Collaborate with a diverse team of experts and visionaries while elevating your career through continuous learning and growth opportunities. FYERS is a fast-scaling, data-first organization focused on building ...

The ideal candidate for this role is someone who is passionate about providing personalized service and is enthusiastic about playing a pivotal role in expanding existing business and enhancing the lifetime value of customers. You should have a proven track record in building and scaling the support department for B2B enterprise SaaS organizations. A deep understanding of customer support strategies for both on-premises and cloud-deployed services is essential for this position. You will be responsible for managing and providing support services to Tookitakis on-premises and Cloud customers worldwide, including traditional banks and fintech companies. Leading an initial 7-member support team...

As a Data Engineer with 4 to 7 years of experience and a budget range of 25 to 30 LPA, you will be responsible for handling the flow and storage of data efficiently to ensure that the model has access to the right datasets. Your role will involve tasks such as Database Design, where you will create efficient database systems using SQL, NoSQL, Hadoop, and Spark to store large volumes of transactional or behavioral data. You will also be involved in ETL Pipelines, building pipelines to collect, transform, and load data from various sources into a format suitable for analysis. Data Integration will be another key aspect of your responsibilities, where you will integrate different data sources, ...