35346 Spark Jobs - Page 34

We are seeking a highly skilled and experienced Senior Data Engineer with strong expertise in Streaming Data and Big Data Technologies , AWS & Scala Mandatory . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Key Responsibilities: Architect and implement scalable data pipelines for real-time, batch, structured, and unstructured data . Design and develop solutions using AWS services such as EMR, Kinesis, Glue, Athena, S3, CloudFormation, API Gateway . Work extensively with streaming platforms like Kafka, Flink, Spark Streaming . Develop and optimize data ingestion workflows using Apache ...

About the Role We are seeking a Senior Python Developer to lead the development of scalable, high-performance backend systems and data-driven applications. You will work on building robust services, APIs, and data workflows that power real-time and batch applications. This role is ideal for a seasoned developer who thrives on solving complex problems, building reliable software, and working across data and engineering teams. Youll play a key role in designing and developing Python-based systems that interface with big data frameworks, cloud platforms, and analytics tools. Key Responsibilities Design and develop robust Python-based backend services and microservices. Build RESTful APIs and in...

Hiring for Data Engineer Required Skills Proven experience designing and managing data flows Expertise in designing systems and APIs for data integration 8+ years of hands-on experience with Linux, Bash, Python, and SQL 4+ years working with Spark and other components in the Hadoop ecosystem 4+ years of experience using AWS cloud services, particularly: EMR Glue Athena Redshift 4+ years of experience managing a development team Deep passion for technology and enthusiasm for solving complex, real-world problems using cutting-edge tools

Looking for Your Dream Job? Join Our Ice Cream Team! Requisition id: R-43480 Role Title: Ice Cream Core HR (hire-to-retire) Global Process Owner Scope: Global Reports to: Head of HR Operations and Services Location: HQ office The Magnum Ice Cream Company in Amsterdam, The Netherlands or Pune, India Terms & Conditions: Full time If you are in the Unilever Ice Cream business or are considering working for the Unilever ice Cream business, you will work for the global, leading Ice Cream player with €8bn Turn Over in 2023. The Ice Cream business is operating in a highly attractive category, as we are part of the 1 trillion snacking and refreshment industry, growing consistently at high pace. We h...

J ob Description: Design, develop, and maintain **ETL/ELT data pipelines** to ingest data from various sources, including databases and flat files, into our data warehouse. Utilize AWS services such as **AWS Lambda**, **AWS DMS**, and **AWS Glue** for data ingestion and transformation processes. Develop and optimize **Snowflake stored procedures** to implement complex business logic and data transformations. Manage and monitor scheduled data workflows using **Apache Airflow**. Work closely with stakeholders to understand project requirements and translate them into technical solutions, ensuring seamless integration with existing data pipelines. Implement and maintain robust **data quality ch...

Experience Level : Mid to Senior [5+ Years] Department : Data Science / AI Engineering Job Summary : We are seeking a highly skilled and mathematically grounded Machine Learning Engineer to join our AI team. The ideal candidate will have 5+ years of ML experience with a deep understanding of machine learning algorithms, statistical modeling, and optimization techniques, along with hands-on experience in building scalable ML systems using modern frameworks and tools. Key Responsibilities : - Design, develop, and deploy machine learning models for real-world applications. - Collaborate with data scientists, software engineers, and product teams to integrate ML solutions into production systems...

Required Experience 4 - 6 Years Skills MongoDB, mysql, Redis + 2 more View full description img {max-height240px;} Responsibilities: Design, build, and maintain complex, scalable, low-latency, high-capacity distributed software systems using Java, C++, Python. Gather requirements from different projects for business analysis and develop Python scripts for data cleaning and transformation. Write testing code, debug programs, and integrate applications with third-party web services. Implement and automate processes using VBA, Python, Alteryx, SQL, etc. Analyze, design, develop, implement, and maintain Java/JEE applications for clients. Develop a deep understanding of existing products develope...

Required Experience 0 - 1 Years Skills DataOps img {max-height240px;} Strong proficiency in MySQL 5. x database management Decent experience with recent versions of MySQL Understanding of MySQLs underlying storage engines, such as InnoDB and MyISAM Tuning of MySQL parameters Administration of MySQL and monitoring of performance Experience with master-master replication configuration in MySQL and troubleshootingreplication Proficiency in writing complex queries, stored procedures, and triggers, eventscheduler Strong Linux shell scripting skills Have strong Unix / Shell scripting skills Familiarity with other SQL/NoSQL databases such as MongoDB, etc. desirable.* Install, Deploy and Manage Mong...

Job description Hiring for Pyspark Architect Mandatory Skills: Pyspark Education: BE/B.Tech/MCA/M.Tech/MSc./MS Responsibilities A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to actively aid the consulting team in different phases of the project including problem definition, effort estimation, diagnosis, solution generation and design and deployment You will explore the alternatives to the recommended solutions based on research that includes literature surveys, information available in public domains, vendor evaluation information, etc. and build POCs You will create requirement specifications from the business needs, define the to-be-pr...

Job description Hiring for Pyspark Architect Mandatory Skills: Pyspark Education: BE/B.Tech/MCA/M.Tech/MSc./MS Responsibilities A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to actively aid the consulting team in different phases of the project including problem definition, effort estimation, diagnosis, solution generation and design and deployment You will explore the alternatives to the recommended solutions based on research that includes literature surveys, information available in public domains, vendor evaluation information, etc. and build POCs You will create requirement specifications from the business needs, define the to-be-pr...

We are currently seeking a Data Engineer to join our team in Remote, Karntaka (IN-KA), India (IN). "Key Responsibilities: Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical stack. Provide thought leadership by recommending the most appropriate technologies and solutions for a given use case, covering the entire spectrum from the application layer to infrastructure. Demonstrate proficiency in coding skills, utilizing languages such as Python, Java, and Scala to efficiently move solutions into production while prioritizing performance, security, scalability, and ...

About the Role: We are seeking an experienced and visionary AI Architect to design, develop, and implement cutting-edge Artificial Intelligence (AI) and Machine Learning (ML) solutions. The ideal candidate will have a deep understanding of AI frameworks, data pipelines, and model deployment strategies, along with the ability to align AI initiatives with business objectives. Key Responsibilities: Lead the design and architecture of AI/ML systems , including data collection, model training, validation, and deployment. Define and implement AI strategies that drive innovation and improve business performance. Collaborate with cross-functional teams data scientists, engineers, and product owners ...

Name of Position: Databricks Lead No. of Positions : 1 Company Name: Whiteblue Cloud Services Location: Chennai, Tamil Nadu Job Description Role Overview: We are looking for an experienced Databricks Lead to design, develop, and optimize large-scale data solutions leveraging Azure Databricks, Azure Data Factory, and PySpark . The ideal candidate will have strong leadership skills, hands-on technical expertise, and a passion for driving cloud-based data modernization initiatives. Key Responsibilities Lead end-to-end data engineering and modernization projects using Azure Databricks and related technologies. Design, develop, and maintain scalable data pipelines and workflows using Azure Data F...

JOB DESCRIPTION Data Engineer Designation Data Engineer Experience 5+ Years Location Mumbai (onsite) Job Summary: We are seeking a highly skilled Data Engineer with deep expertise in Apache Kafka integration with Databricks, structured streaming, and large-scale data pipeline design using the Medallion Architecture. The ideal candidate will demonstrate strong hands-on experience in building and optimizing real-time and batch pipelines, and will be expected to solve real coding problems during the interview. Job Description: • Design, develop, and maintain real-time and batch data pipelines in Databricks. • Integrate Apache Kafka with Databricks using Structured Streaming. • Implement robust ...

About EarnIn As one of the first pioneers of earned wage access, our passion at EarnIn is building products that deliver real-time financial flexibility for those with the unique needs of living paycheck to paycheck. Our community members access their earnings as they earn them, with options to spend, save, and grow their money without mandatory fees, interest rates, or credit checks. We’re fortunate to have an incredibly experienced leadership team, combined with world-class funding partners like A16Z, Matrix Partners, DST, Ribbit Capital, and a very healthy core business with a tremendous runway. We’re growing fast and are excited to continue bringing world-class talent onboard to help sha...

Job Requisition ID # 25WD91911 Position Overview Autodesk is seeking a Principal Data Engineer to lead the design and development of data architecture, pipeline for our data team. In this role, you will shape the future of our data ecosystem, driving innovation across data pipelines, architecture, and cloud platforms. You’ll partner closely with analysts, data scientists, AI/ML Engineers and product teams to deliver scalable solutions that power insights and decision-making across the company. This is an exciting opportunity for a principal data engineer who thrives on solving complex problems, driving best practices, and mentoring high-performing teams. Responsibilities Lead and mentor a te...

Role & responsibilities Lead the design, development, and optimization of ETL/ELT pipelines using Databricks , Python, Spark , and Delta Lake . Architect scalable data solutions using Medallion architecture (Bronze, Silver, Gold layers). Design and implement data models and transformations using SQL and Python . Build and maintain audit frameworks to ensure traceability, compliance, and data lineage. Develop data quality monitoring and automated testing frameworks for pipeline reliability. Perform data analysis to support operational data requests and user queries. Collaborate with clinical data teams to analyse IRT/RTSM datasets . Create and maintain dashboards and reports using BI tools (e...

Manage hiring, building the Pre-Sales team consisting of Solutions Architects in the Data & AI domain. Rapidly scale the designated Field Engineering segment organisation without sacrificing calibre. Build a collaborative culture within a rapid-growth team. To embody and promote Databricks customer-obsessed, teamwork and diverse culture. Support increasing Return on Investment of SA involvement in sales cycles by 2-3x over 18 months. Promote a solution and value-based selling field-engineering organisation. Coach and mentor the Solutions Architect team to understand our customers business needs and identify revenue potential in their accounts. Interface with leadership & C-suite stakeholders...

Roles and Responsibilities : Design, develop, test, deploy and maintain large-scale data pipelines using Airflow on AWS cloud platform. Collaborate with cross-functional teams to identify business requirements and design scalable ETL solutions using Scala/Spark. Develop high-quality code in Java/Kafka to integrate with existing systems and ensure seamless data flow. Troubleshoot complex issues related to pipeline failures, performance optimization, and data quality. Job Requirements : 1-3 years of experience in Data Engineering with expertise in Airflow, Kafka, Scala/Spark. Strong understanding of AWS services including S3, Lambda functions, Step Functions etc. . Proficiency in developing ef...

Key Responsibilities: Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical stack. Provide thought leadership by recommending the most appropriate technologies and solutions for a given use case, covering the entire spectrum from the application layer to infrastructure. Demonstrate proficiency in coding skills, utilizing languages such as Python, Java, and Scala to efficiently move solutions into production while prioritizing performance, security, scalability, and robust data integrations. Collaborate seamlessly across diverse technical stacks, including Cloudera...

Job Duties: At least 6+ years of relevant experience in design, development, complete end-end design of enterprise-wide big data solution. Experience in designing & developing a big data solution using Spark, Scala, AWS Glue, Lambda, SNS/SQS, Cloudwatch is a must. Strong Application development experience in Scala/Python. Strong Database SQL experience, preferably Redshift. Experience in Snowflake is an added advantage. Experience with ETL/ELT process and frameworks is a must. Create integration and application technical design documentation. Conduct peer-reviews of functional design documentation. Complete development, configuration, test cases and unit testing Perform code reviews and ensu...

Job Description: About Us At Bank of America, we are guided by a common purpose to help make financial lives better through the power of every connection. Responsible Growth is how we run our company and how we deliver for our clients, teammates, communities and shareholders every day. One of the keys to driving Responsible Growth is being a great place to work for our teammates around the world. We’re devoted to being a diverse and inclusive workplace for everyone. We hire individuals with a broad range of backgrounds and experiences and invest heavily in our teammates and their families by offering competitive benefits to support their physical, emotional, and financial well-being. Bank of...

Job Title: Data Architect – B2B SaaS Location: Delhi, India Compensation: ₹83 LPA – ₹110 LPA (Based on Experience) Company Type: Well-funded B2B SaaS Product Organization Employment Type: Full-time | On-site About The Role We are looking for an accomplished Data Architect to lead and scale our data infrastructure, empowering high-performance analytics and product innovation across a fast-growing SaaS environment. The ideal candidate will bring deep technical expertise in data engineering, backend architecture, and cloud platforms , along with proven experience in leading data engineering teams within high-growth product companies. Key Responsibilities Lead and mentor a team of Data Engineers...

- The Azure Data Bricks Engineer plays a critical role in establishing and maintaining an efficient data ecosystem within an organization. -This position is integral to the development of data solutions leveraging the capabilities of Microsoft Azure Data Bricks. - The engineer will work closely with data scientists and analytics teams to facilitate the transformation of raw data into actionable insights. - With increasing reliance on big data technologies and cloud-based solutions, having an expert on board is vital for driving data-driven decision-making processes. -The Azure Data Bricks Engineer will also be responsible for optimizing data workflows, ensuring data quality, and deploying sc...

Title: Sr Software Engineer Data Desired Experience: 1-3 years Key Responsibilities: -Design, develop, and maintain robust data pipelines using Kafka, Airflow, and Spark. -Implement data models and schemas for efficient querying. -Develop and optimize code in Scala and Java for data processing tasks. -Monitor and troubleshoot data pipeline performance, ensuring data integrity and availability. -Collaborate with cross-functional teams to ensure seamless data integration and flow across systems. -Stay up-to-date with industry trends and best practices in data engineering and big data technologies. Required Skills: -1-3 years of experience in data engineering space. -Proficiency in Kafka, Airfl...