23086 Pyspark Jobs - Page 39

Role Overview: We are seeking street-smart and technically strong Data Engineers who can take ownership of designing and developing cuttingedge data and AI platforms using Azure-native technologies and Databricks. You will play a critical role in building scalable data pipelines, modern data architectures, and intelligent analytics solutions. Key Responsibilities: Design and implement scalable, metadata-driven frameworks for data ingestion, quality, and transformation across both batch and streaming datasets. Develop and optimize end-to-end data pipelines to process structured and unstructured data, enabling the creation of analytical data products. Build robust exception handling, logging, ...

Introduction In this role, you'll work in one of our IBM Consulting Client Innovation Centers (Delivery Centers), where we deliver deep technical and industry expertise to a wide range of public and private sector clients around the world. Our delivery centers offer our clients locally based skills and technical expertise to drive innovation and adoption of new technology. Your Role And Responsibilities As a Data Engineer at IBM, you'll play a vital role in the development, design of application, provide regular support/guidance to project teams on complex coding, issue resolution and execution. Your primary responsibilities include: Lead the design and construction of new solutions using th...

Data Engineer/Senior Data Engineer Location: Chennai Experience: 2 - 8 Years Role Description The Data Engineer will be responsible for designing and developing data engineering solutions, creating and maintaining data models, building Extract Transform Load (ETL) pipelines, implementing data warehousing, and performing data analytics on large datasets. Qualifications · Data Engineering and Data Modeling skills · Experience with Extract Transform Load (ETL) pipelines · Expertise in implementing data warehousing solutions · Analytical skills for performing data analytics on large datasets · Strong programming skills in languages like Python, Java, or Scala · Experience with Big Data and cloud...

Job Description: About Us At Bank of America, we are guided by a common purpose to help make financial lives better through the power of every connection. Responsible Growth is how we run our company and how we deliver for our clients, teammates, communities, and shareholders every day. One of the keys to driving Responsible Growth is being a great place to work for our teammates around the world. We’re devoted to being a diverse and inclusive workplace for everyone. We hire individuals with a broad range of backgrounds and experiences and invest heavily in our teammates and their families by offering competitive benefits to support their physical, emotional, and financial well-being. Bank o...

At EY, we’re all in to shape your future with confidence. We’ll help you succeed in a globally connected powerhouse of diverse teams and take your career wherever you want it to go. Join EY and help to build a better working world. EY GDS – Data and Analytics (D&A) – Manager – Azure Data Architect As part of our EY-GDS D&A (Data and Analytics) team, we help our clients solve complex business challenges with the help of data and technology. We dive deep into data to extract the greatest value and discover opportunities in key business and functions like Banking, Insurance, Manufacturing, Healthcare, Retail, Manufacturing and Auto, Supply Chain, and Finance. The opportunity We’re looking for M...

The candidate must understand the usage of data Engineering tools for solving business problems and help clients in their data journey. Must have knowledge of emerging technologies used in companies for data management including data governance, data quality, security, data integration, processing, and provisioning. The candidate must possess required soft skills to work with teams and lead medium to large teams. Candidate should be comfortable with taking leadership roles, in client projects, pre-sales/consulting, solutioning, business development conversations, execution on data engineering projects. Role Description: Developing Modern Data Warehouse solutions using Databricks and Azure St...

Job Description: About Us At Bank of America, we are guided by a common purpose to help make financial lives better through the power of every connection. Responsible Growth is how we run our company and how we deliver for our clients, teammates, communities, and shareholders every day. One of the keys to driving Responsible Growth is being a great place to work for our teammates around the world. We’re devoted to being a diverse and inclusive workplace for everyone. We hire individuals with a broad range of backgrounds and experiences and invest heavily in our teammates and their families by offering competitive benefits to support their physical, emotional, and financial well-being. Bank o...

At EY, we’re all in to shape your future with confidence. We’ll help you succeed in a globally connected powerhouse of diverse teams and take your career wherever you want it to go. Join EY and help to build a better working world. EY GDS – Data and Analytics (D&A) – Manager – Azure Data Architect As part of our EY-GDS D&A (Data and Analytics) team, we help our clients solve complex business challenges with the help of data and technology. We dive deep into data to extract the greatest value and discover opportunities in key business and functions like Banking, Insurance, Manufacturing, Healthcare, Retail, Manufacturing and Auto, Supply Chain, and Finance. The opportunity We’re looking for M...

Data Scientist Primary Skills Classification (Decision Trees, SVM), Distance (Hamming Distance, Euclidean Distance, Manhattan Distance), Forecasting (Exponential Smoothing, ARIMA, ARIMAX), Great Expectation, Evidently AI, Hypothesis Testing, ML Frameworks (TensorFlow, PyTorch, Sci-Kit Learn, CNTK, Keras, MXNet), Probabilistic Graph Models, Python/PySpark, R/ R Studio, Regression (Linear, Logistic), SAS/SPSS, Statistical analysis and computing, Tools(KubeFlow, BentoML), T-Test, Z-Test Job requirements Langraph, agentic ai

Responsibilities: Design, develop, and maintain robust web applications using modern frontend and backend technologies. Build and optimize data processing pipelines using PySpark on big data platforms. Lead or contribute to architectural decisions and technical design. Collaborate with cross-functional teams to deliver high-quality solutions. Write clean, efficient, and well-documented code. Mentor junior developers and participate in code reviews. Integrate with APIs and third-party services. Ensure application performance, security, and scalability. Troubleshoot, debug, and resolve complex technical issues. Requirements: 58 years of experience in full stack development. Strong proficiency ...

About Amgen Amgen harnesses the best of biology and technology to fight the world’s toughest diseases, and make people’s lives easier, fuller and longer. We discover, develop, manufacture and deliver innovative medicines to help millions of patients. Amgen helped establish the biotechnology industry more than 40 years ago and remains on the cutting-edge of innovation, using technology and human genetic data to push beyond what’s known today. About The Role Role Description: The role is responsible for designing, building, maintaining, analyzing, and interpreting data to provide actionable insights that drive business decisions. This role involves working with large datasets, developing repor...

At EY, we’re all in to shape your future with confidence. We’ll help you succeed in a globally connected powerhouse of diverse teams and take your career wherever you want it to go. Join EY and help to build a better working world. EY GDS – Data and Analytics (D&A) – Manager – Azure Data Architect As part of our EY-GDS D&A (Data and Analytics) team, we help our clients solve complex business challenges with the help of data and technology. We dive deep into data to extract the greatest value and discover opportunities in key business and functions like Banking, Insurance, Manufacturing, Healthcare, Retail, Manufacturing and Auto, Supply Chain, and Finance. The opportunity We’re looking for M...

At EY, we’re all in to shape your future with confidence. We’ll help you succeed in a globally connected powerhouse of diverse teams and take your career wherever you want it to go. Join EY and help to build a better working world. EY GDS – Data and Analytics (D&A) – Manager – Azure Data Architect As part of our EY-GDS D&A (Data and Analytics) team, we help our clients solve complex business challenges with the help of data and technology. We dive deep into data to extract the greatest value and discover opportunities in key business and functions like Banking, Insurance, Manufacturing, Healthcare, Retail, Manufacturing and Auto, Supply Chain, and Finance. The opportunity We’re looking for M...

About Amgen Amgen harnesses the best of biology and technology to fight the world’s toughest diseases, and make people’s lives easier, fuller and longer. We discover, develop, manufacture and deliver innovative medicines to help millions of patients. Amgen helped establish the biotechnology industry more than 40 years ago and remains on the cutting-edge of innovation, using technology and human genetic data to push beyond what’s known today. About The Role Role Description: This role is responsible for designing, developing, and implementing automation solutions to support Anaplan application workflows and data pipeline validation . The Data Engineer will leverage Python , Pytest , and Selen...

At EY, we’re all in to shape your future with confidence. We’ll help you succeed in a globally connected powerhouse of diverse teams and take your career wherever you want it to go. Join EY and help to build a better working world. EY GDS – Data and Analytics (D&A) – Manager – Azure Data Architect As part of our EY-GDS D&A (Data and Analytics) team, we help our clients solve complex business challenges with the help of data and technology. We dive deep into data to extract the greatest value and discover opportunities in key business and functions like Banking, Insurance, Manufacturing, Healthcare, Retail, Manufacturing and Auto, Supply Chain, and Finance. The opportunity We’re looking for M...

Strong expertise in Python programming: Deep knowledge of Python language features, libraries, and best practices. Extensive experience with Apache Spark and PySpark: Writing optimized Spark jobs, transformations, actions, and working with RDDs and DataFrames. Solid understanding of big data processing concepts: Distributed computing, fault tolerance, partitioning, and data shuffling. Experience with ETL pipeline development: Designing, implementing, and maintaining scalable data pipelines. Proficient in working with large datasets: Handling data ingestion, cleansing, and transformation efficiently. Good understanding of SQL: Writing complex SQL queries and integrating Spark SQL where needed...

At EY, we’re all in to shape your future with confidence. We’ll help you succeed in a globally connected powerhouse of diverse teams and take your career wherever you want it to go. Join EY and help to build a better working world. EY GDS – Data and Analytics (D&A) – Manager – Azure Data Architect As part of our EY-GDS D&A (Data and Analytics) team, we help our clients solve complex business challenges with the help of data and technology. We dive deep into data to extract the greatest value and discover opportunities in key business and functions like Banking, Insurance, Manufacturing, Healthcare, Retail, Manufacturing and Auto, Supply Chain, and Finance. The opportunity We’re looking for M...

Join us as a Software Engineer, PySpark This is an opportunity for a driven Software Engineer to take on an exciting new career challenge Day-to-day, you'll be engineering and maintaining innovative, customer centric, high performance, secure and robust solutions It’s a chance to hone your existing technical skills and advance your career while building a wide network of stakeholders We're offering this role at associate level What you'll do In your new role, you’ll be working within a feature team to engineer software, scripts and tools, as well as liaising with other engineers, architects and business analysts across the platform. You’ll Also Be Producing complex and critical software rapi...

Line of Service Advisory Industry/Sector Not Applicable Specialism Data, Analytics & AI Management Level Associate Job Description & Summary At PwC, our people in data and analytics engineering focus on leveraging advanced technologies and techniques to design and develop robust data solutions for clients. They play a crucial role in transforming raw data into actionable insights, enabling informed decision-making and driving business growth. In data engineering at PwC, you will focus on designing and building data infrastructure and systems to enable efficient data processing and analysis. You will be responsible for developing and implementing data pipelines, data integration, and data tra...

Greetings from Tata Consultancy Services (TCS) TCS has always been in the spotlight for being adept in the next big technologies. What we can offer you is a space to explore varied technologies and quench your techie soul. What we are looking for : AWS Data Engineer Location : Hyderabad Interview Mode : Virtual mode (Microsoft Teams) Design, develop and maintain scalable and robust data pipelines using Apache Spark and Java on AWS Implement ETL/ELT processes to ingest, transform, and load large datasets from various sources into data warehouse on AWS(e.g S3, Redshift) Develop and Optimize Spark applications using Java to process and analyze large Java to process and analyze large volume of d...

Required Qualifications & Skills: Bachelors or Master’s degree in Computer Science / Information Technology / Information Science or equivalent. 5–8 years of experience in Data Engineering, Data Analytics, or Data Warehousing. Minimum 2+ years of experience with AWS Cloud Data Services (S3, Glue, DMS, Redshift, RDS, Aurora). Strong hands-on experience in SQL, Python, PowerShell, and PySpark. Deep knowledge of ETL/ELT frameworks and BI tools such as SSIS, SSRS, Power BI, Athena, QuickSight. Proficiency in data migration (homogeneous & heterogeneous) using AWS DMS, SCT, and related tools. Sound understanding of AI/ML concepts and implementation on real-world datasets (minimum 1 year). AWS Asso...

Job Summary We are seeking a highly skilled Sr. Developer with 6 to 10 years of experience to join our team in a hybrid work model. The ideal candidate will have expertise in Databricks SQL Databricks Delta Lake Databricks Workflows and PySpark. This role involves developing and optimizing data solutions to drive business insights and efficiencies. The position is a day shift role with no travel requirements. Responsibilities Develop and implement data solutions using Databricks SQL to enhance business intelligence capabilities. Optimize data processing workflows with Databricks Delta Lake to ensure data integrity and performance. Design and manage Databricks Workflows to automate data pipel...

About the Role: We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. The ideal candidate will have a strong background in Python, Azure, CI/CD, deployment, and writing efficient data pipelines. You will be responsible for designing, developing, and maintaining robust data infrastructure and pipelines that support our data-driven decision-making processes. Key Responsibilities: Data Pipeline Development: Design, develop, and maintain efficient and scalable data pipelines using Python and Azure tech stack. Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet b...

SUMMARY Job Description: As a Associate SRE, you’ll support the design, deployment, and monitoring of cloud and data systems to ensure reliability, scalability, and uptime. You’ll collaborate with developers and infrastructure teams to maintain production environments and automate routine operations. Responsibilities: Support CI/CD pipeline setup and basic monitoring systems for applications and data pipelines. Assist in automating deployments using tools like Jenkins, Git, and Terraform. Monitor performance metrics and escalate incidents to senior engineers. Maintain documentation and participate in root cause analysis for minor incidents. Work under supervision to implement application and...

SUMMARY Job Summary You will own and manage the reliability of large-scale data pipelines and storage platforms. Your goal is to ensure availability, data accuracy, and performance of analytics systems while driving automation and observability. Responsibilities Design and maintain highly available data pipelines in Databricks , Azure Data Factory , or AWS Glue . Automate reliability checks, error handling, and notification systems. Implement Infrastructure as Code (IaC) for data resources using Terraform or CloudFormation . Optimize ETL performance and improve recovery mechanisms. Configure and manage data observability tools (Monte Carlo, Bigeye, or custom monitoring). Participate in incid...