15947 Pyspark Jobs - Page 29

Job Title: ETL Automation Test Engineer Job Summary: We're looking for an experienced Data Pipeline Test Engineer to join our team. The successful candidate will have strong expertise in Databricks, PySpark, Delta Lake, and ETL testing, with a focus on automation and data integrity. You'll design, develop, and execute test plans for data pipelines, ensuring data accuracy, completeness, and integrity at every stage. Key Responsibilities: Design, develop, and execute test plans for data pipelines using Databricks, PySpark, and Delta Lake. Validate data accuracy, completeness, and integrity at every stage of the pipeline. Develop and maintain automated test frameworks and scripts using Python a...

For over 30 years, Beghou Consulting has been a trusted adviser to life science firms. We combine our strategic consulting services with proprietary technology to develop custom, data-driven solutions that allow life sciences companies to take their commercial operations to new heights. We Are Dedicated To Client Service And Offer a Full Suite Of Consulting And Technology Services, All Rooted In Advanced Analytics, To Enhance Commercial Operations And Boost Sales Of Job You will be responsible and accountable for development and maintenance of data pipelines. Requirements You will work with the business team in US for gathering requirements and provide efficient solutions to business request...

Job Title Manager – Data Engineering & Analytics Job Description Summary Cushman & Wakefield is seeking a highly skilled Manager – Data Engineering & Analytics to lead the design, implementation, and governance of scalable, secure, and high-performing data solutions across the enterprise. Reporting to the Director of Data Engineering & Analytics, this hands-on leadership role focuses on modernizing the data platform, guiding the adoption of emerging technologies, and ensuring alignment with organizational goals and compliance standards. As part of Cushman & Wakefield’s mission to deliver exceptional value through real estate services, this role plays a key part in enabling enterprise-wide da...

Salary: 15 to 25 LPA Exp: 4 to 7 years Location: Gurgaon/Pune/Bengalore Notice: Immediate to 30 days..!! Job Profile: Experienced Data Engineer with a strong foundation in designing, building, and maintaining scalable data pipelines and architectures. Skilled in transforming raw data into clean, structured formats for analytics and business intelligence. Proficient in modern data tools and technologies such as SQL, T-SQL, Python, Databricks, and cloud platforms (Azure). Adept at data wrangling, modeling, ETL/ELT development, and ensuring data quality, integrity, and security. Collaborative team player with a track record of enabling data-driven decision-making across business units. As a Dat...

About The Role Role Purpose The purpose of the role is to support process delivery by ensuring daily performance of the Production Specialists, resolve technical escalations and develop technical capability within the Production Specialists. Do Oversee and support process by reviewing daily transactions on performance parameters Review performance dashboard and the scores for the team Support the team in improving performance parameters by providing technical support and process guidance Record, track, and document all queries received, problem-solving steps taken and total successful and unsuccessful resolutions Ensure standard processes and procedures are followed to resolve all client que...

About The Role Role Purpose The purpose of the role is to support process delivery by ensuring daily performance of the Production Specialists, resolve technical escalations and develop technical capability within the Production Specialists. Do Oversee and support process by reviewing daily transactions on performance parameters Review performance dashboard and the scores for the team Support the team in improving performance parameters by providing technical support and process guidance Record, track, and document all queries received, problem-solving steps taken and total successful and unsuccessful resolutions Ensure standard processes and procedures are followed to resolve all client que...

About The Role Role Purpose This pivotal role is designed to bridge the gap between advanced technological frameworks and pragmatic business solutions, ensuring client needs are met with precision and efficiency. The successful candidate will leverage expertise in cloud data engineering to design, develop, and maintain scalable data infrastructure aligning with strategic business objectives. This position calls for a proficient professional capable of translating complex requirements into innovative data solutions that drive operational excellence and business value across diverse industries. As a vital contributor within our Data Analytics & AI vertical, you will engage with dynamic teams a...

Responsibilities: Work closely with product managers to understand customer needs and translate those into technical requirements. • Build and maintain scalable data management and serving infrastructure on cloud, managing terabytes of data and integrating cost. • Design a comprehensive target data architecture vision working closely with product, design and development and NatCo teams to plan transition towards the target vision • Responsible for implementation of architecture concepts in product development and services • Consult and engage with engineering teams to solve problems at their root cause • Create deliverables that provide exceptional clarity and connections to business objecti...

About Us: At icogz Technologies Pvt Ltd, we are pioneers in harnessing the power of generative AI and proprietary algorithms to transform the complexity of Big Data into clear, actionable insights. Our unique approach recognizes the challenges posed by fragmented data silos, which stem from the diverse perspectives of various stakeholders within a company. By utilizing our advanced proprietary algorithms and streamlined data processing flows, we effectively integrate and analyze vast, disparate datasets. This process delivers a unified, cohesive view of your business operations, enabling real-time analysis and insightful decision-making. Our solutions at icogz Technologies empower businesses...

Engineering Manager Experience: 9 - 19 Years Exp Salary: INR 24-45 Lacs per annum Preferred Notice Period : 30 Days Shift : 10:00 AM to 7:00 PM IST Opportunity Type: Office (Noida) Placement Type: Permanent (*Note: This is a requirement for one of Uplers' Clients) Must have skills required : Engineering manager OR J2EE OR AI/ML OR Scala and Data Engineering OR SQL OR NoSQL OR Azure and Backend OR FullStack OR Kafka Nuaav (One of Uplers' Clients) is Looking for: An Engineering Manager who is passionate about their work, eager to learn and grow, and who is committed to delivering exceptional results. If you are a team player, with a positive attitude and a desire to make a difference, then we ...

Joining: Immediate to 15 days Key Skills: Strong hands-on experience in PySpark Expertise in Big Data technologies Solid understanding of SQL, Data Pipelines, and ETL

Role- Lead Data Analyst Experience- 8+ years Location- Mumbai (onsite) Notice Period- 30 days Job Responsibilities  Gather, analyze, and interpret large datasets to deliver actionable business insights.  Write efficient SQL and PySpark queries to extract, transform, and validate data.  Work with Unity Catalog in Databricks for secure data management and governance.  Develop, maintain, and optimize Power BI dashboards and reports for business stakeholders.  Collaborate with cross-functional teams to understand reporting requirements and translate them into analytical solutions.  Ensure accuracy, consistency, and security of reporting data across systems.  Document processes, queries, a...

About The Role Grade Level (for internal use): 11 Position Summary The candidate will work along with other data scientists and data engineers to support activities related to development and maintenance of products that generate key insights on industry trend by applying various machine learning and statistical techniques on proprietary as well as open data sources. The person will work alongside Product managers, Industry Analysts and Technology experts to develop and deliver the product in accordance with customer requirements and agreed timeline. The candidate should have 10+ years work experience in developing products/services using Data Science and Analytics. Duties And Responsibiliti...

We’re Hiring – GCP Data Engineers 🚀 Join our growing team and work on cutting-edge data engineering projects in the cloud! 📍 Work Locations: Chennai / Bangalore / Pune / Hyderabad 💼 Experience: 5+ years 🔑 Skills Required Expertise in GCP ecosystem – BigQuery, Dataflow, Composer, Pub/Sub, DataProc Strong proficiency in Python, PySpark, SQL Knowledge of HDFS, Hadoop and large-scale data migration/engineering 🎯 Responsibilities Design, build, and optimize data pipelines on GCP Lead and deliver end-to-end data engineering & migration projects independently Collaborate with stakeholders to build scalable and reliable cloud data solutions 📩 Apply Now: Shaista@natobotics.com

About Client: Our Client is a global IT services company headquartered in Southborough, Massachusetts, USA. Founded in 1996, with a revenue of $1.8B, with 35,000+ associates worldwide, specializes in digital engineering, and IT services company helping clients modernize their technology infrastructure, adopt cloud and AI solutions, and accelerate innovation. It partners with major firms in banking, healthcare, telecom, and media. Our Client is known for combining deep industry expertise with agile development practices, enabling scalable and cost-effective digital transformation. The company operates in over 50 locations across more than 25 countries, has delivery centers in Asia, Europe, an...

Total Yrs. of Experience 8 years Relevant Yrs. of experience 6+ years Detailed JD (Roles and Responsibilities) Sr. Data Engineer We are seeking a highly skilled and experienced Cloud Data Engineer to join the Optum Advisory Team. The ideal candidate will bring deep expertise in cloud-based data engineering with a strong background in Azure, Databricks, Azure Data Factory (ADF), and Snowflake. This role requires a self-starter who can work independently with minimal guidance and collaborate effectively with cross-functional teams to deliver high-impact data solutions in the healthcare domain. 5+ years of experience on IT industry in Azure Tech stack 5 years of development experience using too...

Full Time Coimbatore Posted 3 months ago Experience: 4 to 8 years Work Mode: Work From Office Working Days: Monday to Friday (5 days/week) Job Description We are looking for experienced Data Engineering professionals to design, develop, and maintain large-scale data pipelines and solutions. If you’re passionate about transforming data into actionable insights and enjoy working with cutting-edge technologies, we want you on our team! Key Requirements Strong hands-on experience in Hadoop, PySpark, SCALA, and ETL development Proficiency in designing and managing data pipelines Cloud expertise: Experience in at least one platform (AWS, Azure, or GCP) Ability to troubleshoot and optimize complex ...

About the Role: Grade Level (for internal use): 11 Position Summary The candidate will work along with other data scientists and data engineers to support activities related to development and maintenance of products that generate key insights on industry trend by applying various machine learning and statistical techniques on proprietary as well as open data sources. The person will work alongside Product managers, Industry Analysts and Technology experts to develop and deliver the product in accordance with customer requirements and agreed timeline. The candidate should have 10+ years work experience in developing products/services using Data Science and Analytics. Duties and responsibilit...

Key Responsibilities Build, scale, and maintain robust data solutions to support business objectives. Design, implement, and optimize high-performance data pipelines (extraction, loading, transformation, and orchestration) with scalability, reliability, and speed. Lead end-to-end software development projects involving large language models (LLMs), retrieval-augmented generation (RAG) frameworks, and other AI technologies. Champion modern software engineering practices such as CI/CD, infrastructure-as-code, containerization, and cloud-native deployments. Collaborate with business stakeholders to transform use cases into production-ready services and solutions, owning systems from concept to ...

This is a data engineer position - a programmer responsible for the design, development implementation and maintenance of data flow channels and data processing systems that support the collection, storage, batch and real-time processing, and analysis of information in a scalable, repeatable, and secure manner in coordination with the Data & Analytics team. The overall objective is defining optimal solutions to data collection, processing, and warehousing. Must be a Spark Java development expertise in big data processing, Python and Apache spark particularly within banking & finance domain. He/She designs, codes and tests data systems and works on implementing those into the internal infrast...

You will be responsible for working with practice leaders and clients to understand business problems, industry context, data sources, potential risks, and constraints. Your role will involve translating the business program into a solvable Data Science problem by proposing different approaches and their pros and cons. You will need to identify, analyze, and interpret trends or patterns in complex data sets. Additionally, you will build Econometric/market mix models (MMM), multi-touch attribution models (MTA), optimize marketing spend, and come up with implementable recommendations and strategy/plan. Your good written, verbal, and presentation skills will be essential for generating and pres...

As a Data Engineer Grade 5 located in Hyderabad, TL on a contract-to-hire basis, you will utilize your expertise in software engineering to build solutions against delta lakes and relational databases. Your primary responsibilities will include: - Writing Python and SQL code on Azure Databricks using PySpark - Performing analysis, design, implementation, and testing with a focus on automated testing - Extending the functionality of the consistency check engine for sub-ledger data and other tenants - Delving deeply into spark performance optimization due to large data volume To excel in this role, you must meet the following qualifications: - Hold a Bachelor's or master's degree in computer s...

Role Overview: You will be working as a Capabilities and Insights Analytics (C&I-A) Analyst at QuantumBlack, where you will be part of multi-disciplinary environments utilizing data to create real-world impact for organizations globally. Your role will involve influencing recommendations for clients to enhance their businesses and performance. Additionally, you will have the opportunity to grow as a technologist and leader by working on real-life problems across various industries and technical challenges. Key Responsibilities: - Solve challenging business problems for clients across different industries globally, leading research and development of cutting-edge analytical and automation mod...

You will be responsible for Python API Development and monitoring on Credit Bureau Data, Policy Business Implementation, and Production Support. Your key responsibilities will include: - Creating Python pandas + Flask API Services across all PFL products - Automating the Analytics system Service monitoring - Providing Development, UAT & Production management support - Implementing Risk Credit Strategies - Handling Adhoc requirements To qualify for this role, you should have a minimum of 2 years of relevant work experience and have worked with Banks/NBFC's/Fintech/Credit Bureaus. Your qualifications could include M.Sc Statistics, B.E., B.Tech, MA Economics, or MBA. Additionally, you should ha...

At PwC, our people in data and analytics focus on leveraging data to drive insights and make informed business decisions. They utilise advanced analytics techniques to help clients optimise their operations and achieve their strategic goals. In data analysis at PwC, you will focus on utilising advanced analytical techniques to extract insights from large datasets and drive data-driven decision-making. You will leverage skills in data manipulation, visualisation, and statistical modelling to support clients in solving complex business problems. PwC US - Acceleration Center is seeking a highly strong analytical background to work in our Analytics Consulting practice Senior Associate’s will wor...