15947 Pyspark Jobs - Page 41

Key Responsibilities - Design, develop, and maintain scalable ETL pipelines for structured and unstructured data .- Work with Python and PySpark to build efficient data processing frameworks. - Implement data ingestion, transformation, and integration workflows across multiple systems. - Leverage AWS Cloud services (S3, Glue, EMR, Redshift, Lambda, Athena, etc.) for data engineering solutions. - Optimize data pipelines for performance, scalability, and cost efficiency. - Ensure data quality, integrity, and governance across all data processes. - Collaborate with data scientists, analysts, and business stakeholders to provide reliable data solutions. - Troubleshoot, debug, and resolve perform...

Job Title: Senior Data Engineer Experience: 8+ Years Location: Remote Employment Type: Full-time Role Overview We are seeking a highly skilled Senior Data Engineer with proven expertise in designing and building scalable data pipelines. The ideal candidate will have hands-on experience in Python and AWS Glue , strong SQL skills, and deep knowledge of ETL best practices, data modeling, and warehousing . This role requires someone who can work collaboratively with cross-functional teams to deliver high-quality data solutions that drive business insights. Key Responsibilities Design, develop, and optimize ETL pipelines using AWS Glue and Python. Perform complex data transformations and ensure d...

Job Family Data Science & Analysis (India) Travel Required None Clearance Required None What You Will Do Design, develop, and maintain robust, scalable, and efficient data pipelines and ETL/ELT processes. Lead and execute data engineering projects from inception to completion, ensuring timely delivery and high quality. Build and optimize data architectures for operational and analytical purposes. Collaborate with cross-functional teams to gather and define data requirements. Implement data quality, data governance, and data security practices. Manage and optimize cloud-based data platforms ( Azure\AWS). Develop and maintain Python/PySpark libraries for data ingestion, Processing and integrat...

As an Associate Data Scientist at IBM, you will work to solve business problems using leading edge and open-source tools such as Python, R, and TensorFlow, combined with IBM tools and our AI application suites. You will prepare, analyze, and understand data to deliver insight, predict emerging trends, and provide recommendations to stakeholders. In your role, you may be responsible for Implementing and validating predictive and prescriptive models and creating and maintaining statistical models with a focus on big data & incorporating machine learning. techniques in your projects Writing programs to cleanse and integrate data in an efficient and reusable manner Working in an Agile, collabora...

About The Role Project Role : Application Developer Project Role Description : Design, build and configure applications to meet business process and application requirements. Must have skills : PySpark Good to have skills : NA Minimum 5 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As an Application Developer, you will design, build, and configure applications to meet business process and application requirements. A typical day involves collaborating with various teams to understand their needs, developing innovative solutions, and ensuring that applications are aligned with business objectives. You will engage in problem-solving activiti...

An AI Data Scientist at IBM is not just a job title - it’s a mindset. You’ll leverage the watsonx,AWS Sagemaker,Azure Open AI platform to co-create AI value with clients, focusing on technology patterns to enhance repeatability and delight clients. We are seeking an experienced and innovative AI Data Scientist to be specialized in foundation models and large language models. In this role, you will be responsible for architecting and delivering AI solutions using cutting-edge technologies, with a strong focus on foundation models and large language models. You will work closely with customers, product managers, and development teams to understand business requirements and design custom AI sol...

About The Role Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : PySpark Good to have skills : NA Minimum 3 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating data pipelines, ensuring data quality, and implementing...

As a Big Data Engineer, you will develop, maintain, evaluate, and test big data solutions. You will be involved in data engineering activities like creating pipelines/workflows for Source to Target and implementing solutions that tackle the clients needs. Your primary responsibilities include: Design, build, optimize and support new and existing data models and ETL processes based on our clients business requirements. Build, deploy and manage data infrastructure that can adequately handle the needs of a rapidly growing data driven organization. Coordinate data access and security to enable data scientists and analysts to easily access to data whenever they need too Required education Bachelo...

About The Role Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : Data Engineering Good to have skills : NA Minimum 3 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating data pipelines, ensuring data quality, and imp...

About The Role Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : Data Engineering Good to have skills : NA Minimum 3 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating data pipelines, ensuring data quality, and imp...

About The Role Project Role : Application Lead Project Role Description : Lead the effort to design, build and configure applications, acting as the primary point of contact. Must have skills : Microsoft Azure Databricks Good to have skills : NA Minimum 5 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As an Application Lead, you will lead the effort to design, build, and configure applications, acting as the primary point of contact. Your typical day will involve collaborating with various teams to ensure that application development aligns with business objectives, overseeing project timelines, and facilitating communication among stakeho...

About The Role Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : Data Engineering Good to have skills : NA Minimum 3 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating data pipelines, ensuring data quality, and imp...

Greetings, Dear Candidate, Urgent requirement for-Python Engineer- CMM Level 5 client- Permanent position Experience Required: 5 to 10 years overall Location-Bangalore/Hyderabad/Pune-Hybrid Mode NP-Immediate to 30days max JD- Hands on experience in data piplines and ETL/ELT processes using Python Strong understand and experience of ELT/ETL framework like Apache Airflow, Pandas, Pyspark, Dagster, others Hands on experience in Python development using Numpy, Pandas, Keras, others. Hands on programming experience in Apache Airflow and Big Query / Azure Databricks is must If anyone interested Kindly share your cv to manasa@skyonn.com pls refer your friends and colleagues also Thank you..

At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. Generative AI Consultant for Supply Chain Business Process – Senior Consultant The opportunity We’re looking for Senior Level Consultants with expertise in Generative AI who can work on end-to-end pipelines, enabling data curation, building and fine-tuning Generative AI models, and deploying them into scalable production streams. Analytics, A...

Job Title: Data Architect ( GCP / Databricks) Location: Noida, India Experience: 10+ Years About The Role We are seeking an experienced Data Engineer Architect to design, architect, and optimize large-scale enterprise data platforms . The ideal candidate will bring deep expertise in Databricks, PySpark, Google Cloud Platform (GCP), Unity Catalog, ETL processes, and Terraform , with a proven track record of delivering enterprise-grade, scalable, and secure data engineering solutions. Key Responsibilities Architect and design modern, scalable, and secure data engineering platforms on Google Cloud Platform (GCP). Define, implement, and optimize ETL/ELT pipelines using Databricks, PySpark, and U...

Experience: 5-9 Years Location: Bangalore,Chennai,Gurgaon,Kolkata,Pune Work Mode: Hybrid Mandatory Skills: Azure Datafactory,Snowflake,ETL,Sql,python,pyspark. Primary Roles And Responsibilities Developing Modern Data Warehouse solutions using Snowflake, Databricks and ADF. Ability to provide solutions that are forward-thinking in data engineering and analytics space Collaborate with DW/BI leads to understand new ETL pipeline development requirements. Triage issues to find gaps in existing pipelines and fix the issues Work with business to understand the need in the reporting layer and develop a data model to fulfill reporting needs Help joiner team members to resolve issues and technical cha...

At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. Generative AI Consultant for Supply Chain Business Process – Senior Consultant The opportunity We’re looking for Senior Level Consultants with expertise in Generative AI who can work on end-to-end pipelines, enabling data curation, building and fine-tuning Generative AI models, and deploying them into scalable production streams. Analytics, A...

Experience: 5-9 Years Location: Bangalore,Chennai,Gurgaon,Kolkata,Pune Work Mode: Hybrid Mandatory Skills: Azure Datafactory,Snowflake,ETL,Sql,python,pyspark. Primary Roles And Responsibilities Developing Modern Data Warehouse solutions using Snowflake, Databricks and ADF. Ability to provide solutions that are forward-thinking in data engineering and analytics space Collaborate with DW/BI leads to understand new ETL pipeline development requirements. Triage issues to find gaps in existing pipelines and fix the issues Work with business to understand the need in the reporting layer and develop a data model to fulfill reporting needs Help joiner team members to resolve issues and technical cha...

Role: Principal Data Engineer Location: Delhi Mode: Hybrid Type: Contract Job Description: We are seeking a Principal Data Engineer with strong enterprise-level data architecture and engineering experience, specifically within the logistics and manufacturing domains . This role is heavily focused on architecting and developing scalable data solutions using the Azure ecosystem , including Databricks , ADF , Synapse , and PySpark . Key Responsibilities: 90% focus on data architecture and hands-on development 10% involvement in stakeholder collaboration Design and build scalable data pipelines and models in Azure Implement best practices in data engineering and architecture Required Skills: Str...

Mandatory Skills - ETL, Python, SQL, AWS Desired Skills - PySpark, Databricks, Terraform, Angular Experience - 3 years - 5 years Shift Timings - 2:30 PM -11:30 PM Location - Pune / Mumbai / Chennai / Bangalore / Hyderabad Strong experience in Python, PySpark, and SQL Angular for front-end development Experience with AWS services (S3, CloudWatch, IAM, SNS, Lambda) Terraform knowledge Experience with Git, Octopus, and Azure DevOps for CI/CD. Develop and manage ETL processes using Python, PySpark, and SQL. Work with Databricks for data processing. Utilize AWS services like S3, CloudWatch, IAM, SNS, and Lambda. Familiarity with Terraform for infrastructure management. Work with Angular for front...

Mandatory Skills - ETL, Python, SQL, AWS Desired Skills - PySpark, Databricks, Terraform, Angular Experience - 3 years - 5 years Shift Timings - 2:30 PM -11:30 PM Location - Pune / Mumbai / Chennai / Bangalore / Hyderabad Strong experience in Python, PySpark, and SQL Angular for front-end development Experience with AWS services (S3, CloudWatch, IAM, SNS, Lambda) Terraform knowledge Experience with Git, Octopus, and Azure DevOps for CI/CD. Develop and manage ETL processes using Python, PySpark, and SQL. Work with Databricks for data processing. Utilize AWS services like S3, CloudWatch, IAM, SNS, and Lambda. Familiarity with Terraform for infrastructure management. Work with Angular for front...

Roles and Responsibilities * Design, develop, and maintain large-scale data pipelines using Azure Data Factory (ADF) to extract, transform, and load data from various sources. Desired Candidate Profile * Develop complex data workflows using Azure Databricks and PySpark to process large datasets. * Troubleshoot issues with existing data pipelines and implement improvements for better performance. **Job Requirements** * 6-11 years of experience in designing and developing data pipelines using Azure Data Factory (ADF) or similar tools. * Strong understanding of big data technologies such as Hadoop, Spark, and NoSQL databases. * Proficiency in programming languages like Python or R for scripting...

Job Title : Senior Data Engineer Job Summary A Sr Data Engineer is responsible for designing, building, and maintaining scalable data pipelines and infrastructure to support data collection, transformation, and analysis. They work closely with data scientists, analysts, and business stakeholders to ensure data is accessible, reliable, and optimized for performance. Key responsibilities include developing ETL processes, managing data warehouses and lakes, ensuring data quality, and implementing best practices for data governance and security. Overall having experience between 6 to 10 years of experience in IT and minimum 3+ years experience in cloud. Hands-on in Architecture, Design, Delivery...

We are currently considering only candidates based in Bangalore who can attend in-person interviews after CV shortlisting. The Modern Data Engineer is responsible for designing, implementing, and maintaining scalable data architectures using cloud technologies, primarily on AWS, to support the next evolutionary stage of the Investment Process. They build robust data pipelines, optimize data storage, and access patterns, and ensure data quality while collaborating across engineering teams to deliver high-value data products Key Responsibilities • Implement and maintain data pipelines for ingestion, transformation, and delivery • Ensure data quality through validation and monitoring processes ...

We're hiring Data Engineers with 5+ yrs experience in SQL, Python, PySpark, and cloud platforms (Azure/AWS). Build scalable data pipelines, optimize ETL workflows, and work with Databricks/Snowflake. Cloud certs & Airflow are a plus.