23086 Pyspark Jobs - Page 13

Role Overview: As a Data Engineer at our company, you will be responsible for developing, optimizing, and managing robust ETL pipelines to handle large-scale data processing and transformation. You will be working on building and maintaining data lake architectures on AWS S3 and data warehouse solutions on Redshift for efficient storage and retrieval. Additionally, you will use Databricks for advanced data processing, pipeline orchestration, and integration with downstream AI/ML systems. Your role will also involve leveraging various AWS services for scalable, production-grade data solutions and developing data processing logic in Python, PySpark, SQL, and other programming languages. Key Re...

Relevant experience in Pyspark and Azure Databricks. Proficiency in integrating, transforming, and consolidating data from various structured and unstructured data sources. Good experience in SQL or native SQL query languages. Strong experience in implementing Databricks notebooks using Python. Good experience in Azure Data Factory, ADLS, Storage Services, Serverless architecture, Azure functions. Exposure to basic issues in working within a Cloud (Azure) environment. Advanced working SQL knowledge and experience working with relational/non-Relational databases, query. Exposure to Jira and Confluence required

Role & responsibilities Job Title: Data Engineer Experience: 3 - 7 years Location: Hybrid Employment Type: Full-time About the Role We are looking for a Data Engineer who will play a key role in designing, building, and maintaining robust data pipelines that power our AI-based analytics and dashboarding solutions. The candidate should have a strong foundation in data modeling, ETL workflows, and cloud-based data infrastructure. Youll work closely with analysts, AI engineers, and business stakeholders to ensure data reliability, scalability, and performance across all projects. Key Responsibilities Design, develop, and maintain data pipelines to collect, clean, and process structured and unst...

As a Manager at EY in the Consulting Microsoft Fabric team, your role will involve designing, developing, and maintaining distributed systems using Microsoft Fabric, including One Lake, Azure Data Factory (ADF), Azure Synapse, Notebooks, Data Warehouse, and Lakehouse. You will play a crucial role in architecting and implementing enterprise data platforms and data management practices, ensuring the delivery of high-quality solutions that meet business requirements. Collaborating with system architects, business analysts, and stakeholders is essential to understand their requirements and convert them into technical designs. Your responsibilities will include designing, building, testing, deplo...

At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. AI Automation Architect for Customer Business Process – Manager The opportunity We’re looking for Manager Level Architects with expertise in Generative AI who can work on end-to-end pipelines, enabling data curation, building and fine-tuning Generative AI models, and deploying them into scalable production streams. Analytics, AI innovation, a...

Role & responsibilities 1. Data Architecture Design: Design and implement scalable, reliable, and maintainable data architectures. 2. Data Integration: Develop ETL processes to integrate data from various sources into a centralized data warehouse. Ensure data quality and integrity throughout the integration process. 3. Database Management: Administer and optimize databases for high performance and availability. Implement security measures to safeguard data against unauthorized access. 4. Data Modelling: Create and maintain data models for efficient storage and retrieval of information. Collaborate with data scientists and analysts to translate data needs into effective structures. 5. Coding ...

At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. AI Automation Architect for Customer Business Process – Manager The opportunity We’re looking for Manager Level Architects with expertise in Generative AI who can work on end-to-end pipelines, enabling data curation, building and fine-tuning Generative AI models, and deploying them into scalable production streams. Analytics, AI innovation, a...

At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. AI Automation Architect for Customer Business Process – Manager The opportunity We’re looking for Manager Level Architects with expertise in Generative AI who can work on end-to-end pipelines, enabling data curation, building and fine-tuning Generative AI models, and deploying them into scalable production streams. Analytics, AI innovation, a...

At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. AI Automation Architect for Customer Business Process – Manager The opportunity We’re looking for Manager Level Architects with expertise in Generative AI who can work on end-to-end pipelines, enabling data curation, building and fine-tuning Generative AI models, and deploying them into scalable production streams. Analytics, AI innovation, a...

Introduction In this role, you'll work in one of our IBM Consulting Client Innovation Centers (Delivery Centers), where we deliver deep technical and industry expertise to a wide range of public and private sector clients around the world. Our delivery centers offer our clients locally based skills and technical expertise to drive innovation and adoption of new technology. Your Role And Responsibilities As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and Azure Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, ...

Responsibilities: Only 8+ allowed Design, develop & maintain ETL processes using AWS Glue, lambda, Python & SQL Optimize performance with Airflow workflows or Step Functions Sound knowledge on Python, spark & optimisation techniques Knowledge of Redshift Office cab/shuttle

o Python (strong proficiency) o PySpark, Spark o FastAPI o Cloud deployment experience o Kubernetes o Familiarity with LangSmith, LangFuse, or similar LLM monitoring tools.

Responsibilities A day in the life of an InfoscionAs part of the Infosys delivery team, your primary role would be to provide best fit architectural solutions for one or more projects. You would also provide technology consultation and assist in defining scope and sizing of work You would implement solutions, create technology differentiation and leverage partner technologies. Additionally, you would participate in competency development with the objective of ensuring the best-fit and high quality technical solutions. You would be a key contributor in creating thought leadership within the area of technology specialization and in compliance with guidelines, policies and norms of Infosys.If y...

Company Name: Top MNC Position Name: Sr.Python Developer Experience: 4 to 8 yrs & 8 to 12 yrs Location: Chennai & Bangalore Notice Period: Immediate Or Serving NP (Below 30 Days) Primary Skills: Strong Coding Exp in Python, Spark & Hive with BigData Environments Exposure Job Description Strong expertise in Python (Advanced OOP, multithreading, data structures, exception handling). Hands-on experience with Apache Spark / PySpark. Strong working knowledge of Hive (HiveQL, partitions, performance tuning). Knowledge of Big Data technologies including: HDFS, YARN Kafka (optional but preferred) Airflow / Oozie / Luigi for workflow orchestration SQL and NoSQL databases Experience handling large-sca...

Lumen connects the world. We are igniting business growth by connecting people, data and applications – quickly, securely, and effortlessly. Together, we are building a culture and company from the people up – committed to teamwork, trust and transparency. People power progress. We’re invested in providing the flexibility you need to thrive and deliver lasting impact. Apply now to continue digitally connecting the world and shaping the future Should have knowledge of data warehousing concepts and best practices, Extract, Transfer and Load (ETL), performance tuning Should have good knowledge of Oracle, PLSQL, SQL, Unix, Shell Scripting Should have good knowledge of Informatica Exposure to Jav...

Role: Data Engineering Remote Looking for 5-10 years of Exp • Design, develop & maintain ETL/ELT pipelines • Ingest & transform data from APIs, DBs, files, streams • Build real-time & batch processing solutions • Data validation, quality & cleansing • Translate business needs into data models • Ensure data security, access control & compliance • Maintain documentation & best practices Snowflake, AWS/GCP/Azure, Databricks (Lakehouse + Spark), BigQuery (Google Cloud), Azure Synapse / AWS Redshift, Kafka, Azure Data Factory, AWS Glue, Apache Spark, Apache Hadoop, Airflow, dbt, PySpark, SQL, Real-time & Batch Pipelines, Data Quality & Governance, Informatica, Talend, Collibra.

Our Purpose Title and Summary Consultant, Advisors & Consulting Services, Performance Analytics Consultant Performance Analytics Advisors & Consulting Services Services within Mastercard is responsible for acquiring, engaging, and retaining customers by managing fraud and risk, enhancing cybersecurity, and improving the digital payments experience. We provide value-added services and leverage expertise, data-driven insights, and execution. Our Advisors & Consulting Services team combines traditional management consulting with Mastercards rich data assets, proprietary platforms, and technologies to provide clients with powerful strategic insights and recommendations. Our teams work with a div...

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together. Primary Responsibilities Analyze business requirements & functional specifications Be able to determine the impact ...

About The Role Project Role : Data Architect Project Role Description : Define the data requirements and structure for the application. Model and design the application data structure, storage and integration. Must have skills : Microsoft Azure Analytics Services Good to have skills : NA Minimum 12 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a Data Architect, you will define the data requirements and structure for the application. Your typical day will involve modeling and designing the application data structure, storage, and integration, ensuring that the architecture aligns with business needs and technical specifications. You wil...

About The Role Project Role : Application Lead Project Role Description : Lead the effort to design, build and configure applications, acting as the primary point of contact. Must have skills : PySpark Good to have skills : NA Minimum 5 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As an Application Lead, you will lead the effort to design, build, and configure applications, acting as the primary point of contact. You will be responsible for overseeing the application development process and ensuring successful project delivery. Roles & Responsibilities:- Expected to be an SME- Collaborate and manage the team to perform- Responsible for t...

About The Role Job Title - S&C Global Network - AI - Supply Chain Analytics - Consultant Management Level: 9-Team Lead/Consultant Location: Bengaluru, BDC7C Must-have skills: Supply Chain Analytics Good to have skills: Ability to leverage design thinking, business process optimization, and stakeholder management skills. Job Summary : This role involves driving strategic initiatives, managing business transformations, and leveraging industry expertise to create value-driven solutions. Candidate should have good understanding of statistics/analytical/optimization methods and approaches. The ideal candidate should be able to bring in meaningful data driven insights supporting with statistical c...

About The Role Job Title - S&C Global Network - AI - Supply Chain Analytics - Consultant Management Level: 9-Team Lead/Consultant Location: Bengaluru, BDC7C Must-have skills: Supply Chain Analytics Good to have skills: Ability to leverage design thinking, business process optimization, and stakeholder management skills. Job Summary : This role involves driving strategic initiatives, managing business transformations, and leveraging industry expertise to create value-driven solutions. Candidate should have good understanding of statistics/analytical/optimization methods and approaches. The ideal candidate should be able to bring in meaningful data driven insights supporting with statistical c...

About The Role Job Title - S&C Global Network - AI - Supply Chain Analytics - Manager Management Level: 7-Manager Location: Gurugram, DDC1A, NonSTPI Must-have skills: Supply Chain Analytics Good to have skills: Ability to leverage design thinking, business process optimization, and stakeholder management skills. Job Summary : This role involves driving strategic initiatives, managing business transformations, and leveraging industry expertise to create value-driven solutions. Roles & Responsibilities: Provide strategic advisory services, conduct market research, and develop data-driven recommendations to enhance business performance. Professional & Technical Skills: - Relevant experience in ...

About The Role Job Title - S&C Global Network - AI - Supply Chain Analytics - Consultant Management Level: 9-Team Lead/Consultant Location: Bengaluru, BDC7C Must-have skills: Supply Chain Analytics Good to have skills: Ability to leverage design thinking, business process optimization, and stakeholder management skills. Job Summary : This role involves driving strategic initiatives, managing business transformations, and leveraging industry expertise to create value-driven solutions. Candidate should have good understanding of statistics/analytical/optimization methods and approaches. The ideal candidate should be able to bring in meaningful data driven insights supporting with statistical c...

About The Role Job Title - S&C Global Network - AI - Supply Chain Analytics - Consultant Management Level: 9-Team Lead/Consultant Location: Bengaluru, BDC7C Must-have skills: Supply Chain Analytics Good to have skills: Ability to leverage design thinking, business process optimization, and stakeholder management skills. Job Summary : This role involves driving strategic initiatives, managing business transformations, and leveraging industry expertise to create value-driven solutions. Candidate should have good understanding of statistics/analytical/optimization methods and approaches. The ideal candidate should be able to bring in meaningful data driven insights supporting with statistical c...