15947 Pyspark Jobs - Page 25

Profile : AWS Data Engineer Mandate Skills : AWS + Databricks + Pyspark + SQL role Location : Bangalore /Pune /Hyderabad /Chennai /Gurgaon Notice Period : Immediate Key Responsibilities Design, build, and maintain scalable data pipelines to collect, process, and store data from multiple datasets Optimize data storage solutions for performance, scalability, and cost-efficiency Develop and manage ETL/ELT processes with schema transformations and data slicing/dicing Collaborate with cross-functional teams to understand requirements and accelerate feature development Create curated datasets for downstream consumption and end-user reporting Automate deployment and CI/CD processes using GitHub wor...

As a Snowflake Data Engineer at Barclays, you will play a crucial role in building and maintaining systems that collect, store, process, and analyze data, ensuring its accuracy, accessibility, and security. You will be based out of Pune and will work towards harnessing cutting-edge technology to manage robust, scalable, and secure infrastructure for seamless delivery of digital solutions. **Key Responsibilities:** - Build and maintain data architecture pipelines for transferring and processing durable, complete, and consistent data. - Design and implement data warehouses and data lakes to manage appropriate data volumes, velocity, and adhere to required security measures. - Develop processin...

Role Overview: You will be working as a Lead Spark Scala Engineer under ICG TTS Operations Technology (OpsTech) Group to help in the implementation of the next generation Digital Automation Platform and Imaging Workflow Technologies. Your role will involve managing development teams in the distributed systems Eco-System and being a strong team player. It is expected that you will have superior technical knowledge of current programming languages, technologies, and other leading-edge development tools to contribute to applications, systems analysis, and programming activities. Key Responsibilities: - Develop, test, and deploy production-grade Spark applications in Scala, ensuring optimal perf...

Job Description Where Data Does More. Join the Snowflake team. Our Solutions Engineering organization is seeking a Data Platform Architect to join our Field CTO team who can provide leadership in working with both technical and business executives in the design and architecture of the Snowflake Cloud Data Platform as a critical component of their enterprise data architecture and overall ecosystem. In this role you will work directly with the sales team and channel partners to understand the needs of our customers, strategize on how to navigate winning sales cycles, provide compelling value-based demonstrations, support enterprise Proof of Concepts, and ultimately close business. You will lev...

As a Mid-level Big Data Engineer with a strong focus on AWS, your role will involve designing, developing, and maintaining scalable data processing pipelines using Spark/PySpark and Hadoop on AWS. You will be responsible for implementing and optimizing data workflows on AWS EMR, EC2, and ECS, as well as developing and integrating RESTful APIs and AWS Gateway services using Scala/Java. Collaboration with cross-functional teams to understand data requirements and ensure data quality, performance, and reliability will be crucial. Additionally, you will participate in code reviews, testing, and deployment processes. Key Responsibilities: - Design, develop, and maintain scalable data processing p...

Where Data Does More. Join the Snowflake team. Our Solutions Engineering organization is seeking a Data Platform Architect to join our Field CTO team who can provide leadership in working with both technical and business executives in the design and architecture of the Snowflake Cloud Data Platform as a critical component of their enterprise data architecture and overall ecosystem. In this role you will work directly with the sales team and channel partners to understand the needs of our customers, strategize on how to navigate winning sales cycles, provide compelling value-based demonstrations, support enterprise Proof of Concepts, and ultimately close business. You will leverage your exper...

As an experienced professional with 3+ years of experience in building and operating production-grade applications and services across the stack, you will be responsible for the following key responsibilities: - Demonstrating strong programming skills in Python and/or Scala and SQL to write modular, testable, and well-documented code for batch and streaming workloads. - Hands-on experience with modern data engineering stacks including: - Distributed processing using Apache Spark (Databricks preferred), PySpark/Scala. - Orchestration with Azure Data Factory or Apache Airflow; implementing event-driven patterns with Azure Functions/Logic Apps. - Working with storage & formats such as Delta Lak...

Role Overview: As part of the team at American Airlines, you will play a crucial role in transforming the technology delivery process to enhance the experience for customers and team members. Your responsibilities will include working closely with source data application teams and product owners to design, implement, and support analytics solutions that provide valuable insights for decision-making. Additionally, you will be involved in implementing data migration and data engineering solutions using Azure products and services, along with traditional data warehouse tools. Your role will encompass various aspects of the development lifecycle, such as design, cloud engineering, data modeling,...

As a Mid-Level Big Data Engineer at Experian FSD in Hyderabad, India, you will be a key member of the dynamic data engineering team. Your role will involve working on cutting-edge data platforms and services, contributing to the global data ecosystem of Experian. You will play a crucial part in building scalable data pipelines, integrating APIs, and optimizing big data solutions on AWS. **Key Responsibilities:** - Design, develop, and maintain scalable data processing pipelines using Spark/PySpark and Hadoop on AWS. - Implement and optimize data workflows on AWS EMR, EC2, and ECS. - Develop and integrate RESTful APIs and AWS Gateway services using Scala/Java. - Collaborate with cross-functio...

Role Overview: As a GenAI Developer specializing in Python and Agentic AI, you will be responsible for developing and deploying GenAI applications using Python and modern AI frameworks. You will build and optimize RAG pipelines, integrate LLMs, and manage vector databases. Collaborating with cross-functional teams including data scientists, DevOps, and product managers, you will implement and maintain agentic AI workflows using orchestration tools. Your role will also involve ensuring code quality, scalability, and performance of AI-driven applications while staying updated with the latest advancements in GenAI and agentic systems. Key Responsibilities: - Develop and deploy GenAI application...

Role Overview: You have an exciting opportunity as a Software Engineer III at JPMorgan Chase within the Consumer and Community Banking - Wealth Management. In this role, you will be a crucial member of an agile team responsible for designing and delivering secure, stable, and scalable technology products to support various business functions and achieve strategic goals. Key Responsibilities: - Execute software solutions, design, development, and technical troubleshooting with innovative approaches to problem-solving. - Create secure and high-quality production code while maintaining algorithms that run synchronously with systems. - Produce architecture and design artifacts for complex applic...

Role Overview: You will be working as a Solutions Architect at Koantek, collaborating with customers to design scalable data architectures using Databricks technology and services. Your role will involve building secure, highly scalable big data solutions, focusing on achieving tangible, data-driven outcomes while ensuring simplicity and operational effectiveness. With your technical expertise and business acumen, you will engage in complex technology discussions, demonstrating the value of the Databricks platform throughout the sales process. Collaborating with Account Executives, you will interact with customers" technical leaders to become a trusted advisor who delivers concrete results. ...

As Associate Manager, Data Engineering, You Will Lead the team of Data Engineers and develop innovative approaches on performance optimization & automation Analyzing enterprise specifics to understand current-state data schema and data model and contribute to define future-state data schema, data normalization and schema integration as required by the project Apply coding expertise, best practices and guidance in Python, SQL, Informatica and cloud data platform development to members of the team Collaborate with clients to harden, scale, and parameterize code to be scalable across brands and regions Understanding business objectives and develop business intelligence applications that help to...

At Yum! We’re looking for a Software Engineer to add to our dynamic and rapid scaling team. We’re making this investment to help us optimize our digital channels and technology innovations with the end goal of creating competitive advantages for our restaurants around the globe. We’re looking for a solid lead engineer who brings fresh ideas from past experiences and is eager to tackle new challenges in our company. We’re in search of a candidate who is knowledgeable about and loves working with modern data integration frameworks, big data, and cloud technologies. Candidates must also be proficient with data programming languages (e.g., Python and SQL). The Yum! data engineer will build a var...

As a Senior Developer at MarsDevs, your role will involve designing and optimizing large-scale data pipelines using PySpark, building, training, and deploying Machine Learning and Generative AI models, writing and optimizing complex SQL queries, architecting and managing Data Warehouses, working with Microsoft Fabric for data integration and analytics, partnering with cross-functional teams to deliver AI-driven solutions, and ensuring high standards of data quality, governance, and security. Key Responsibilities: - Design and optimize large-scale data pipelines using PySpark. - Build, train, and deploy Machine Learning and Generative AI models. - Write and optimize complex SQL queries for la...

8+ years of Data Engineering experience in a Production cloud infrastructure AWS services - Glue, Redshift, RDS, CloudWatch, CloudFormation, S3, IAM etc. 5+ years experience working in a Big Data ecosystem processing data; includes file systems, data structures/databases, automation, security, messaging, movement, etc. • 6+years in Glue development and 4+ years of experience in development in Python. • Experience in translation of Data mapping documents into PySpark ETL code compatible to be able to run on Glue. • Proficiency in Python and SQL is a must.

Python (Pandas, PySpark) Data engineering & workflow optimization Delta Tables, Parquet Good-to-Have: Databricks Apache Spark, DBT, Airflow Advanced Pandas optimizations PyTest/DBT testing frameworks

Job Description The Lead Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in a collaborative and agile environment. Responsibilities: Responsible for design and development of big data solutions. Partner with domain experts, product managers, analyst, and data scientists to develop Big Data pipelines in Hadoop or Snowflake Responsible for delivering data as a service framework Responsible for moving all legacy workloads to cloud platform Work with data scientist to build Client pipelines using heterogeneous sour...

Location : Bangalore, Chennai, Hyderabad, Pune, Kolkata 6+ Years Experience in the Google Cloud on Big Data/ Analytics/ Data Lake / Datawarehouse DataProc , DataPrep, Data Plex , Cloud Bigtable , Dataflow , Cloud Composer , BigQuery, Databricks, Kafka, Nifi, CDC processing, Snowflake, Datastore , Firestore, Docker, App Engine , Spark, Cloud Data Fusion, Apigee API Management, Kafka, Attunity, Golden Gate, Map Reduce, Hadoop, Hive, HBase, Cassandra, PySpark, Flume, Hive, Impala Must Have : Design & Implement ETL/data pipeline and Data Lake/ Datawarehouse in Google Cloud Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data so...

Experience: 7+ Years Location: Bangalore Key Responsibilities Design, develop, and maintain scalable ETL pipelines using AWS Glue and PySpark . Lead the data engineering team , ensuring best practices in coding, testing, and deployment. Implement workflow orchestration solutions for complex data pipelines. Optimize data pipelines and queries for advanced SQL performance. Collaborate with data scientists, analysts, and stakeholders to deliver reliable data solutions. Ensure data quality, governance, and security across systems. Drive continuous improvements in ETL processes and architecture. Requirements 7+ years of hands-on experience in Data Engineering . Strong expertise in AWS Glue , ETL ...

About CITI Citi's mission is to serve as a trusted partner to our clients by responsibly providing financial services that enable growth and economic progress. We have 200 years of experience helping our clients meet the world's toughest challenges and embrace its greatest opportunities. Analytics and Information Management (AIM) Citi AIM was established in 2003, and is located across multiple cities in India – Bengaluru, Chennai, Pune and Mumbai. It is a global community that objectively connects and analyzes information, to create actionable intelligence for our business leaders. It identifies fact-based opportunities for revenue growth in partnership with the businesses. The function bala...

Primary skills:Technology->Architecture->Architecture - ALL,Technology->Cloud Platform->Amazon Webservices Architecture,Technology->Cloud Platform->Azure Development & Solution Architecting,Technology->Cloud Platform->Google Cloud - Architecture,Technology->Data on Cloud-DataStore->Snowflake Skill - Skill specifics Snowflake Architect Snowflake, Snowflake Architect Pyspark Architect Pyspark Developer, Pyspark Architect IICS Architect IICS, IICS Architect, Informatica Intelligent Could Services Spark Architect Spark, Spark Architect GCP Architect Google Cloud Platform BigQuery, GCP Architect AWS Architect AWS, AWS Architect Databrick Architect Databricks, databricks Architect Azure Architect ...

Working with Us Challenging. Meaningful. Life-changing. Those aren't words that are usually associated with a job. But working at Bristol Myers Squibb is anything but usual. Here, uniquely interesting work happens every day, in every department. From optimizing a production line to the latest breakthroughs in cell therapy, this is work that transforms the lives of patients, and the careers of those who do it. You'll get the chance to grow and thrive through opportunities uncommon in scale and scope, alongside high-achieving teams. Take your career farther than you thought possible. Bristol Myers Squibb recognizes the importance of balance and flexibility in our work environment. We offer a w...

5+ yr exp with Azure Data Eng Stack (Data Factory, Databricks, Synapse, Pyspark) Np-0-10 days 3 virtual interview Chennai/ Bangalore/ Hyderabad CTC upto 21 LPA Exp- Power BI, SFTP, Messaging, APIs CTC - upto 21 LPA Required Candidate profile Good understanding of Data Warehouse Architecture, Data Modelling and design concepts

a) Pyspark b) Python AWS experience 8+ Years of experience in ETL development