23086 Pyspark Jobs - Page 37

Join Amgen’s Mission of Serving Patients At Amgen, if you feel like you’re part of something bigger, it’s because you are. Our shared mission—to serve patients living with serious illnesses—drives all that we do. Since 1980, we’ve helped pioneer the world of biotech in our fight against the world’s toughest diseases. With our focus on four therapeutic areas –Oncology, Inflammation, General Medicine, and Rare Disease– we reach millions of patients each year. As a member of the Amgen team, you’ll help make a lasting impact on the lives of patients as we research, manufacture, and deliver innovative medicines to help people live longer, fuller happier lives. Our award-winning culture is collabo...

Senior Data Engineer Bangalore, India About the Role: Seeking a Senior Data Engineer with 4.5+ years of hands-on experience in designing and building scalable data pipelines on GCP using BigQuery, Airflow/Composer, Python, and SQL . The role demands strong analytical and programming skills, CI/CD (CircleCI), Terraform expertise, and a solid grasp of ETL/ELT frameworks and data governance. Key Responsibilities: Design, develop, and maintain robust ETL/ELT pipelines on GCP. Optimize and manage BigQuery datasets for performance and scalability. Automate data workflows using Terraform, CircleCI, and Airflow. Ensure data quality, security, and compliance across systems. Collaborate with cross-fun...

A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to lead the engagement effort of providing high-quality and value-adding consulting solutions to customers at different stages- from problem definition to diagnosis to solution design, development and deployment. You will review the proposals prepared by consultants, provide guidance, and analyze the solutions defined for the client business problems to identify any potential risks and issues. You will identify change Management requirements and propose a structured approach to client for managing the change using multiple communication mechanisms. You will also coach and create a vision for ...

We are seeking a highly skilled and experienced AWS Data Engineer to join our dynamic team. The ideal candidate will have a strong background in cloud-based data engineering with at least 4 years of hands-on experience in AWS . You will be responsible for designing, building, and maintaining scalable data pipelines and solutions using modern data engineering tools and frameworks. Key Responsibilities: Design and implement robust, scalable, and high-performance data pipelines on AWS. Develop and maintain ETL workflows using Airflow . Write efficient and optimized code in Python for data processing and transformation. Work with SQL to query, manipulate, and analyze large datasets. Collaborate ...

Job Description Alimentation Couche-Tard Inc., (ACT) is a global Fortune 200 company. A leader in the convenience store and fuel space with over 17,000 stores in 31 countries, serving more than 6 million customers each day It is an exciting time to be a part of the growing Data Engineering team at Circle K. We are driving a well-supported cloud-first strategy to unlock the power of data across the company and help teams to discover, value and act on insights from data across the globe. With our strong data pipeline, this position will play a key role partnering with our Technical Development stakeholders to enable analytics for long term success. About The Role We are looking for a Data Engi...

Knowledge & Skills Must Have Skills: AWS Cloud Computing & Cloud Networking Concepts Deep understanding of AWS Well-Architected Framework Expertise in common AWS services : EC2, S3, Auto Scaling Group, RDS, Load balancers etc Platform automation IaaC : CloudFormation, Terraform Dev Ops pipelines using Jenkins Hands-on experience in Linux Shell Scripting Configuration management tool: Ansible Version control tool: GitHub Add-on Skills: Programming languages: Python Observability tool such as Datadog

Line of Service Advisory Industry/Sector Not Applicable Specialism Microsoft Management Level Associate Job Description & Summary At PwC, our people in data and analytics engineering focus on leveraging advanced technologies and techniques to design and develop robust data solutions for clients. They play a crucial role in transforming raw data into actionable insights, enabling informed decision-making and driving business growth. Those in data and automation at PwC will focus on automating data internally using automation tools or software to streamline data-related processes within the organisation. Your work will involve automating data collection, data analysis, data visualisation, or a...

Job Description: This position will be responsible for design, build and maintenance of data pipelines running on Airflow, Spark on the AWS Cloud platform. Roles and Responsibility: Build and maintain all facets of Data Pipelines for Data Engineering team. Build the pipelines required for optimal extraction, transformation, and loading of data from a wide variety of data sources using Spark, Python and Airflow. Work with internal and external stakeholders to assist with data-related technical issues and data quality issues. Engage in proof of concepts, technical demos and interaction with customers and other technical teams Participate in the agile ceremonies Ability to solve complex data-dr...

As a Data Engineer at our company, your role will involve designing, developing, and maintaining data pipelines to extract data from Oracle Symphony via APIs, process and store it in the Databricks Lakehouse platform, and integrate it into Oracle EPM for financial planning, reporting, and analytics. Key Responsibilities: - Design and implement end-to-end pipelines from Oracle Symphony into Databricks Lakehouse. - Develop efficient ETL/ELT processes in Databricks (PySpark, Delta Lake) to transform, cleanse, and enrich data. - Build and maintain data flows from Databricks into Oracle EPM to support reporting, forecasting, and planning. - Ensure data quality, consistency, and governance across ...

Job Description: You will be responsible for leveraging your strong expertise in Snowflake, Hadoop ecosystem, PySpark, and SQL to enable data-driven decision-making across the organization. Key Responsibilities: - Utilize your expertise in Snowflake, Hadoop ecosystem, PySpark, and SQL to support data-driven decision-making. - Play a key role in enabling the organization to make informed decisions based on data analysis. Qualifications Required: - Strong expertise in Snowflake, Hadoop ecosystem, PySpark, and SQL. Note: No additional details about the company were provided in the job description. Job Description: You will be responsible for leveraging your strong expertise in Snowflake, Hadoop...

Job Description: We are looking for a Developer skilled in Python and modern cloud and data technologies. The role involves working on projects using AWS Cloud, Scala, PowerShell, Terraform, and PySpark , contributing to the development, automation, and optimization of digital systems. Key Responsibilities: Work with Python, AWS Cloud Computing, Scala, PowerShell, Terraform, and PySpark to develop and deploy robust applications. Build and maintain scalable and efficient systems using modern DevOps and cloud tools. Support automation, deployment, and configuration management processes. Collaborate with cross-functional teams to ensure high-quality project delivery. Essential Skills: Python (p...

Job Description: This position will be responsible for design, build and maintenance of data pipelines running on Airflow, Spark on the AWS Cloud platform. Roles and Responsibility: Build and maintain all facets of Data Pipelines for Data Engineering team. Build the pipelines required for optimal extraction, transformation, and loading of data from a wide variety of data sources using Spark, Python and Airflow. Work with internal and external stakeholders to assist with data-related technical issues and data quality issues. Engage in proof of concepts, technical demos and interaction with customers and other technical teams Participate in the agile ceremonies Ability to solve complex data-dr...

Role Overview: You will be responsible for designing and building data pipelines, Lakehouse architectures (BronzeSilverGold), and semantic models in Microsoft Fabric. Your role will involve developing data ingestion and transformation workflows using Data Factory, Dataflows Gen2, SQL, and PySpark. Additionally, you will integrate Fabric with Power BI for analytics and reporting, ensuring data quality, governance, and performance optimization. It will be crucial for you to implement DevOps (CI/CD, Git) practices for Fabric assets and stay updated with Microsoft Fabric roadmap (Cortex AI, Copilot, Real-Time Intelligence). Key Responsibilities: - Design and build data pipelines, Lakehouse archi...

Role & responsibilities Design and Develop Data Pipelines: Create, maintain, and optimize scalable Extract, Transform, Load (ETL) and ELT pipelines using PySpark on a distributed computing environment (e.g., Databricks, AWS EMR, Azure Synapse). • Code Migration and Modernization: Lead the effort to re-engineer existing Python-based data processes, functions, and analytical logic (including legacy systems or complex Pandas transformations) into efficient and performant PySpark code. • Performance Tuning: Profile, optimize, and fine-tune PySpark jobs for maximum speed and efficiency, focusing on techniques like partitioning, caching, broadcast variables, and query optimization to handle teraby...

At American Express, our culture is built on a 175-year history of innovation, shared values and Leadership Behaviors, and an unwavering commitment to back our customers, communities, and colleagues. As part of Team Amex, you'll experience this powerful backing with comprehensive support for your holistic well-being and many opportunities to learn new skills, develop as a leader, and grow your career. Here, your voice and ideas matter, your work makes an impact, and together, you will help us define the future of American Express. How will you make an impact in this role? There are hundreds of opportunities to make your mark on technology and life at American Express. Here’s just some of wha...

🚀 We're Hiring: Data Engineer (2–4 Years Experience) About AEWEE AEWEE is a data engineering and analytics company focused on helping global organizations unlock the power of data. We leverage cloud technologies, data pipelines, and advanced analytics to deliver scalable solutions that drive business growth and efficiency. Our team is passionate about innovation, automation, and building modern data ecosystems that empower smarter decision-making across industries. About the Role We’re looking for a skilled Data Engineer to join our growing data team at AEWEE. If you’re passionate about designing scalable data solutions, building pipelines, and working with Azure and Databricks, this is the ...

Position: Azure Data Engineer Location: Remote Duration: 12 Months Contract Shift Timing: 11:00 AM – 8:00 PM (India Time) Pay - Rs 65K/Month Job Description: Looking for 5+ Years experience in Advanced SQL, Azure Data Factory (ADF), Databricks (with PySpark). Experience with Azure Synapse (added advantage). Strong understanding of Big Data processing approaches and schema-on-read methodologies. Hands-on experience with Azure Data Factory, Azure Databricks (PySpark), and Azure Data Lake Storage (ADLS Gen2). Strong development and coding experience in Python (mandatory – PySpark) and SQL. Knowledge of Power BI DAX (good to have). Proven experience in implementing robust ETL/ELT pipelines withi...

Job ID: Pyt-ETP-Pun-1125 Location: Pune Company Overview Bridgenext is a Global consulting company that provides technology-empowered business solutions for world-class organizations. Our Global Workforce of over 800 consultants provide best in class services to our clients to realize their digital transformation journey. Our clients span the emerging, mid-market and enterprise space. With multiple offices worldwide, we are uniquely positioned to deliver digital solutions to our clients leveraging Microsoft, Java and Open Source with a focus on Mobility, Cloud, Data Engineering and Intelligent Automation. Emtec’s singular mission is to create “Clients for Life” – long-term relationships that...

As a Data Analyst within the company, your role will involve partnering closely with business and S&T teams to prepare final analysis reports for stakeholders. This will enable them to make important decisions based on various facts and trends. You will lead data requirement, source analysis, data analysis, data transformation, and reconciliation activities. Your interactions will include teams such as DG, DPM, EA, DE, EDF, PO, and D &Ai for historical data requirements and sourcing data for the Mosaic AI program to scale solutions to new markets. Your key responsibilities will include: - Leading data requirement, source analysis, data analysis, data transformation, and reconciliation activi...

About This Role Want to elevate your career by being a part of the world's largest asset manager? Do you thrive in an environment that fosters positive relationships and recognizes stellar service? Are analyzing complex problems and identifying solutions your passion? Look no further. BlackRock is currently seeking a candidate to become part of our Global Investment Operations Data Engineering team. We recognize that strength comes from diversity, and will embrace your rare skills, eagerness, and passion while giving you the opportunity to grow professionally and as an individual. We know you want to feel valued every single day and be recognized for your contribution. At BlackRock we strive...

Pyspark + SQL - 6to 10 yrs - (Preferred Chennai) Role & responsibilities: Outline the day-to-day responsibilities for this role. Preferred candidate profile: Specify required role expertise, previous job experience, or relevant certifications.

Design develop and maintain scalable data pipelines using Apache Spark on Databricks Write efficient and production ready PySpark or Scala code for data transformation and ETL processes Integrate data from various structured and unstructured sources into a unified platform Implement Delta Lake and manage data versioning updates and schema evolution Optimize data processing workflows for performance scalability and cost efficiency Collaborate with data scientists analysts and business stakeholders to deliver high quality datasets Implement data quality checks validation routines and logging mechanisms Monitor and debug production jobs using Databricks jobs notebooks and clusters Ensure securi...

Bachelor s degree in Data Science, Statistics, Mathematics, Computer Science, Economics, or a related field. 8+ years of experience in data engineering, with a focus on scalable architecture design. Strong proficiency in Python, PySpark, Python-based UI libraries and Gradio or Streamlit. Strong proficiency in Python, especially in Pandas and NumPy. Strong proficiency in Databricks environment and Native Apps Hands-on experience with Databricks (notebooks, jobs, lakehouse architecture). Experience building charts and dashboards using Matplotlib, Seaborn, or Plotly. Experience in python webapp and rest API development Good to have understanding of SAS (PROC steps, data handling, stored process...

Analyze existing Hadoop, Pig, and Spark scripts from Dataproc and refactor them into Databricks-native PySpark. Implement data ingestion and transformation pipelines using Delta Lake best practices. Apply conversion rules and templates for automated code migration and testing. Conduct data validation between legacy and migrated environments (schema, count, and data-level checks). Collaborate on developing AI-driven tools for code conversion, dependency extraction, and error remediation. Ensure best practices for code versioning, error handling, and performance optimization. Participate in UAT, troubleshooting, and post-migration validation activities. Technical Skills Core: Python, PySpark, ...

Lead design and execution of Dataproc Databricks PySpark migration roadmap. Define modernization strategy , including data ingestion, transformation, orchestration, and governance. Architect scalable Delta Lake and Unity Catalog -based solutions. Manage and guide teams on code conversion, dependency mapping, and data validation. Collaborate with platform, infra, and DevOps teams to optimize compute costs and performance. Own the automation & GenAI acceleration layer , integrating code parsers, lineage tools, and validation utilities. Conduct performance benchmarking, cost optimization, and platform tuning (Photon, Auto-scaling, Delta Caching). Mentor senior and mid-level developers, ensuring...