28509 Pyspark Jobs - Page 2

Company Description Version 1 has celebrated over 28 years in Technology Services and continues to be trusted by global brands to deliver solutions that drive customer success. Version 1 has several strategic technology partners including Microsoft, AWS, Oracle, Red Hat, OutSystems and Snowflake. We’re also an award-winning employer reflecting how employees are at the heart of Version 1. We’ve been awarded: Innovation Partner of the Year Winner 2023 Oracle EMEA Partner Awards, Global Microsoft Modernising Applications Partner of the Year Award 2023, AWS Collaboration Partner of the Year - EMEA 2023 and Best Workplaces for Women by Great Place To Work in UK and Ireland 2023. As a consultancy ...

Condé Nast is a global media company producing the highest quality content with a footprint of more than 1 billion consumers in 32 territories through print, digital, video and social platforms. The company’s portfolio includes many of the world’s most respected and influential media properties including Vogue, Vanity Fair, Glamour, Self, GQ, The New Yorker, Condé Nast Traveler/Traveller, Allure, AD, Bon Appétit and Wired, among others. Job Description Location: Bengaluru, KA About The Role: Condé Nast is seeking a motivated and skilled Machine Learning Engineer I to support the productionization of machine learning projects in Databricks or AWS environments for the Data Science team. This r...

Want to be a bswifter? At bswift we've been transforming benefits administration since 1996, making it simpler, smarter, and more human. Our state-of-the-art, cloud-based technology and services empower employees to understand, manage, and love their benefits. From downtown Chicago, and remotely across the country, we serve thousands of companies and millions of people nationwide, reducing administrative burdens and freeing HR teams to focus on creating thriving, people-first workplaces. We're looking for motivated and goal-driven individuals who share our passion for delivering excellence and creating solutions that make a difference. The reward is a fun, flexible and creative environment w...

Country/Region: IN Requisition ID: 31949 Work Model: Position Type: Salary Range: Location: INDIA - BENGALURU - HP Title: Sr Technical Lead-Data Engg Description: Area(s) of responsibility Skills: Azure Data Engineer Experience: 8 to 10 Years , 4+ Years of experience in Azure Data Services Location: Mumbai/Pune/ Noida/ Bangalore/ Hyderabad/Chennai Azure Lead with experience in Azure ADF, ADLS Gen2, Databricks, PySpark and Advanced SQL Responsible for designing and implementing secure, scalable, and highly available cloud-based solutions and estimation on Azure Cloud 4 Years of experience in Azure Databricks and PySpark Experience in Performance Tuning Experience with integration of different...

CGI Information Systems and Management Consultants is looking for Hadoop Developer to join our team Roles and Responsibility Design, develop, and implement Hadoop-based solutions for complex data processing needs. Collaborate with cross-functional teams to identify business requirements and design scalable data architectures. Develop high-quality code that meets performance, security, and reliability standards. Troubleshoot and optimize Hadoop clusters for optimal performance. Participate in code reviews to ensure adherence to coding standards and best practices. Stay updated with the latest trends and technologies in Hadoop development. Job Requirements Strong understanding of Hadoop ecosys...

Responsibilities: * Design, develop & maintain Azure data solutions using Databricks, Synapse, Factor, PySpark & SQL. * Collaborate with cross-functional teams on project delivery & optimization.

Role & responsibilities Job description Role Overview The Data Engineer will be responsible for designing, building, and optimizing data pipelines and platforms to enable advanced analytics and business intelligence across the organization. This role requires strong expertise in Databricks , SQL , AWS , and Airflow , combined with a solid foundation in data analysis and engineering best practices. The ideal candidate will collaborate with cross-functional stakeholders and ensure scalable, secure, and high-performing data solutions. Location: Bengaluru Key Responsibilities Technical Design Develop and maintain data pipelines and ETL workflows using Databricks and Airflow. Good understanding o...

Job Title: Consultant/ Sr. Consultant - Data Engineer Job Summary: Responsible for implementation of data ETL to on-board and ingest data sets to cloud data management platforms. Work closely with analytics and data science teams to support their end-to-end data acquisition needs. Interact with internal stakeholders to ensure data capabilities are aligned with overall data strategy. Key Responsibilities: Develop and maintain ETL pipelines to support data processing, analytics, and visualization needs. Collaborate with cross-functional teams to understand business requirements and provide data-driven solutions. Manage multiple disparate data sources and integrate them into a cohesive data pla...

Job Title: Data Engineer Location: Bengaluru Experience: 5+Years (Work from Office) Job Summary: We are looking for a skilled and experienced Data Engineer with over 5 years of experience in data engineering and data migration projects. The ideal candidate should possess strong expertise in SQL, Python, data modeling, data warehousing, and ETL pipeline development. Experience with big data tools like Hadoop and Spark, along with AWS services such as Redshift, S3, Glue, EMR, and Lambda, is essential. This role provides an excellent opportunity to work on large-scale data solutions, enabling data-driven decision-making and operational excellence. Key Responsibilities: Design, build, and mainta...

SUMMARY Role Title Application Security Specialist (L3) SSDLC Operations Operational hours 24 7 Role Purpose Own operational SSDLC for 180 in-house apps + 900 third party/SaaS , drive adoption of Client SSDLC processes/tools, integrate security in Agile/DevOps/CI CD , coordinate remediation, and deliver pre release security reviews and monthly reporting. Key Responsibilities Drive SSDLC adoption across design build test release operate Conduct security reviews before release and enforce gates Integrate SAST/SCA/Secrets (Checkmarx, GitHub AS) and ASPM (Apiiro) into CI/CD (GitHub Actions/PR checks) Coordinate DAST enablement with DAST Ops; ensure pipelines block on critical findings Triage fin...

Key Responsibilities Design, develop, and maintain scalable data pipelines and ETL processes Build and operate data platforms using modern cloud and big data technologies Act as an individual contributor (~60%), delivering high-quality, production-ready code Coach and mentor data engineers on best practices and engineering standards Perform code reviews, design reviews, and ensure testing and deployment quality Monitor and optimize data infrastructure performance and reliability Resolve complex technical challenges and manage technical debt Ensure DevSecOps practices are embedded into daily engineering work Collaborate with Engineering Managers, Product Owners, Business Analysts, and Scrum M...

Minimum 5+ years of development and design experience in experience as Data Engineer Experience on Big Data platforms and distributed computing (Hadoop, Spark, HBase, Hive) Experience in data pipeline software engineering and best practice in python

Responsibilities Design, develop, and maintain data pipelines using Azure Data Factory. Manage and optimize data storage using Azure Data Lake Gen 2. Build and process large-scale data solutions with Azure Databricks and Apache Spark. Create interactive reports and dashboards in Power BI for business insights. Collaborate with cross-functional IT and business teams to translate requirements into data solutions. Ensure data quality, security, and compliance in line with organizational standards. Core Skills Required Azure Data Factory: Experience in building and managing data pipelines. Azure Data Lake Gen 2: Proficient in data storage and management within Azure's data lake environment. Azur...

Key Responsibilities Design, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks and Apache Spark. Build data ingestion, transformation, and processing frameworks for structured and unstructured data. Work with Azure services including ADLS Gen2, Azure Data Factory (ADF), Azure Synapse, Azure SQL, Delta Lake, and Event Hubs. Optimize and tune Spark jobs for performance, cost efficiency, and scalability. Implement advanced analytics and machine learning workflows using Databricks notebooks. Ensure data quality, data governance, and compliance throughout the data lifecycle. Collaborate with data architects, analysts, and business stakeholders to understand requirements and ...

Role: Databricks Engineer Location: Hyderabad Exo Required: 5-10 years Tech Stack: Databricks, Data factory, Python, Pyspark Responsibilities: Design, build, and maintain scalable and reliable data pipelines for batch and real-time processing. Develop and manage data infrastructure using cloud platforms (e.g., Azure, AWS, GCP). Implement CI/CD pipelines for data workflows and analytics applications. Monitor and optimize data systems for performance, reliability, and cost-efficiency. Automate deployment, testing, and monitoring of data services and infrastructure. Collaborate with data scientists, analysts, and business teams to ensure data availability and quality. Manage data versioning, li...

Role & responsibilities Must Have: Extensive expertise in designing and implementing data load processes using Azure Data Factory, Azure Databricks, Delta Lake, Azure Delta Lake Storage and Python/PySpark Proficient with Databricks & Python Senior developers with Full Database/ Datawarehouse/ DataMart development capabilities Senior SME with SQL Server required. Proficiency with standard file formats and markup language technology such as JSON, XML, XSLT. Proficiency with Azure Cloud data environments highly desired. Excellent verbal communication skills. Good problem-solving skills. Attention to detail. Proficiency with Data Warehousing principals and design Nice to have: Familiarity with o...

Job description Hiring for Spark Developer Mandatory Skills: Spark, Python, SQL Responsibilities A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to actively aid the consulting team in different phases of the project including problem definition, effort estimation, diagnosis, solution generation and design and deployment You will explore the alternatives to the recommended solutions based on research that includes literature surveys, information available in public domains, vendor evaluation information, etc. and build POCs You will create requirement specifications from the business needs, define the to-be-processes and detailed functional...

Job description Hiring for Pyspark Develper Mandatory Skills: Pyspark, Python, Big data, ETL Responsibilities A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to actively aid the consulting team in different phases of the project including problem definition, effort estimation, diagnosis, solution generation and design and deployment You will explore the alternatives to the recommended solutions based on research that includes literature surveys, information available in public domains, vendor evaluation information, etc. and build POCs You will create requirement specifications from the business needs, define the to-be-processes and detail...

Responsibilities A day in the life of an InfoscionAs part of the Infosys delivery team, your primary role would be to provide best fit architectural solutions for one or more projects. You would also provide technology consultation and assist in defining scope and sizing of work You would implement solutions, create technology differentiation and leverage partner technologies. Additionally, you would participate in competency development with the objective of ensuring the best-fit and high quality technical solutions. You would be a key contributor in creating thought leadership within the area of technology specialization and in compliance with guidelines, policies and norms of Infosys.If y...

Job description Hiring for Pyspark Develper Mandatory Skills: Pyspark, Python, Big data, ETL Responsibilities A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to actively aid the consulting team in different phases of the project including problem definition, effort estimation, diagnosis, solution generation and design and deployment You will explore the alternatives to the recommended solutions based on research that includes literature surveys, information available in public domains, vendor evaluation information, etc. and build POCs You will create requirement specifications from the business needs, define the to-be-processes and detail...

Job description Hiring for Hadoop Developer Mandatory Skills: Hadoop, bigdata, spark Responsibilities A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to actively aid the consulting team in different phases of the project including problem definition, effort estimation, diagnosis, solution generation and design and deployment You will explore the alternatives to the recommended solutions based on research that includes literature surveys, information available in public domains, vendor evaluation information, etc. and build POCs You will create requirement specifications from the business needs, define the to-be-processes and detailed funct...

Job description Hiring for Azure Databrick Engineer Mandatory Skills: Databrick, Data lake, Azure Responsibilities A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to actively aid the consulting team in different phases of the project including problem definition, effort estimation, diagnosis, solution generation and design and deployment You will explore the alternatives to the recommended solutions based on research that includes literature surveys, information available in public domains, vendor evaluation information, etc. and build POCs You will create requirement specifications from the business needs, define the to-be-processes and d...

Job description Hiring for Pyspark Develper-Female Candidates Mandatory Skills: Pyspark, Python, Big data, ETL Responsibilities A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to actively aid the consulting team in different phases of the project including problem definition, effort estimation, diagnosis, solution generation and design and deployment You will explore the alternatives to the recommended solutions based on research that includes literature surveys, information available in public domains, vendor evaluation information, etc. and build POCs You will create requirement specifications from the business needs, define the to-be-pr...

Job description Hiring for Pyspark Develper-Female Diversity Mandatory Skills: Pyspark, Python, Big data, ETL Responsibilities A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to actively aid the consulting team in different phases of the project including problem definition, effort estimation, diagnosis, solution generation and design and deployment You will explore the alternatives to the recommended solutions based on research that includes literature surveys, information available in public domains, vendor evaluation information, etc. and build POCs You will create requirement specifications from the business needs, define the to-be-pro...

Job description Hiring for Spark Developer Mandatory Skills: Spark, Python, SQL Responsibilities A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to actively aid the consulting team in different phases of the project including problem definition, effort estimation, diagnosis, solution generation and design and deployment You will explore the alternatives to the recommended solutions based on research that includes literature surveys, information available in public domains, vendor evaluation information, etc. and build POCs You will create requirement specifications from the business needs, define the to-be-processes and detailed functional...