23086 Pyspark Jobs - Page 43

Role & responsibilities Developing Modern Data Warehouse solutions using Databricks and Azure Stack Ability to provide solutions that are forward-thinking in data engineering and analytics space Collaborate with DW/BI leads to understand new ETL pipeline development requirements. Triage issues to find gaps in existing pipelines and fix the issues Work with business to understand the need in reporting layer and develop data model to fulfill reporting needs Drive technical discussion with client architect and team members Orchestrate the data pipelines in scheduler via Airflow Skills and Qualifications: Bachelor's and/or masters degree in computer science or equivalent experience. Good underst...

Hiring for Data Enginee r with - Experience - 7+ Years Notice Period - Immediate Joiner- 20 days Mode of Work - Work from Office Technical Skills (Required) Azure Data Factory (ADF): design, develop and orchestrate scalable data pipelines Azure Databricks: build and manage Spark clusters, notebooks, and jobs Microsoft Fabric (OneLake): ingest, organize and transform data within the Fabric ecosystem PySpark & SparkSQL: author complex transformations and SQL queries for largescale data processing Delta Tables: create, optimize and maintain Delta Lake tables (VACUUM, ZORDER, OPTIMIZE) JSON Processing: parse, validate and transform semistructured JSON datasets API Integration: consume and build ...

Role: Application Architect - Data Governance Duration:3-6 Months Contract to Hire Location: Pune, Maharashtra 411014, India Technical Experience: 3 years of experience in owning and managing a technical platform including but not limited to: environment management, user access management, uptime management, platform enhancements, platform updates, SaaS platform performance, etc. 1 to 2 years of experience in managing a small team of contractors building and maintaining system to system integrations, metadata/data extractions, and data quality rules. Advanced knowledge in writing SQL and Python with a demonstrated ability to coach others. 2 years of experience managing large data quality rul...

As an ETL Developer at our company, you will be responsible for designing, developing, and implementing performant ETL pipelines using Python API (pySpark) of Apache Spark on AWS EMR. Your role will involve the following key responsibilities: - Programming with Python with a strong proficiency in Python - Utilizing functional programming concepts and Data Modelling - Hands-on experience in developing ETL data pipelines using pySpark on AWS EMR - Understanding Sparks Data frame and API - Configuring EMR clusters on AWS - Dealing with AWS S3 object storage from Spark - Troubleshooting Spark jobs and monitoring them using Spark UI Qualifications required for this role include: - 5+ years of exp...

Your opportunity to make a real impact and shape the future of financial services is waiting for you. Let's push the boundaries of what's possible together. As a Senior Director of Software Engineering at JPMorgan Chase within the Consumer and Community Banking division, you will be responsible for leading various technical domains, overseeing the activities of multiple departments, and fostering cross-functional collaboration. Your technical expertise will be utilized across different teams to promote the adoption and implementation of advanced technical methods, helping the firm stay ahead of industry trends, best practices, and technological advancements. - Leads multiple technology and p...

As an Azure Databricks Architect at UST, you will be responsible for designing and implementing cloud-native solutions on Microsoft Azure. Your role will involve leading and implementing Azure infrastructure projects, architecting cloud-native solutions using Azure, Databricks, PySpark, Python, and AKS, and collaborating with different teams to ensure alignment with business and technical requirements. Additionally, you will be migrating application services to the cloud, building Proof of Concepts (PoCs), providing architectural leadership, and ensuring security principles are integrated into solution designs. Your key responsibilities will include: - Leading and implementing Azure infrastr...

Embark on a transformative journey as an AI Incubator Data Scientist at Barclays, where you'll spearhead the evolution of our digital landscape, driving innovation and excellence. You'll harness cutting-edge technology to revolutionize our digital offerings, ensuring unapparelled customer experiences. Seeking a talented Data Scientist to join our Data Science team in particular our AI incubator where you'll spearhead the evolution of our digital landscape, driving innovation dedicated to driving commercial value from data across the Corporate Bank. In this role, you will have the opportunity to shape and lead projects with other members of the team - leveraging machine learning techniques an...

Role: Data Engineer- Scala Years of Experience: 5 to 10 yrs Location: Bangalore, Hyderabad, Chennai, Pune, Gurugram Date of Interview: 15th Nov, 2025 Mode of Interview: Virtual Role & responsibilities Must-Have Good work experience & Proven experience working on Python (Data Engineering), Scala. Good work experience on Big Data Platforms like Hadoop, Spark, Scala, Hive, Impala, SQL Experience working on Data Engineering projects Good Understanding of SQL Good understanding of Unix and HDFS commands Communication Experience working on Data Analytics and Pyspark Exposure to Tableau & SAS Required Technical Skill Set: Knowledge of Data Lakes environment, Scala, Data pipelines, SQL, AWS

About the role We’re looking for a Senior Python Data Engineer (4+ years) with deep hands-on experience in Azure Databricks, PySpark, Spark SQL, and Power BI. You’ll design, build, and optimize reliable data platforms and analytics solutions—partnering with product, analytics, and business teams in a fast-paced, collaborative environment. What you’ll do • Design and implement scalable ETL/ELT pipelines on Azure Databricks using PySpark/Spark SQL. • Model data for semantic layers powering BI and analytics use cases. • Build and maintain Power BI datasets, reports, and dashboards (DAX, data modeling, Power Query). • Write, refactor, and optimize Python notebooks for transformation, feature eng...

1. Design the end-to-end Test Strategy and Test plans for project deliveries and get signoffs from stakeholders 2. Able to drive enterprise-wide Automation Frameworks suited for Enterprise data platforms with data being fed by different business entities 3. Having good hands-on experience with Microsoft Azure along with Data Platform Testing. Should have azure Data Bricks, ADF, ADLS and Medallion Architectures based product testing experience 4. Should have hands-on knowledge of Python based testing frameworks i.e. PySpark/Py-Test and Python Data Quality libraries 5. Must have experience with Automaton framework design in Azure, CICD integration in pipelines and maintaining the overall Test ...

Total years of experience: Relevant years of experience: CTC (Including Variable): ECTC: Notice Period: Reason for change: Current location: Willingness to relocate (If other than Hyderabad ): Yes Availability for a teams call (two slots): Brief write up on how you could be a great fit for the role : Required Skills • 5- years of IT experience • Advanced skill in configuring and supporting software applications • Good knowledge in writing complex SQL queries • Experience with object-oriented language (python is a Plus) • Strong understanding of job orchestration tools like Jenkins and/or Airflow, code repository tools like Git and/or Bitbucket • Strong software development and debug/troubles...

We are seeking a Junior Data Engineer who is enthusiastic about working with data, quick to learn new technologies, and ready to take on challenges in a collaborative environment. If youre a team player with a positive attitude and a drive to grow, we encourage you to apply! Roles and Responsibilities: Assist in building and maintaining data ingestion, processing, and analytical pipelines using AWS services (ETL/ELT preferred) Support the design of data models for performance and reliability Work with big data, relational databases, NoSQL, and data warehouse solutions under guidance Collaborate with senior engineers on developing, monitoring, and maintaining data pipelines Handle moderate da...

Hello, Greetings from GlobalData!!! Urgent job openings for Team Lead - Python :- if job requirement is matching to your profile please share your updated resume @ mail id (m.salim@globaldata.com). Mention Subject Line :- Applying for Team Lead - Python @ GlobalData(Hyd) Share your details in the mail :- Full Name : Mobile # : Qualification : Company Name : Designation : Total Work Experience Years : Current CTC : Expected CTC : Notice Period : Current Location : If out-station willing to relocate to Hyd? : Office Address : 3rd Floor, Jyoti Pinnacle Building, Opp to Prestige IVY League Appt, Kondapur Road, Hyderabad, Telangana-500081. About the Role :- Roles and Responsibilities :- Design an...

- Design and develop scalable data engineering solutions. - Build and maintain ETL (Extract, Transform, Load) pipelines. - Implement data warehousing solutions. - Utilize Databricks, PySpark, and AWS services. Required Candidate profile - Contribute to the adoption of DevOps and Agile practices. Core Competencies: - SQL - Databricks - PySpark - Python - Pandas and NumPy AWS Cloud: S3, Lambda, Glue, CodeCommit, and Redshift.

Azure Data Engineer Skills:Azure Data Factory, Data Bricks, Data Warehousing, Data Modelling, SQL, ETL, Python, Pyspark Location-Pune/ Bangalore Exp-5+ Years Pkg-15 LPA MAX Send CV: riya.imaginators@gmail.com

Role: Data Engineering Remote Looking for 5-10 years of Exp • Design, develop & maintain ETL/ELT pipelines • Ingest & transform data from APIs, DBs, files, streams • Build real-time & batch processing solutions • Data validation, quality & cleansing • Translate business needs into data models • Ensure data security, access control & compliance • Maintain documentation & best practices Snowflake, AWS/GCP/Azure, Databricks (Lakehouse + Spark), BigQuery (Google Cloud), Azure Synapse / AWS Redshift, Kafka, Azure Data Factory, AWS Glue, Apache Spark, Apache Hadoop, Airflow, dbt, PySpark, SQL, Real-time & Batch Pipelines, Data Quality & Governance, Informatica, Talend, Collibra.

About Product : 5day.io is a modern work management tool that makes managing projects, tasks, timesheets, and collaboration easy. Our platform is designed for project management, timesheet management, collaboration, and task management to make your work more efficient and effective. What role looks like: We are looking for a high impact, hands on role for SDE III with a deep specialization in Generative AI and Agentic AI. You will be responsible for driving our most ambitious AI initiatives, responsible for the end-to-end lifecycle of our agentic systems and generative models. You'll work closely with cross functional teams including product management, Engineering, to translate complex busi...

Role: Azure Data Engineer Experience: 6–8 Years Mandatory Skills: Python: 6–8 years Databricks: 6–8 years Apache PySpark: 6–8 years Azure Cloud: 6–8 years Apache Airflow: 6–8 years SQL & Data Warehousing Concepts: 6–8 years

Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : Databricks Unified Data Analytics Platform Good to have skills : Microsoft Azure Databricks, Microsoft Azure Analytics Services, PySpark Minimum 5 Year(s) Of Experience Is Required Educational Qualification : 15 years full time education Summary: As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your ...

Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : Databricks Unified Data Analytics Platform Good to have skills : Microsoft Azure Databricks, Microsoft Azure Analytics Services, PySpark Minimum 5 Year(s) Of Experience Is Required Educational Qualification : 15 years full time education Summary: As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your ...

CACI International Inc is an American multinational professional services and information technology company headquartered in Northern Virginia. CACI provides expertise and technology to enterprise and mission customers in support of national security missions and government transformation for defense, intelligence, and civilian customers. CACI has approximately 23,000 employees worldwide. Headquartered in London, CACI Ltd is a wholly owned subsidiary of CACI International Inc., a publicly listed company on the NYSE with annual revenue in excess of US $6.2bn. Founded in 2022, CACI India is an exciting, growing and progressive business unit of CACI Ltd. CACI Ltd currently has over 2000 intell...

Job description Hiring for Hadoop Developer Mandatory Skills: Hadoop, bigdata, spark Responsibilities A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to actively aid the consulting team in different phases of the project including problem definition, effort estimation, diagnosis, solution generation and design and deployment You will explore the alternatives to the recommended solutions based on research that includes literature surveys, information available in public domains, vendor evaluation information, etc. and build POCs You will create requirement specifications from the business needs, define the to-be-processes and detailed funct...

We are seeking an experienced GCP Data Engineer with 5+ years of hands-on experience in designing, developing, and optimizing large-scale data pipelines using Google Cloud Platform (GCP) services The ideal candidate should be proficient in BigQuery and DataProc (PySpark), with a strong understanding of data modeling, query performance optimization, and cloud-based ETL architectures The role involves collaborating with cross-functional teams to translate business requirements into efficient data workflows, ensuring data quality, consistency, and scalability Proficiency in Python and SQL is essential, along with the ability to troubleshoot, debug, and optimize existing pipelines Experience wit...

Required Skills 5- years of IT experience Advanced skill in configuring and supporting software applications Good knowledge in writing complex SQL queries Experience with object-oriented language (python is a Plus) Strong understanding of job orchestration tools like Jenkins and/or Airflow, code repository tools like Git and/or Bitbucket Strong software development and debug/troubleshooting skills Advanced ability to research, analyze and recommend improvements to the organizations use of the software applications Advanced skill in creating and maintaining technical documentation Nice to Have Skills: Prior Production Support experience Working knowledge of Spark and AWS technologies like S3,...

Role – AIML Data Scientist Location: Indore (AIA Indore) Exp: 3 to 12 years Job Description Be a hands on problem solver with consultative approach, who can apply Machine Learning & Deep Learning algorithms to solve business challenges Use the knowledge of wide variety of AI/ML techniques and algorithms to find what combinations of these techniques can best solve the problem Improve Model accuracy to deliver greater business impact Estimate business impact due to deployment of model Work with the domain/customer teams to understand business context , data dictionaries and apply relevant Deep Learning solution for the given business challenge Working with tools and scripts for sufficiently pr...