23086 Pyspark Jobs - Page 34

About the Role: We are seeking an experienced Big Data Engineer to design, build, and maintain robust big data infrastructure and pipelines. The ideal candidate will handle large-scale data processing, ensure data quality, and enable analytics and data-driven decision-making across the organization. Key Responsibilities: 1. Big Data Pipeline Development Design, develop, and maintain end-to-end data pipelines for structured and unstructured data. Implement batch and real-time data processing solutions using Hadoop, Spark, Kafka , and related technologies. Ensure high performance, scalability, and reliability of data processing workflows. 2. Data Integration & Management Collect, clean, transf...

About the Role: We are looking for a talented Big Data Developer to design, develop, and maintain large-scale data processing systems. The ideal candidate will work with structured and unstructured data, build scalable data pipelines, and support analytics and business intelligence initiatives. Key Responsibilities: 1. Big Data Development Design, develop, and maintain big data processing solutions using frameworks such as Hadoop, Spark, Hive, or Kafka. Implement ETL pipelines for ingestion, transformation, and storage of large datasets. Optimize data pipelines for performance, scalability, and reliability. 2. Data Integration & Management Integrate data from multiple sources including relat...

About LeadSquared: LeadSquared is a fast-growing, global SaaS platform powering sales execution and marketing automation for high-velocity businesses. With customers across EdTech, Financial Services, Healthcare, and more, our platform processes billions of customer interactions and transactions each month. We are on a mission to make every sales process intelligent, automated, and outcome-driven and data & AI are central to that mission. Role Overview: We are looking for a strategic and hands-on Senior Director/AVP – Data & AI to lead the vision, design, and execution of LeadSquared’s data platform and AI roadmap. This leader will be responsible for building scalable infrastructure, driving...

Knowledge of Pyspark, Big Data experience, AWS EMR, S3, IAM, Lambda, SNS, SQS Required Candidate profile AWS, Terraform, Pyspark, Java Spark with Kafka streaming, Scala, Databricks

As a Senior Software Engineer, you will act as a technical guide, mentor, and coach, providing guidance and support to your team, while fostering a positive and productive work environment. You will be expected to be hands on and delivery focused while optimizing development schedule and time to market. You will also be responsible for setting project and squad goals, prioritizing and completing initiatives, and delivering results in a product-led, empowered environment. Your leadership skills, ability to handle multiple collaborators and experience in software development and software engineering will be important to your success in this role What You Will Deliver Design and document sophis...

Job Title: Data Engineer – Dynamics 365 Customer Insights & Power BI Experience: 5 - 7 years Time Zone: Starts from 7:00 AM IST (Australian shift) Role Summary As a Data Engineer specialising in Microsoft Dynamics 365 Customer Insights and Power BI, you will play the role as a developer and an analyst. Your primary responsibility will be to develop, manage, and optimize our Microsoft Customer Insights platform, ensuring it effectively captures and analyzes customer data. Additionally, you will utilize Power BI to create insightful, data-driven reports and semantic models that inform our business strategies and enhance customer engagement. This role requires a unique blend of technical expert...

Experienced in MS Azure services, including developing, deploying, and configuring Azure Web Apps/App Services, with cloud migration and integration exposure. Quick learner with feedback and proven experience supporting client project environments. Required Candidate profile Core: Azure Modern Data Platform, SQL Server (T-SQL) Power BI, Synapse, Azure Data Bricks Supporting: Azure ML, Azure infrastructure, Python, PowerShell, .Net, Data Factory, Azure Analytics Services

AI Engineer Gen AI + Python Python + Generative AI • Strong python programming skills. • Experience in workflow orchestration using Apache Airflow. Required Candidate profile Preferable experience in working with hadoop ecosystem - hdfs, hive. • Experience in working with a job scheduler - preferably Control-M.

Data Engineer (Microsoft Fabric / Azure) Location: Coimbatore Notice Period: Immediate joiners preferred We are scaling a strategic Digital Operations capability and seeking a pragmatic, delivery-focused Data Engineer to design, build and operate end-to-end data platforms that drive product-performance analytics. Key responsibilities Engineer robust ETL/ELT pipelines and data models to ingest, transform and surface large volumes of structured & unstructured product and customer data. Own data lake migrations and One-Lake availability; operationalize data for cross-functional stakeholders. Implement CI/CD for data deployments and maintain pipeline observability and data quality. Deliver dashb...

Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : PySpark Good to have skills : Python (Programming Language), Apache Spark, Google BigQuery Minimum 5 Year(s) Of Experience Is Required Educational Qualification : 15 years full time education Summary: Seeking a forward-thinking professional with an AI-first mindset to design, develop, and deploy enterprise-grade solutions using Generative and Agentic AI frameworks that drive innovatio...

Project Role : Application Developer Project Role Description : Design, build and configure applications to meet business process and application requirements. Must have skills : Apache Spark Good to have skills : Python (Programming Language), Google BigQuery, PySpark Minimum 5 Year(s) Of Experience Is Required Educational Qualification : 15 years full time education Summary: Seeking a forward-thinking professional with an AI-first mindset to design, develop, and deploy enterprise-grade solutions using Generative and Agentic AI frameworks that drive innovation, efficiency, and business transformation. As an Application Developer, you will design, build, and configure applications to meet bu...

Role & responsibilities Mandatory Skills Python | PySpark | SQL | Traditional ML | MLOps Job Summary We are looking for an AI/ML Engineer who can design, build, and maintain scalable data pipelines and MLOps systems. This role involves performing experiments, statistical analysis, EDA, feature engineering, and productionizing ML models. You will be responsible for optimizing training and inference, enabling the organization to build, deploy, and scale AI/ML-driven solutions. Roles & Responsibilities Build and maintain data ingestion, transformation, and feature engineering pipelines Conduct experiments, statistical analysis & EDA to generate insights for model improvement Ensure pipelines ar...

Job Description Role: Data Analyst/ Engineer Job Summary Senior Level Data Engineer / Data Analyst technical lead with data analytics experience, Databricks, Pyspark and Python This is a key role that requires senior/lead with great communication skills who is very proactive with risk & issue management. Experience And Education Required 10+ years of experience as Data Analyst / Data Engineer with Databricks on AWS expertise in designing and implementing scalable, secure, and cost-efficient data solutions on AWS Job Profile Hands-on data analytics experience with Databricks on AWS, Pyspark and Python Must have prior experience with migrating a data asset to the cloud using a GenAI automation...

Job Description ETL Automation & SQL Expert Location: Bangalore / Chennai Experience: 4+ Years Joiners: Immediate / 15 Days preferred About the Role We are looking for an experienced ETL Automation Engineer with strong SQL expertise to support data quality, ETL pipeline validation, automation, and cloud-based data workflows. The ideal candidate should have hands-on experience in Python/PySpark, AWS services, and test automation for data pipelines. Key Responsibilities Perform ETL testing , data validation, and data quality checks across multiple data pipelines. Design and implement automation frameworks for ETL and data validation processes. Write complex and optimized SQL queries for data a...

Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : PySpark Good to have skills : NA Minimum 3 Year(s) Of Experience Is Required Educational Qualification : 15 years full time education Summary: Seeking a forward-thinking professional with an AI-first mindset to design, develop, and deploy enterprise-grade solutions using Generative and Agentic AI frameworks that drive innovation, efficiency, and business transformation. As a Data Engi...

Role & responsibilities Develop, optimize, and maintain data pipelines and ETL processes. Work with large-scale structured and unstructured datasets using PySpark/Spark. Design and implement data models, manage schema changes, and ensure data quality. Integrate data from various sources into centralized repositories or data lakes. Implement and manage workflow orchestration using Apache Airflow. Work closely with cross-functional teams to understand business data requirements. Collaborate on data architecture, cloud infrastructure, and scalability improvements. Monitor and troubleshoot data processes; ensure data reliability and integrity. Use Kafka for real-time data ingestion and streaming...

We are looking for an experienced Senior Data Engineer who can design, build, and optimize modern data pipelines and data platforms. The ideal candidate has strong experience in cloud (GCP/AWS/Azure), ETL/ELT, distributed systems, and hands-on coding in Python/Scala/SQL. Key Responsibilities Data Engineering & Pipelines Design, develop, and maintain scalable data pipelines using cloud-native services (GCP/AWS/Azure). Build ETL/ELT workflows for structured and unstructured data. Implement batch and real-time data ingestion using tools like Kafka / PubSub / Kinesis. Automate data workflows using Airflow / Composer / Azure Data Factory . Data Modelling & Warehousing Design data models (dimensio...

About the Role: We are seeking a Senior Data Engineer to drive the development of our next-generation Data Lakehouse architecture. You will be responsible for designing, building, and optimizing massive-scale, low-latency data pipelines that support real-time analytics and Machine Learning applications. Key Responsibilities: - Design and build highly optimized, production-grade ETL/ELT pipelines using Apache Spark (PySpark/Scala) to process petabytes of data. - Architect and manage the Data Lakehouse using open-source technologies like Delta Lake or Apache Hudi for ACID transactions and data quality. - Integrate and process real-time data streams using technologies such as Apache Kafka or Ki...

We are a technology consulting firm operating in Cloud Data Engineering and Analytics, helping enterprise customers build reliable, scalable data platforms and analytics products. Our teams deliver end-to-end data lakes, real-time streaming pipelines, and production-grade ML feature stores using Databricks and modern cloud data tooling. Role & Responsibilities : - Design, build, and maintain scalable batch and streaming ETL pipelines on Databricks using Delta Lake and Delta Live Tables (DLT). - Develop and optimize Spark/PySpark jobs for performance, cost-efficiency, and reliability; tune cluster sizing and autoscaling policies. - Implement data quality, observability, lineage and monitoring...

Project Role : Custom Software Engineer Project Role Description : Develop custom software solutions to design, code, and enhance components across systems or applications. Use modern frameworks and agile practices to deliver scalable, high-performing solutions tailored to specific business needs. Must have skills : Python (Programming Language) Good to have skills : Apache Spark, Google BigQuery, PySpark Minimum 5 Year(s) Of Experience Is Required Educational Qualification : 15 years full time education Summary: Seeking a forward-thinking professional with an AI-first mindset to design, develop, and deploy enterprise-grade solutions using Generative and Agentic AI frameworks that drive inno...

We are seeking an experienced GCP Data Engineer with 5+ years of hands-on experience in designing, developing, and optimizing large-scale data pipelines using Google Cloud Platform (GCP) services The ideal candidate should be proficient in BigQuery and DataProc (PySpark), with a strong understanding of data modeling, query performance optimization, and cloud-based ETL architectures The role involves collaborating with cross-functional teams to translate business requirements into efficient data workflows, ensuring data quality, consistency, and scalability Proficiency in Python and SQL is essential, along with the ability to troubleshoot, debug, and optimize existing pipelines Experience wit...

We are seeking a skilled Azure Data Engineer with hands-on experience in modern data engineering tools and platforms within the Azure ecosystem . The ideal candidate will have a strong foundation in data integration, transformation, and migration , along with a passion for working on complex data migration projects . Job Title: Azure Data Engineer Location: Remote work Work Timings: 5:00 PM to 2:00 AM IST No of Openings: 2 Please Note: This is a pure Azure-specific role . If your expertise is primarily in AWS or GCP , we kindly request that you do not apply . Roles and responsibilities: Lead the migration of large-scale SQL workloads from on-premise environments to Azure, ensuring high data ...

Role & responsibilities Data Engineer, sql, Python. Databbricks, spark, Pyspark

JOB DETAILS: Job Title: Databricks Developer Locations: Hyderabad Client: MNC Client Experience Range: 3 to 8 years Required Skills: ETL/ELT pipelines using Databricks and Apache Spark, PySpark , Advanced SQL Skillset, Automation & DevOps Integration Job Description: Computer Science University degree or equivalent work experience At least 1.5-2.5 years for Aanalyst and 4-5 years of total exp Experience in Databricks, Pyspark and ADF etc. Experience in Databricks end-to-end implementation. Expert in developing solutions and delivering the high-quality projects using Pyspark and Databricks Proficiency with Azure Data Factory, Azure DevOps, and cloud-native data services. Experience integratin...

Role Overview: We are looking for an experienced Azure Data Architect to design and implement enterprise-grade data solutions on Microsoft Azure. The ideal candidate will have strong expertise in Azure data services , PySpark , and Power BI , with a proven ability to architect scalable, secure, and high-performing data platforms. This role requires hands-on experience in big data processing, advanced analytics, and data visualization, along with leadership in defining best practices for data governance and performance optimization. Key Responsibilities: Architect and implement Azure-based data solutions leveraging services such as Azure Synapse, ADLS Gen2, Azure Data Factory (ADF), Purview ,...