22907 Pyspark Jobs

About Position: We are seeking for Data engineer with AWS, databricks experience. Role: Data Engineer Location: All Persistent Location Experience: 5 to 12 Years Job Type: Full Time Employment What You'll Do: Must have knowledge and working experience in Databricks, PySpark and in AWS, Apache Airflow, ETL, SQL Life science domain experience (Pharma), Redshift, BI Tools (Tableau/Power BI) Designing, creating, testing and maintaining the complete data management & processing systems. Working closely with the stakeholders & solution architect. Ensuring architecture meets the business requirements. Building highly scalable, robust & fault-tolerant systems. Discovering data acquisitions opportuni...

Job Purpose and Impact The Professional, Data Engineering job designs, builds and maintains moderately complex data systems that enable data analysis and reporting. With limited supervision, this job collaborates to ensure that large sets of data are efficiently processed and made accessible for decision making. Key Accountabilities DATA & ANALYTICAL SOLUTIONS: Develops moderately complex data products and solutions using advanced data engineering and cloud based technologies, ensuring they are designed and built to be scalable, sustainable and robust. DATA PIPELINES: Maintains and supports the development of streaming and batch data pipelines that facilitate the seamless ingestion of data f...

About the Role We are seeking an experienced Data Engineer Lead to design, develop, and maintain scalable data solutions on Azure and Databricks as part of our enterprise data modernization initiatives. The ideal candidate will have a strong background in data pipeline development, data integration frameworks , and cloud-based data engineering , with deep expertise in tools such as Databricks, Azure Data Factory, Alteryx, Ab Initio, Talend, and Informatica . This role will lead the design and delivery of high-performance, governed data architectures for large-scale enterprise clients, driving data reliability, compliance, and analytics readiness. Key Responsibilities Design and implement sca...

Key Responsibilities Design and maintain secure, high-performance ETL and data processing pipelines. Develop and manage telemetry data workflows using Kafka or MQTT . Build and optimize databases ( PostgreSQL , MySQL , Oracle , InfluxDB , TimescaleDB ). Automate data workflows using Apache Airflow for scheduling and orchestration. Implement secure data handling with encryption, signing, and validation. Integrate MLFlow for model tracking and ML lifecycle management. Create dashboards and visualizations using Splunk and Grafana . Deploy and manage data solutions in AWS , Azure , or GCP . Collaborate with data science and DevOps teams to ensure seamless data-to-ML integration. Stay current wit...

We are seeking an experienced and highly skilled Senior Data Engineer to join our organization. The successful candidate will demonstrate deep technical expertise in Snowflake, strong analytical and problem-solving abilities, and the capability to manage client interactions with professionalism and clarity. This role requires a proactive individual who can deliver robust data solutions, collaborate effectively across teams, and adapt to evolving technologies. Design, develop, and maintain efficient, scalable data pipelines and solutions using Snowflake. Collaborate with clients and internal stakeholders to understand business requirements and translate them into effective technical solutions...

Job Title: Python Developer Experience Required: 0-1 Years Location: [Hyderabad] Employment Type: Full-Time About the Role We are looking for a skilled Python Developer to join our development team. The ideal candidate should have strong experience in building python application and looking for a person who can work and complete the tasks independently. Key Responsibilities Develop efficient Python scripts to automate repetitive tasks, data pipelines, and system processes. Write modular, reusable, and optimized code following best practices. Build command-line tools, schedulable scripts (cron, Task Scheduler), and automation workflows. Required Skills Strong proficiency in Python . Knowledge...

Warm Greetings from SP Staffing!! Role: Azure Data Engineer Experience Required :6 to 11 yrs Work Location :PAN India Required Skills, Azure Databricks Azure Delta lake Interested candidates can send resumes to nandhini.spstaffing@gmail.com

Mandatory Skills Pyspark / Python scripting SQL (complex query writing & optimization) AWS Cloud experience AWS services (Glue, S3, CloudWatch, CLI, etc.) Good-to-Have Skills Basic understanding of data modelling Knowledge of ETL/ELT concepts Familiarity with Data Warehousing and Data Lake concepts Shell scripting Experience working with APIs Daily Responasibilities Develop modular and reusable Python/Pyspark scripts for integrating data from APIs, databases, SFTP, and S3 into targets such as S3 and Databricks Work with multiple file formats: Excel, CSV, JSON, XML, Parquet Write and optimize complex SQL queries for performance and scalability Operate within AWS cloud environments to manage d...

Project description Support one of the top Australian banks as they seek to modernise their data and analytics platform. You will be working directly with IT and business stakeholders in Data and Platform team to implement banks data strategy to become the best AI bank of the world. Responsibilities We are looking for a Software Engineer with strong experience in designing, building, and maintaining production-grade software systems. The role focuses on automation, quality, and resilience across the software development lifecycle, with end-to-end ownership of features from design through deployment and monitoring. Key Responsibilities Design and implement robust, scalable software solutions....

Project description We are seeking a Palantir Foundry & AIP Engineer with hands-on experience across the full Foundry ecosystem and Palantir's Artificial Intelligence Platform (AIP). This role goes beyond data engineeringyou will design, build, and operationalize AI-powered workflows, agents, and applications that drive tangible business outcomes. The ideal candidate is a self-starter, able to translate complex business needs into scalable technical solutions, and confident working directly with stakeholders to maximize the value of Foundry and AIP. Responsibilities Data & Workflow EngineeringDesign, develop, and maintain scalable pipelines, transformations, and applications within Palantir ...

Metyis is a Netherlands based fast-growing international company with a diverse team of over 700+ professionals with a mix of Analytics, Data Science and Technology Team. We are looking for an experienced Data Engineer to join our team with below requirements: Responsibilities: Independently lead & manage execution of data engineering projects Engineer complete technical solutions to solve concrete business challenges in the areas of digital marketing, eCommerce, Business Intelligence and self-service analytics Collect functional and non-functional requirements, consider technical environments, business constraints and enterprise organizations Support our clients in executing their Big Data ...

Description: We are seeking an technical leader for a hybrid Senior Programmer and Data engineer & Solutions Architect role. This position is for a hands-on expert who not only possesses mastery in Apache Spark development but also has a proven track record of architecting complex, end-to-end data solutions on AWS and on-premise. You will be the technical leader for new data initiatives, responsible for designing the architectural blueprints for our data lake platforms and then leading the hands-on implementation of those designs. We need a strategic thinker who can solve our most complex technical challenges and proactively drive things to add values to the domain.. Requirements: Core Techn...

Job Summary: Role Overview We are seeking a hands-on Data Quality Analyst to strengthen the integrity and reliability of enterprise data assets. The ideal candidate will blend technical skills in Databricks, PySpark, and Python with practical experience using Ataccama for rule-based data quality management. This role sits at the intersection of data engineering, governance, and analyticsresponsible for implementing automated quality checks, root cause analysis, and continuous data improvement. Job Description: Key Responsibilities 1. Data Quality Development & Monitoring Design and implement automated data quality rules and validation checks using Databricks (Delta Lake) and PySpark . Build ...

Role: Databricks Technical Lead/ Data Architect Experience: 8-13 years Location: Bangalore, Chennai, Delhi, Pune, Kolkata About Tredence: Tredence focuses on last-mile delivery of powerful insights into profitable actions by uniting its strengths in business analytics, data science and software engineering. The largest companies across industries are engaging with us and deploying their prediction and optimization solutions at scale. Head quartered in the San Francisco Bay Area, we serve clients in the US, Canada, Europe, and South East Asia. We are seeking an experienced data scientist who apart from the required mathematical and statistical expertise also possesses the natural curiosity an...

Role & responsibilities Job Location: Bangalore Mode of work: Hybrid Exp: 5-8 Years Looking for Immediate joiners only DQ Tools: Ataccama DQMS, Informatica DQ, Collibra (trainable skill) Data Platforms: Databricks, Azure Synapse, Snowflake Data Engineering: SQL, PySpark, Python Integration & Automation: Azure Data Factory, Airflow, APIs Visualization & Monitoring: Power BI, ServiceNow (for issue tracking) Governance & Quality Management: Metadata management, data lineage tracking, KPI reporting Preferred candidate profile

Role : Gen AI Engineer Experience : 4 to 13 Years Relevant Experience : 2+ Years Location : Pan India Notice Period : Immediate to 90 days Mode of Interview - In-Person (29th November 2025) at Chennai Job Description: 1. Be a hands on problem solver with consultative approach, who can apply Machine Learning & Deep Learning algorithms to solve business challenges Use the knowledge of wide variety of AI/ML techniques and algorithms to find what combinations of these techniques can best solve the problem Improve Model accuracy to deliver greater business impact Estimate business impact due to deployment of model 2. Work with the domain/customer teams to understand business context , data dictio...

Job Title: GCP Data Engineer Location: Bangalore, Chennai, Pune, Hyderabad and Vadodara Experience: 7 Years - 10 Years Notice Period: Immediate Joiners ( Preferred) Key Responsibilities: Design, develop, and maintain data ingestion and transformation pipelines on GCP using BigQuery, PySpark, and Airflow DAGs. Implement data processing and analytics workflows aligned with performance, scalability, and reliability standards. Collaborate with cross-functional teams (data architects, analysts, and application developers) to deliver integrated solutions. Optimize BigQuery SQL scripts for efficiency and cost management. Develop reusable data processing components and automation frameworks. (Option...

Description: Senior Data Engineer (Spark & Lakehouse) Location: Remote, India (Preferred: Bangalore/Pune) Experience: 6+ Years Domain: Data Engineering / Big Data About the Role: We are seeking a Senior Data Engineer to drive the development of our next-generation Data Lakehouse architecture. You will be responsible for designing, building, and optimizing massive-scale, low-latency data pipelines that support real-time analytics and Machine Learning applications. Key Responsibilities: - Design and build highly optimized, production-grade ETL/ELT pipelines using Apache Spark (PySpark/Scala) to process petabytes of data. - Architect and manage the Data Lakehouse using open-source technologies ...

Description : We are a technology consulting firm operating in Cloud Data Engineering and Analytics, helping enterprise customers build reliable, scalable data platforms and analytics products. Our teams deliver end-to-end data lakes, real-time streaming pipelines, and production-grade ML feature stores using Databricks and modern cloud data tooling. Role & Responsibilities : - Design, build, and maintain scalable batch and streaming ETL pipelines on Databricks using Delta Lake and Delta Live Tables (DLT). - Develop and optimize Spark/PySpark jobs for performance, cost-efficiency, and reliability; tune cluster sizing and autoscaling policies. - Implement data quality, observability, lineage ...

Role : Data Scientist AIML Experience : 4 to 13 Years Relevant Experience : 2+ Years Location : Pan India Notice Period : Immediate to 60 days Mode of Interview - In-Person (29th November 2025) Job Description: 1. Be a hands on problem solver with consultative approach, who can apply Machine Learning & Deep Learning algorithms to solve business challenges Use the knowledge of wide variety of AI/ML techniques and algorithms to find what combinations of these techniques can best solve the problem Improve Model accuracy to deliver greater business impact Estimate business impact due to deployment of model 2. Work with the domain/customer teams to understand business context , data dictionaries ...

- Mastery of Azure data services such as Azure SQL Database, Azure Cosmos DB, and Azure Data Lake Storage. - Deep expertise in ETL (Extract, Transform, Load) processes and data integration. - Advanced SQL and database querying optimization skills. - Experience in leading and managing data engineering projects.

5+ years data warehouse testing experience, 2+ years of Azure Cloud experience. Strong understanding of data marts and data warehouse concepts Expert in SQL with the ability to create source-to-target comparison test cases in SQL Creation of test plans, testcases, traceability matrix, closure reports Proficient with Pyspark, Python, Git, Jira, JTMF Agile: Kanban and Scrum Azure Data Bricks, Data Factory, DevOps Mainframe EBCDIC and COBOL copy books Automation tools(Tosca BI/DI, Autosys) ETL, Excel Macros, Informatica Microsoft certified: AZ-900 Azure Fundamentals, DP-900 azure data fundamentals, DP-203 azure data engineer associate Mandatory Skills : ETL Testing . Experience : 5-8 Years .

As an Application Developer, you will lead IBM into the future by translating system requirements into the design and development of customized systems in an agile environment. The success of IBM is in your hands as you transform vital business needs into code and drive innovation. Your work will power IBM and its clients globally, collaborating and integrating code into enterprise systems. You will have access to the latest education, tools and technology, and a limitless career path with the world’s technology leader. Come to IBM and make a global impact Responsibilities: Responsible to manage end to end feature development and resolve challenges faced in implementing the same Learn new te...

8+ years of experience in data engineering, with a focus on scalable architecture design. Strong proficiency in Python, PySpark, Python-based UI libraries and Gradio or Streamlit. Strong proficiency in Python, especially in Pandas and NumPy. Strong proficiency in Databricks environment and Native Apps Hands-on experience with Databricks (notebooks, jobs, lakehouse architecture). Experience building charts and dashboards using Matplotlib, Seaborn, or Plotly. Experience in python webapp and rest API development Good to have understanding of SAS (PROC steps, data handling, stored processes). Understanding of data structures, statistical methods, and basic analytical modeling. Good understanding...

Job Description: Develop, optimize, and maintain end-to-end ETL/ELT pipelines using Databricks , PySpark , and Delta Lake for high-performance data processing. Design and implement scalable data lakehouse architectures , including bronzesilver–gold layers following the Medallion Architecture . Perform advanced data transformations , cleansing, and enrichment to ensure data quality, consistency, and business readiness. Work closely with stakeholders to gather requirements and translate them into efficient Databricks workflows and data models. Implement and manage job orchestration , monitoring, and debugging using Databricks Jobs, Workflows, or Airflow. Optimize PySpark code, cluster configur...