23086 Pyspark Jobs - Page 21

Experience At least 5 years of experience in AWS based projects. Technical skills Proficiency in Python and PySpark for data engineering tasks. Big Data Strong knowledge of Big Data technologies and data warehousing concepts. AWS services Experience with AWS Data Engineering stack, including S3, RDS, Athena, Glue, Lambda, and Step Functions. SQL Strong SQL skills for data manipulation and querying. CI CD Experience with CI CD tools like Terraform and Git Actions. Soft skills Good communication skills and ability to work in a multicultural team. About Virtusa Teamwork, quality of life, professional and personal development: values that Virtusa is proud to embody. When you join us, you join a ...

Main purpose: Act as the Data Architect to ensure the delivery of all commitments on time and in high quality. Manage the data model and schema Work closely with product and engineering teams to provide evaluation and recommendations of design patterns and solutions for data platforms with a focus on Graph based CDP, ETL and Lambda architectures Provide technical inputs to agile processes, such as epic, story, and task definition to resolve issues and remove barriers throughout the lifecycle of client engagements Work with the Marcels data, product and engineering teams to deliver on all requests pertaining to Data, Features, Graph Schema, Feeds and Integrations, Data Science Models and othe...

About the Company: Everest DX – We are a Digital Platform Services company, headquartered in Stamford. Our Platform/Solution includes Orchestration, Intelligent operations with BOTs’, AI-powered analytics for Enterprise IT. Our vision is to enable Digital Transformation for enterprises to deliver seamless customer experience, business efficiency and actionable insights through an integrated set of futuristic digital technologies. Digital Transformation Services - Specialized in Design, Build, Develop, Integrate, and Manage cloud solutions and modernize Data centers, build a Cloud-native application and migrate existing applications into secure, multi-cloud environments to support digital tra...

About Position: We are seeking a skilled and detail-oriented Quality Analyst (QA) to support the migration of ETL workflows from Informatica to PySpark. The ideal candidate will have hands-on experience in validating data pipelines, ensuring data integrity, and working closely with development and business teams to ensure a smooth transition. Role: QA Engineer Location: All Persistent Locations Experience: 10+ Years Job Type: Full Time Employment What You'll Do: Understand existing Informatica workflows and validate their PySpark equivalents. Develop and execute test plans, test cases, and test scripts for ETL processes. Perform data validation, reconciliation, and regression testing across ...

Job ID: Pyt-ETP-Pun-1125 Location: Pune Company Overview Bridgenext is a Global consulting company that provides technology-empowered business solutions for world-class organizations. Our Global Workforce of over 800 consultants provide best in class services to our clients to realize their digital transformation journey. Our clients span the emerging, mid-market and enterprise space. With multiple offices worldwide, we are uniquely positioned to deliver digital solutions to our clients leveraging Microsoft, Java and Open Source with a focus on Mobility, Cloud, Data Engineering and Intelligent Automation. Emtec’s singular mission is to create “Clients for Life” – long-term relationships that...

About Position: We are seeking a highly skilled and experienced Data Engineer with expert-level proficiency in Azure Databricks, Apache Spark, and Azure Data Services. The ideal candidate will have a strong background in building production-grade Delta Live Tables (DLT) pipelines, implementing Medallion Architecture, and managing infrastructure using Terraform. This role demands hands-on experience in real-time data ingestion, structured streaming, and CI/CD practices. Familiarity with the Healthcare domain and Snowflake is a plus. Role: Senior Data Engineer Location: All Persistent Locations Experience: 6+ Years Job Type: Full Time Employment What You'll Do: Design, develop, and maintain sc...

Job description Hiring for Azure Databrick Engineer Mandatory Skills: Databrick, Data lake, Azure Responsibilities A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to actively aid the consulting team in different phases of the project including problem definition, effort estimation, diagnosis, solution generation and design and deployment You will explore the alternatives to the recommended solutions based on research that includes literature surveys, information available in public domains, vendor evaluation information, etc. and build POCs You will create requirement specifications from the business needs, define the to-be-processes and d...

Role & responsibilities 5+ years of experience in data engineering roles. • Proficiency in programming languages such as Scala or Python, or Java. • Expertise in distributed data processing frameworks like Apache Spark or Flink. • Strong knowledge of cloud platforms (e.g., AWS) and cloud-native data platforms like Databricks or Redshift. • Solid understanding of SQL, schema design, dimensional modeling, and ETL best practices. • Experience with workflow orchestration tools such as Apache Airflow. • Familiarity with CI/CD pipelines, preferably GitHub Actions. • Strong technical documentation and issue-tracking skills. • Demonstrated ability to adopt and integrate new technologies.

Description - External At American Express, our culture is built on a 175-year history of innovation, shared values and Leadership Behaviors, and an unwavering commitment to back our customers, communities, and colleagues. As part of Team Amex, you'll experience this powerful backing with comprehensive support for your holistic well-being and many opportunities to learn new skills, develop as a leader, and grow your career. Here, your voice and ideas matter, your work makes an impact, and together, you will help us define the future of American Express. How will you make an impact in this role? This Position is in GSG Advanced Analytics team as part of GSG MIS COE, is looking for full time c...

Seeking a Senior Data Engineer skilled in ETL, Palantir Foundry, Python, PySpark, and AWS. Build scalable data solutions, ensure quality, and collaborate in Agile teams. 5+ years’ experience required; utility domain is a plus. Required Candidate profile Senior Data Engineer expert in cloud, ETL, and big data tools. Strong in Python, PySpark, and AWS with proven ability to deliver secure, scalable data solutions.

Data Engineer Databricks Location : Hyderabad ( Hybrid- 3 days) Experience : 7-10yrs The Impact : As a member of the Content Externalization Team, you will be responsible for analysis, design, architecture, development, and support of ETL pipelines using Spark framework in Databricks platform. The ideal candidate should have expertise with cutting edge technologies and a desire to drive change through all alignment across the enterprise. The role requires the candidate to be a hands-on problem solver and developer helping to extend and manage the ETL pipelines. Whats in it for you: Exposure to work on Latest cutting-edge Technologies. Opportunity to grow personally and professionally. Exposu...

Position Summary We are looking for a highly skilled and experienced Data Engineering Manager to lead our data engineering team. The ideal candidate will possess a strong technical background, strong project management abilities, and excellent client handling/stakeholder management skills. This role requires a strategic thinker who can drive the design, development and implementation of data solutions that meet our clients’ needs while ensuring the highest standards of quality and efficiency. Job Responsibilities Technology Leadership – Lead guide the team independently or with little support to design, implement deliver complex cloud-based data engineering / data warehousing project assignm...

Responsibilities Design, develop, and deploy data processing solutions using Azure Databricks. Collaborate with data engineers, architects, and business analysts to gather and analyze requirements. Develop, optimize, and manage complex data pipelines and workflows. Ensure data quality, integrity, and security across various data sources. Troubleshoot and resolve performance issues related to Databricks solutions. Document technical specifications and maintain comprehensive project documentation. Provide technical guidance and mentorship to junior developers. Qualifications Bachelor's degree in Computer Science, Information Technology, or related field. Minimum of 5 years of experience in dat...

About Position: Data Engineer with extensive experience in leading large-scale data migration projects, specializing in PySpark and Databricks. Proven ability to design and optimize high-performance data pipelines, refactor legacy ETL processes, and ensure data integrity across modern cloud platforms. Adept at collaborating with architects and guiding engineering teams to align solutions with enterprise data strategies. Role: Senior Data Engineer Location: Pune Experience: 6 to 9 Years Job Type: Full Time Employment What You'll Do: Design and optimise PySpark pipelines for large-scale data migration. Refactor stored procedures and ETL logic from Oracle to Databricks SQL. Implement validation...

🌟 We're Hiring: PySpark Data Engineer! 🌟 We are seeking a highly skilled Senior PySpark Data Engineer to design, build, and optimize large-scale data pipelines and distributed systems. Beyond deep expertise in Apache Spark (PySpark) and automation, this role requires the ability to manage stakeholders, ensure timely delivery, and assess requirements. You will play a critical role in bridging business needs with technical execution, ensuring high-quality, scalable, and reliable data solutions. 📍 Location: Hyderabad, India ⏰ Work Mode: Flexible office & remote 💼 Role: PySpark Data Engineer What You'll Do Technical Delivery Data Pipeline Development: Design, develop, and optimize scalable ETL/E...

Role & responsibilities Data Pipeline Development Design, build, and maintain scalable ETL/ELT pipelines for batch and streaming data. Develop real-time data ingestion frameworks using Apache Kafka (topics, partitions, consumer groups). Build large-scale distributed processing jobs using Spark / PySpark . Big Data Engineering Work with Hadoop, HDFS, Hive for storing and processing big datasets. Create and optimize Hive tables (managed/external, ORC/Parquet). Implement performance tuning in Hive (partitioning, bucketing). Data Transformation & Processing Develop complex data transformation logics using PySpark & Python . Optimize Spark jobs (shuffle, caching, broadcast joins, partitioning). E...

3+ years experienced engineer who has worked on GCP environment and its relevant tools/services. (Big Query, Data Proc, Data flow, Cloud Storage, Terraform, Tekton , Cloudrun , Cloud scheduler, Astronomer/Airflow, Pub/sub, Kafka, Cloud Spanner streaming etc) 1 or 2 + years of strong experience in Python development (Object oriented/Functional Programming, Pandas, Pyspark etc) 1 or 2 + years of strong experience in SQL Language (CTEs, Window functions, Aggregate functions etc). Keywords : - airflow,kafka,pyspark,pandas,cloud storage,python development,terraform,cloud spanner,python,java,spark,hadoop,aws,hive,big data,mapreduce,javascript,mysql,postgresql,oracle,sql server,oops,gcp,bigquery,da...

About Position: As a Data Engineer, you will design and manage robust data pipelines using Azure Databricks, Spark, and Kafka. You'll process large-scale datasets and implement CDC pipelines to power analytics platforms. Your role includes modeling data using the Medallion Architecture and ensuring data quality and governance. Collaboration with analytics teams for visualization and insights is key to success. Role: Data Engineer Location: Mumbai Experience: 5 to 11 Years Job Type: Full Time Employment What You'll Do: Design, build, and maintain scalable and efficient data pipelines using Azure Data Factory and Azure Databricks. Implement Change Data Capture (CDC) pipelines to ingest data fr...

Required Skills & Experience: Experience: 6-11 years of hands-on experience in Data Quality Assurance, Data Testing, or ETL Testing roles. SQL Expertise: Advanced proficiency in SQL: Ability to write complex queries, subqueries, analytical functions (Window functions), CTEs, and stored procedures for data validation, reconciliation, and analysis. Experience with various SQL databases (e.g., SQL Server, Oracle, PostgreSQL, MySQL, Snowflake, BigQuery). Strong understanding of database concepts: normalization, indexing, primary/foreign keys, and data types. Data Testing Methodologies: Solid understanding of data warehousing concepts, ETL processes, and various data testing strategies (e.g., sou...

Position - Lead Sr Software Engineer - Python & React Exp -7-10 yrs Work Mode- Remote Responsibilities & Skills:- ● Have 7+ years of industry experience delivering Enterprise products with 3 years in a senior engineer position. ● Have experience in building or scaling platforms, APIs, or enterprise-grade SaaS products using Python, React. ● Have experience in leading high-performing engineering teams of 5+ members ensuring high-quality, timely delivery of features with strong attention to performance, scalability, and maintainability. ● An in-depth understanding of design principles and thinks technology as a solution enablement tool. ● Have a logical mindset and an aptitude to resolve highl...

Our Purpose Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential. Title And Summary Technical Manager, Data Engineering Background Mastercard is on a journey to modernize and advance our Finance Technology landscape, covering Billing, Financial Planning, Acc...

Overview We are seeking a detail-oriented ETL cum BI Engineer with strong expertise in Tableau and Databricks to maintain, optimize, and enhance our reporting infrastructure. This role focuses on dashboard upkeep, data flow management, and enabling new analytical capabilities across multiple regions and business units. The ideal candidate will ensure high data accuracy, reliability, and availability for recurring reporting requirements. Key Role and Responsibilities Dashboard Maintenance & Optimization Maintain and troubleshoot existing Tableau dashboards to ensure accuracy, performance, and reliability. Oversee active dashboards used for recurring reporting, ensuring they remain current and...

To define structure & standards for data processing, reporting & complex computation projects.Python coding for enterprise projects, PySpark SQL for data access & transformation, integration with databases & UI platforms, maths/statistics computation Required Candidate profile Must-Haves: Extensive experience defining scalable architectures using Python, specifically for data & analytical solutions using PySpark. Architect our next-generation data & analytics platforms!

Role: Azure Data Engineer Location: All EXL Location Work Mode: Hybrid Key Responsibilities Design and develop ETL/ELT pipelines using Azure Data Factory, Snowflake, and DBT. Build and maintain data integration workflows from various data sources to Snowflake. Write efficient and optimized SQL queries for data extraction and transformation. Work with stakeholders to understand business requirements and translate them into technical solutions. Monitor, troubleshoot, and optimize data pipelines for performance and reliability. Maintain and enforce data quality, governance, and documentation standards. Collaborate with data analysts, architects, and DevOps teams in a cloud-native environment. M...

Responsibilities: Perform exploratory data analysis and various data analysis Build robust data pipeline jobs Design, implement and monitor Machine Learning models Conduct A/B testing Collaborate with cross-functional teams Required Qualifications : Proficiency in Python. Bachelors's degree in Mathematics, Statistics, or a related field. Strong proficiency in data analytics and statistical analysis Excellent problem-solving skills and ability to deliver actionable insights. Strong communication skills and ability to collaborate effectively with cross-functional teams. Preferred Qualifications: Proficiency in PySpark and SQL Master's degree in Mathematics, Statistics, or a related field. Expe...