15802 Pyspark Jobs

As part of the AI & Data organization, the Enterprise Business Intelligence (EBI) team at NXP is crucial to the success of data analytics. The team offers scalable data solutions, platforms, and methodologies that empower business users to create self-service analytics and make data-informed decisions. **Key Responsibilities:** - Lead, mentor, and grow a team of Data Engineers, fostering innovation, collaboration, and continuous improvement. - Oversee the design, development, and optimization of ETL/ELT pipelines and data workflows across multiple cloud and on-premise environments. - Ensure data solutions align with enterprise architecture standards, focusing on performance, scalability, sec...

Summary: As a Business Intelligence Engineer at Innover, you will be a highly motivated individual with 5+ years of experience in enterprise and/or hosted applications development. Your expertise in cloud application development, design, and implementation will be crucial in this role. You should have broad experience in various types of databases, distributed, and web-based reporting & analytics applications. Technical Skills and Key Interests: Minimum Qualifications: - Bachelor's in computer science or equivalent work experience. - At least 2 years of experience in developing reports, views, summarizations, and dashboards using Reporting tools such as SAP BO, Tableau, Cognos, PowerBI, or T...

As an Infoscion, a typical day in your role involves interfacing with the client for quality assurance, issue resolution, and ensuring high customer satisfaction. You will be responsible for understanding requirements, creating and reviewing designs, validating architecture, and ensuring high levels of service offerings to clients in the technology domain. Your contributions will include participating in project estimation, providing inputs for solution delivery, conducting technical risk planning, performing code reviews, and unit test plan reviews. Leading and guiding your teams towards developing optimized high-quality code deliverables, continual knowledge management, and adherence to or...

As a candidate for this role, you should have over 10 years of experience in designing and implementing scalable data solutions. This includes creating end-to-end ETL/ELT pipelines using Databricks (PySpark) and Snowflake. You will be responsible for building and managing secure, scalable data platforms using PostgreSQL and DynamoDB that are tailored to the needs of the applications. Your tasks will also involve developing real-time and batch ingestion pipelines from various sources like APIs, logs, files, and databases. Key Responsibilities: - Apply transformation logic to clean, enrich, and normalize data for analytics and reporting. - Optimize pipeline performance and storage costs by imp...

Job Description: You will be responsible for understanding Pyspark interface and integration to handle complexities of multiprocessing, such as distributing data, code, and collecting output from workers on a cluster of machines. Key Responsibilities: - Analyze and solve moderately complex problems - Create new solutions by leveraging existing methods and procedures - Understand the strategic direction set by senior management and align with team goals - Interact with direct supervisor, peers, and/or management levels at a client or within Accenture - Provide guidance on determining methods and procedures for new assignments - Make decisions that impact the team - Manage small teams and/or w...

As a data infrastructure expert at MPOWER, you will be responsible for designing, building, and maintaining data pipelines in Python and AWS to support the company's business decisions. Your key responsibilities will include: - Designing, building, and maintaining data pipelines in Python and AWS, utilizing services like AWS Redshift, AWS Glue, and MWAA. - Ensuring data quality and integrity through validation, cleansing, and error handling techniques. - Monitoring, troubleshooting, and optimizing the performance of data pipelines for reliability and scalability. - Establishing user needs, monitoring access and security, and capacity planning for database storage requirements. - Building dat...

Job Description: Python Developer / Data Engineer – Talend to Python Migration About the Role We are seeking experienced Python Developers/Data Engineers to join our team for a strategic migration project, moving enterprise ETL workloads from Talend to a modern Python-based ingestion framework on Google Cloud Platform (GCP). You will work closely with architects, platform engineers, and automation teams to design, build, and optimize scalable data pipelines, leveraging Airflow, Cloud Run, Dataproc, and BigQuery. Key Responsibilities * ETL Migration & Development * Analyze existing Talend jobs/modules and design equivalent Python-based ETL workflows. * Develop, test, and deploy Python scripts...

The Opportunity: A data analyst is responsible for organizing data related to sales numbers, market research, logistics, or other behaviors. They utilize technical expertise to ensure data is accurate and high-quality. Data is then analyzed, designed, and presented in a way that assists individuals, businesses, and organizations make better decisions. THE OPPORTUNITY: As a Data Scientist, you will leverage advanced statistical methods and machine learning techniques to analyze complex datasets, generate insights, and solve business problems. You will be responsible for developing and deploying ML models, performing deep-dive analyses, and collaborating with stakeholders to ensure the data st...

DESCRIPTION Are you passionate about transforming complex data into actionable business insights at a global scale? RBS Brand Experience (formerly APIE) is seeking an experienced Business Intelligence Engineer who thrives on ambiguity and can decipher evolving business needs to shape data-driven solutions. As a Business Intelligence Engineer, you'll be at the intersection of data and business strategy, translating complex requirements into actionable analytics solutions. You'll partner with stakeholders to unlock insights that elevate our global work authorization experiences and drive program scalability. Key job responsibilities A successful candidate will demonstrate: Advanced SQL skills ...

Are you passionate about building robust Python applications and leveraging big data technologies to solve complex business challenges? Do you thrive in environments where your technical expertise drives meaningful impact? We are seeking an experienced Python Lead Developer to join our dynamic LSH Data Platform Team. You will play a pivotal role in designing, developing, and maintaining Python-based data processing applications and services that support scalable big data solutions. If you're excited about combining deep Python skills with cutting-edge technologies like Apache Spark, PySpark, GraphDB, SQL, and relational databases, we want to hear from you. About You – experience, education, ...

DESCRIPTION Are you passionate about transforming complex data into actionable business insights at a global scale? RBS Brand Experience (formerly APIE) is seeking an experienced Business Intelligence Engineer who thrives on ambiguity and can decipher evolving business needs to shape data-driven solutions. As a Business Intelligence Engineer, you'll be at the intersection of data and business strategy, translating complex requirements into actionable analytics solutions. You'll partner with stakeholders to unlock insights that elevate our global work authorization experiences and drive program scalability. Key job responsibilities A successful candidate will demonstrate: Advanced SQL skills ...

As a Python Web Scraper at HiLabs in Bengaluru, Karnataka, you will play a crucial role in designing and building scalable web scraping solutions using Python/PySpark. Your responsibilities will include developing enterprise-grade scraping services, working with structured and unstructured data, implementing data validation processes, and collaborating with cross-functional teams to ensure smooth data flow between systems. Your technical expertise in Python, web scraping tools and libraries, PySpark, cloud-based platforms, and databases will be essential for optimizing data workflows and ensuring data accuracy, consistency, and availability. **Responsibilities:** - Design and build scalable,...

Key Skill – IAM role setup , S3 setup through GitHub cdk , good with python Scripting to be able to create pipeline , policies setup etc & must have experience in: Kubernetes, AWS cloud, CICD especially GitHub Actions Highly adaptable to changing circumstances. Interest in continuously learning new skills and technologies. Experience with programming and scripting languages (Python, Bash, PowerShell). Experience with incident and response management. Experience with Agile and DevOps development methodologies. Experience with container technologies and supporting tools (e.g. Docker Swarm, Kubernetes). Run and maintain our production infrastructure hosted on AWS/AZURE. Experience with working ...

3 to 5 years of experience in data engineering roles. Strong expertise in PySpark Hands-on experience with Databricks. Experience with cloud platforms (Azure Data Lake, AWS S3, or GCP BigQuery).

Data Axle Inc. has been an industry leader in data, marketing solutions, sales, and research for over 50 years in the USA. Data Axle now has an established strategic global centre of excellence in Pune. This centre delivers mission-critical data services to its global customers powered by its proprietary cloud-based technology platform and by leveraging proprietary business and consumer databases. Data Axle India is recognized as a Great Place to Work! This prestigious designation is a testament to our collective efforts in fostering an exceptional workplace culture and creating an environment where every team member can thrive. As an Associate Data Scientist at Data Axle, you will be joinin...

Role Overview: As a proficient individual, your main responsibility will be to systematically develop and promote technology solutions that meet both functional and non-functional requirements. Your outcomes should include developing and promoting technical solutions supporting business requirements, ensuring IT requirements are met, and service quality is maintained when introducing new services. It is essential to consider the cost-effectiveness of proposed solutions and set FAST goals while providing feedback to mentees. Your role also includes being innovative and technically sound in project analysis, upholding standards and best practices, providing innovative contributions to automate...

Role Overview: You will be responsible for capturing user requirements and translating them into business and digitally enabled solutions across a range of industries. Your role will involve designing, developing, optimizing, and maintaining data pipelines that adhere to ETL principles and business goals. Additionally, you will be solving complex data problems to deliver insights that help the business achieve its goals, sourcing data from various touchpoints, creating data products for the analytics team, and fostering a culture of sharing and operational efficiency of data and analytical solutions. Key Responsibilities: - Designing, developing, optimizing, and maintaining data pipelines th...

Role Overview: You will be joining the Anti-Financial Crime (AFC) Technology team in Pune, India, working as a Senior Engineer, VP specializing in designing, developing, and testing engineering solutions for Transaction Monitoring (TM) systems to detect Money Laundering or Terrorism Financing. You will have the opportunity to work on challenging problems, optimizing data flows, and implementing solutions using Cloud and BigData technologies. Key Responsibilities: - Create efficient ETL workflows to extract, transform, and load data into TM systems, ensuring data quality and integrity. - Collaborate with developers and architects to design scalable and sustainable solutions that meet business...

You will be responsible for the following tasks in the role of Data Engineer (MS Fabric): - Hands-on experience with ETL processes using PySpark, SQL, Microsoft Fabric, and other relevant technologies. - Collaborate with clients and other stakeholders to understand data requirements and design efficient data models and solutions. - Optimize and tune existing data pipelines for performance and scalability. - Ensure data quality and integrity throughout the data pipeline. - Document technical designs, processes, and procedures. - Stay updated on emerging technologies and best practices in data engineering. - Build CICD pipeline using Github. Qualifications required for this role include: - Bac...

As a Data Engineer / Analyst at our company, your main responsibilities will include: - Designing, developing, and maintaining data solutions on Azure Databricks using Python, SQL, and PySpark. - Working extensively with Delta Lake and relational databases to build scalable data pipelines. - Performing analysis, design, implementation, and automated testing of data solutions. - Enhancing the consistency check engine for sub-ledger data and other tenants. - Optimizing Spark performance for very large datasets (10s of billions of rows, 200+ columns). - Identifying, troubleshooting, and resolving data issues proactively. - Collaborating with cross-functional teams to ensure data reliability, co...

As a Senior Analyst - Data Engineering at EY-Strategy and Transactions - SaT DnA, you will play a crucial role in supporting cloud and on-prem platform analytics and data engineering projects across engagement teams. Your primary responsibilities will involve conceptualizing, designing, developing, deploying, and maintaining complex technology solutions to solve business problems for clients. You will work closely with technical architects, product and business subject matter experts, back-end developers, and solution architects, contributing to the design, development, and evolution of modern data warehousing solutions and data integration build-outs using cutting-edge tools and platforms f...

As the Lead Technical Architect for Strategy & Optimization Data Lake & Analytics, your role will involve the following responsibilities: - Manage Project Delivery, including overseeing scope, timelines, budget, resource allocation, and risk mitigation. - Develop and maintain robust data ingestion pipelines, covering batch, streaming, and API sources. - Provide architectural inputs during incident escalations and act as the final authority for RCA documentation and closure related to ADF, Power BI, and Databricks. - Define and enforce data governance, metadata, and quality standards across different zones. - Monitor performance, optimize data formats such as Parquet, and tune for cost-effici...

As a Manager - Business Consulting at HSBC, you will play a crucial role in providing analytical support to Global and Regional business teams across risk operations analytics, product analytics, and customer journey and experience. Your responsibilities will include: - Conducting Data Quality Analysis and exploratory data analysis to identify appropriate datasets for analytics use cases. - Developing data-based approaches to business problems through descriptive and predictive analytics. - Creating Trend Analysis and Dashboards using Visualization techniques primarily on Qliksense. - Executing projects and analysis within agreed timelines, ensuring accuracy and quality. - Delivering end-to-...

Role Overview: As a Senior Data Engineer with 10-12 years of experience, you will play a crucial role in building scalable batch and streaming data pipelines using Databricks (PySpark) and Snowflake. You will lead the design, implementation, and optimization of application data stores using PostgreSQL, DynamoDB, and advanced SQL. Your strong programming skills in SQL, Python, and PySpark will be essential for efficient data processing. Additionally, you will work with structured (CSV, SQL), semi-structured (JSON, XML), and unstructured (PDF, logs) data formats. Key Responsibilities: - Design and build scalable batch and streaming data pipelines using Databricks (PySpark) and Snowflake. - Lea...

**Job Description:** As a Full Stack AI Developer at GlobalLogic, you will be responsible for developing and implementing AI solutions using a variety of programming languages and data processing libraries. Your role will involve working on distributed computing platforms, API development, and cloud platforms to create cutting-edge solutions for our clients. **Key Responsibilities:** - Must Have: Proficiency in programming languages such as Python, Java, or Scala - Must Have: Experience with data processing libraries like Pandas, NumPy, and Scikit-learn - Must Have: Proficient in distributed computing platforms like Apache Spark (PySpark, Scala), Torch, etc. - Must Have: Proficiency in API d...