15947 Pyspark Jobs - Page 43

Role Description Job Title: Data Engineer – AWS & PySpark Location: Trivandrum/Kochi/Chennai/Hyderabad/Bangalore/Pune/Noida Experience Required: 5–8 years Employment Type: Full-Time Department: Data Engineering / Technology Job Summary: We are seeking a skilled Data Engineer with 5–8 years of experience to design, implement, and maintain robust, scalable data architectures on AWS . The ideal candidate will be highly proficient in PySpark and experienced with AWS cloud data services including S3, Glue, and Redshift . You will work closely with cross-functional teams to enable seamless data flows and ensure efficient ETL pipelines across our cloud infrastructure. Key Responsibilities: Design, ...

Role Description Job Title: Data Engineer – AWS & PySpark Location: Trivandrum/Kochi/Chennai/Hyderabad/Bangalore/Pune/Noida Experience Required: 5–8 years Employment Type: Full-Time Department: Data Engineering / Technology Job Summary We are seeking a skilled Data Engineer with 5–8 years of experience to design, implement, and maintain robust, scalable data architectures on AWS . The ideal candidate will be highly proficient in PySpark and experienced with AWS cloud data services including S3, Glue, and Redshift . You will work closely with cross-functional teams to enable seamless data flows and ensure efficient ETL pipelines across our cloud infrastructure. Key Responsibilities Design, im...

Role Description UST is looking for a skilled Snowflake Engineer with 5 to 7 years of experience to join our team and contribute to the development and maintenance of efficient data solutions on the Snowflake cloud data platform. The ideal candidate should possess strong data engineering skills, expertise in Snowflake, and proficiency in data processing technologies, with a specific focus on PySpark. Snowflake Data Warehouse Design, implement, and optimize data warehouses on the Snowflake platform. Ensure effective utilization of Snowflake features for scalable and high-performance data storage. Data Pipeline Development Develop, implement, and optimize end-to-end data pipelines on Snowflake...

Role Description Key Responsibilities: Big Data Architecture: Design, develop, and maintain scalable and distributed data architectures capable of processing large volumes of data. Data Storage Solutions: Implement and optimize data storage solutions using technologies such as Hadoop, Spark, and PySpark. PySpark Development: Develop and implement efficient ETL processes using PySpark to extract, transform, and load large datasets. Performance Optimization: Optimize PySpark applications for better performance, scalability, and resource management. Qualifications Proven experience as a Big Data Engineer with a strong focus on PySpark. Deep understanding of Big Data processing frameworks and te...

3+ years of experience in production support of the Databricks platform. Implement and maintain the Databricks platform on AWS. Develop and maintain automation scripts and tools for platform provisioning and routine tasks using Terraform. Install, configure, and maintain Databricks clusters and workspaces. Implement, and maintain the Databricks platform, including workspace setup, user and group management, access control, and security configurations. Provide technical support to data engineering, data science, and application teams. Perform restores/recoveries, troubleshoot service issues, and resolve platform-related problems. Monitor and manage cluster performance, resource utilization, a...

Role Description Develop and manage ETL workflows using Azure Data Factory (ADF). Design and implement data pipelines using PySpark on Azure Databricks. Work with Azure Synapse Analytics, Azure Data Lake, and Azure Blob Storage for data ingestion and transformation. Optimize Spark jobs for performance and scalability in Databricks. Automate data workflows and implement error handling & monitoring in ADF. Collaborate with data engineers, analysts, and business teams to understand data requirements. Implement data governance, security, and compliance best practices in Azure. Debug and troubleshoot PySpark scripts and ADF pipeline failures. 4+ years of experience in ETL development with Azure D...

Role Description Role Proficiency: Independently develops error free code with high quality validation of applications guides other developers and assists Lead 1 – Software Engineering Outcomes Understand and provide input to the application/feature/component designs; developing the same in accordance with user stories/requirements. Code debug test document and communicate product/component/features at development stages. Select appropriate technical options for development such as reusing improving or reconfiguration of existing components. Optimise efficiency cost and quality by identifying opportunities for automation/process improvements and agile delivery models Mentor Developer 1 – Sof...

Role Description Azure Infrastructure Consultant – Cloud & Data Integration Experience: 8+ Years Employment Type: Full-Time Industry: Information Technology / Cloud Infrastructure / Data Engineering Job Summary: We are looking for a seasoned Azure Infrastructure Consultant with a strong foundation in cloud infrastructure, data integration, and real-time data processing. The ideal candidate will have hands-on experience across Azure and AWS platforms, with deep knowledge of Apache NiFi, Kafka, AWS Glue, and PySpark. This role involves designing and implementing secure, scalable, and high-performance cloud infrastructure and data pipelines. Key Responsibilities: Design and implement Azure-base...

Role Description Job Title: Azure Solution Architect Experience: 10+ years Location: Trivandrum Notice Period: Immediate Job Description: We are looking for a highly skilled Azure PySpark Solution Architect to design and implement scalable data solutions on Microsoft Azure. The ideal candidate will possess strong expertise in Azure services, PySpark, and enterprise-grade solution architecture to drive efficient data processing and analytics workflows. Key Responsibilities: Design and implement end-to-end data solutions using Azure Data Services and PySpark. Develop high-performance ETL pipelines leveraging Azure Databricks, Azure Data Factory, and Synapse Analytics. Architect scalable, secur...

Role Description Job: Data Engineer Experience: 7+ years Mandatory Skill: Python, Pyspark, Azure Databricks, SQL Location: Trivandrum, Bangalore, Kochi Required Skills & Experience Experience with large-scale distributed data processing systems. Expertise in data modelling, testing, quality, access, and storage. Proficiency in Python, SQL, and experience with Databricks and DBT. Experience implementing cloud data technologies (GCP, Azure, or AWS). Knowledge of improving the data development lifecycle and shortening lead times. Agile delivery experience. Skills Python,Pyspark,Azure Databricks, SQL

Role Description Role Proficiency: This role requires proficiency in developing data pipelines including coding and testing for ingesting wrangling transforming and joining data from various sources. The ideal candidate should be adept in ETL tools like Informatica Glue Databricks and DataProc with strong coding skills in Python PySpark and SQL. This position demands independence and proficiency across various data domains. Expertise in data warehousing solutions such as Snowflake BigQuery Lakehouse and Delta Lake is essential including the ability to calculate processing costs and address performance issues. A solid understanding of DevOps and infrastructure needs is also required. Outcomes...

Role Description Role Proficiency: This role requires proficiency in developing data pipelines including coding and testing for ingesting wrangling transforming and joining data from various sources. The ideal candidate should be adept in ETL tools like Informatica Glue Databricks and DataProc with strong coding skills in Python PySpark and SQL. This position demands independence and proficiency across various data domains. Expertise in data warehousing solutions such as Snowflake BigQuery Lakehouse and Delta Lake is essential including the ability to calculate processing costs and address performance issues. A solid understanding of DevOps and infrastructure needs is also required. Outcomes...

Role Description Role Proficiency: Act creatively to develop applications and select appropriate technical options optimizing application development maintenance and performance by employing design patterns and reusing proven solutions account for others' developmental activities Outcomes Interpret the application/feature/component design to develop the same in accordance with specifications. Code debug test document and communicate product/component/feature development stages. Validate results with user representatives; integrates and commissions the overall solution Select appropriate technical options for development such as reusing improving or reconfiguration of existing components or c...

Role Description Experience Range: 6-12 years Location: Trivandrum, Kochi, Bangalore Mandatory Skills Programming Languages: Scala, Spark, PySpark, Python, SQL Big Data Technologies: Hadoop, Hive, Pig, MapReduce ETL & Data Engineering: Data Warehouse Design, ETL, Data Analytics, Data Mining, Data Cleansing Cloud Platforms: GCP, Azure Tools & Frameworks: Apache Hadoop, Airflow, Kubernetes, Containers Other Skills: Data pipeline creation, optimization, troubleshooting, and data validation Work Experience: 6+ years in Data Warehouse and Big Data technologies 4+ years of hands-on experience with Scala, Spark, PySpark, Python, and SQL 3+ years in strategic data planning, governance, and standard ...

Role Description Job Overview: UST is seeking a skilled Snowflake Engineer with 6 to 10 years of experience to join our team. The ideal candidate will play a key role in the development, implementation, and optimization of data solutions on the Snowflake cloud platform. The candidate should have strong expertise in Snowflake , and a solid understanding of ETL processes, along with proficiency in data engineering and data processing technologies. This role is essential for designing and maintaining high-performance data pipelines and data warehouses , focusing on scalability and efficient data storage. Key Responsibilities Snowflake Data Warehouse Development: Design, implement, and optimize ...

Join our digital revolution in NatWest Digital X In everything we do, we work to one aim. To make digital experiences which are effortless and secure. So we organise ourselves around three principles: engineer, protect, and operate. We engineer simple solutions, we protect our customers, and we operate smarter. Our people work differently depending on their jobs and needs. From hybrid working to flexible hours, we have plenty of options that help our people to thrive. This role is based in India and as such all normal working days must be carried out in India. Job Description Join us as a Quality Automation Specialist, Selenium In this key role, you’ll be undertaking and enabling automated t...

Job Summary A Sr Data Engineer is responsible for designing, building, and maintaining scalable data pipelines and infrastructure to support data collection, transformation, and analysis. They work closely with data scientists, analysts, and business stakeholders to ensure data is accessible, reliable, and optimized for performance. Key responsibilities include developing ETL processes, managing data warehouses and lakes, ensuring data quality, and implementing best practices for data governance and security. • Overall having experience between 6 to 10 years of experience in IT and minimum 3+ years’ experience in cloud. • Hands-on in Architecture, Design, Delivery of all aspects of Data Engi...

Job Title: Senior Databricks Developer Location: Remote About Us: [Insert 1-2 sentences about your company's mission and the impact of its data team.] The Role: We are looking for a passionate Databricks Developer to architect and implement our next-generation data platform. You will be instrumental in building robust, scalable data pipelines, optimizing our cloud data warehouse, and creating powerful visualizations that drive key business decisions. Key Responsibilities: Design, develop, and maintain efficient and reliable ETL/ELT pipelines using Databricks, PySpark, and Spark SQL. Build and optimize our cloud data warehouse (e.g., Snowflake, BigQuery, Synapse) for performance and scalabili...

Position: Data Engineer (Databricks + Power BI) Experience: 3+ Years Location: Remote About the Role We are seeking a skilled Data Engineer with hands-on expertise in Databricks and Power BI . The ideal candidate will design, build, and optimize scalable data pipelines, integrate datasets across platforms, and deliver actionable insights through interactive dashboards. Key Responsibilities Design and develop ETL pipelines using Databricks (PySpark/SQL) . Integrate data from diverse sources (SQL, APIs, cloud storage) into Databricks. Optimize Spark jobs for performance, scalability, and cost-efficiency . Implement and manage Delta Lake, Lakehouse architecture , and data governance. Design and...

About Company: Our Client Corporation provides digital engineering and technology services to Forbes Global 2000 companies worldwide. Our Engineering First approach ensures we can execute all ideas and creatively solve pressing business challenges. With industry expertise and empowered agile teams, we prioritize execution early in the process for impactful results. We combine logic, creativity and curiosity to build, solve, and create. Every day, we help clients engage with new technology paradigms, creatively building solutions that solve their most pressing business challenges and move them to the forefront of their industry. Job Title : GCP Big Data Engineer Key Skills : PySpark , Airflow...

About Client: Our Client is a global IT services company headquartered in Southborough, Massachusetts, USA. Founded in 1996, with a revenue of $1.8B, with 35,000+ associates worldwide, specializes in digital engineering, and IT services company helping clients modernize their technology infrastructure, adopt cloud and AI solutions, and accelerate innovation. It partners with major firms in banking, healthcare, telecom, and media. Our Client is known for combining deep industry expertise with agile development practices, enabling scalable and cost-effective digital transformation. The company operates in over 50 locations across more than 25 countries, has delivery centers in Asia, Europe, an...

About Client: Our Client is a global IT services company headquartered in Southborough, Massachusetts, USA. Founded in 1996, with a revenue of $1.8B, with 35,000+ associates worldwide, specializes in digital engineering, and IT services company helping clients modernize their technology infrastructure, adopt cloud and AI solutions, and accelerate innovation. It partners with major firms in banking, healthcare, telecom, and media. Our Client is known for combining deep industry expertise with agile development practices, enabling scalable and cost-effective digital transformation. The company operates in over 50 locations across more than 25 countries, has delivery centers in Asia, Europe, an...

About Client: Our Client is a global IT services company headquartered in Southborough, Massachusetts, USA. Founded in 1996, with a revenue of $1.8B, with 35,000+ associates worldwide, specializes in digital engineering, and IT services company helping clients modernize their technology infrastructure, adopt cloud and AI solutions, and accelerate innovation. It partners with major firms in banking, healthcare, telecom, and media. Our Client is known for combining deep industry expertise with agile development practices, enabling scalable and cost-effective digital transformation. The company operates in over 50 locations across more than 25 countries, has delivery centers in Asia, Europe, an...

About Client: Our Client is a global IT services company headquartered in Southborough, Massachusetts, USA. Founded in 1996, with a revenue of $1.8B, with 35,000+ associates worldwide, specializes in digital engineering, and IT services company helping clients modernize their technology infrastructure, adopt cloud and AI solutions, and accelerate innovation. It partners with major firms in banking, healthcare, telecom, and media. Our Client is known for combining deep industry expertise with agile development practices, enabling scalable and cost-effective digital transformation. The company operates in over 50 locations across more than 25 countries, has delivery centers in Asia, Europe, an...

The Position:- We are seeking a highly skilled Data Scientist to lead the development and optimization of our Forecasting and Measurement & Verification (M&V) platform. This platform supports forecasting and M&V for approximately 3 million distributed energy devices at once. As a senior member of the Forecasting team, you will collaborate closely with data engineers, software engineers, and SREs to enhance core algorithms, improve scalability, and deliver new services. You will play a key role in translating business needs into robust data science solutions that run in production at scale. This position is ideal for someone who is passionate about using data science to drive real-world impac...