31 Dataframes Jobs

Role & responsibilities : Data Pipeline Development & Management Design and implement robust, scalable ETL/ELT pipelines using Databricks and Apache Spark Process large volumes of structured and unstructured data Develop and maintain data workflows using Databricks workflows, Apache Airflow, or similar orchestration tools Optimize data processing jobs for performance, cost efficiency, and reliability Implement incremental data processing patterns and change data capture (CDC) mechanisms Databricks Platform Engineering Build and maintain Delta Lake tables and implement medallion architecture (bronze, silver, gold layers) Develop streaming data pipelines using Structured Streaming and Delta Li...

As a Data Engineer - Pyspark at Barclays, you will be spearheading the evolution of the digital landscape, driving innovation and excellence. You will be utilizing cutting-edge technology to revolutionize digital offerings, ensuring unparalleled customer experiences. As part of a team of developers, you will be responsible for delivering the technology stack, using strong analytical and problem-solving skills to understand business requirements and deliver quality solutions. **Key Responsibilities:** - Hands-on experience in Pyspark and strong knowledge of Dataframes, RDD, and SparkSQL. - Hands-on experience in Pyspark performance optimization techniques. - Hands-on experience in developing,...

As a candidate for this role, you would be responsible for the following: - Understanding of Spark core concepts like RDDs, DataFrames, DataSets, SparkSQL, and Spark Streaming. - Experience with Spark optimization techniques. - Deep knowledge of Delta Lake features like time travel, schema evolution, data partitioning. - Ability to design and implement data pipelines using Spark and Delta Lake as the data storage layer. - Proficiency in Python/Scala/Java for Spark development and integrate with ETL process. - Knowledge of data ingestion techniques from various sources (flat files, CSV, API, database). - Understanding of data quality best practices and data validation techniques. Qualificatio...

As an experienced candidate with over 8 years of overall experience and 5+ years of relevant experience, your role will involve the following key responsibilities: - Strong understanding of Spark core including RDDs, DataFrames, DataSets, SparkSQL, and Spark Streaming - Proficiency in Delta Lake features such as time travel, schema evolution, and data partitioning - Designing and building data pipelines using Spark and Delta Lake - Solid experience in Python/Scala/Java for Spark development - Knowledge of data ingestion from files, APIs, and databases - Familiarity with data validation and quality best practices - Working knowledge of data warehouse concepts and data modeling - Hands-on expe...

As a skilled PySpark Developer with hands-on experience in Reltio MDM, your role will involve designing and implementing scalable data processing solutions using PySpark and integrating with Reltio's cloud-native MDM platform. Key Responsibilities: - Develop and maintain data pipelines using PySpark in distributed computing environments such as AWS EMR and Databricks. - Integrate and synchronize data between enterprise systems and the Reltio MDM platform. - Design and implement data transformation, cleansing, and enrichment processes. - Collaborate with data architects, business analysts, and Reltio solution architects to ensure high-quality data modeling. - Work on API-based integration bet...

Join us as a Data Engineer at Barclays, where you will spearhead the evolution of our infrastructure and deployment pipelines, driving innovation and operational excellence. You will harness cutting-edge technology to build and manage robust, scalable and secure infrastructure, ensuring seamless delivery of our digital solutions. To be successful as a Data Engineer, you should have experience with: Hands on experience in pyspark and strong knowledge on Dataframes, RDD and SparkSQL. Hands on Experience in developing, testing and maintaining applications on AWS Cloud. Strong hold on AWS Data Analytics Technology Stack (Glue, S3, Lambda, Lake formation, Athena). Design and implement scalable an...

Role Overview: You will be responsible for developing Python back end solutions with GenAI exposure. Your key responsibilities will include: - Demonstrating strong experience in Python programming, specifically with Django and Flasks frameworks - Utilizing dataframes, Pyspark, SQL, and Databricks SQL for effective data processing - Working with ADLS Unity Catalog for data storage and management - Applying Agile methodology to software development projects - Leveraging GenAI for enhanced AI capabilities - Operating in an Azure DevOps environment for efficient project management Qualification Required: You should possess the following qualifications: - 5+ years of experience in Python developm...

As an FDR/CVR Data Analyst, you will play a critical role in ensuring the safety and efficiency of aircraft operations by performing in-depth analysis of flight data. You will work with large datasets from flight recorders to monitor aircraft health and support return-to-service processes. Collaborating with various teams, you will interpret data, develop reports, and ensure compliance with industry standards. Additionally, you will contribute to the development of new data analysis methodologies and tools, provide technical expertise, and maintain thorough documentation. - Perform detailed readouts and analysis of FDR and CVR data to support aircraft health monitoring and return-to-service ...

As a Data Engineer specializing in Spark & Delta Lake, your role will involve understanding key Spark concepts such as RDDs, DataFrames, DataSets, SparkSQL, and Spark Streaming. You will also be expected to have experience in optimizing Spark performance and a deep understanding of Delta Lake features like time travel, schema evolution, and data partitioning. Your responsibilities will include designing and implementing data pipelines using Spark and Delta Lake as the data storage layer. Proficiency in Python, Scala, or Java for Spark development and integration with ETL processes is essential. Additionally, you should possess knowledge of data ingestion techniques from various sources such ...

Join us as a Data Engineer responsible for supporting the successful delivery of Location Strategy projects to plan, budget, agreed quality and governance standards. You'll spearhead the evolution of our digital landscape, driving innovation and excellence. You will harness cutting-edge technology to revolutionise our digital offerings, ensuring unparalleled customer experiences. To be successful as a Data Engineer you should have experience with: Hands on experience in pyspark and strong knowledge on Dataframes, RDD and SparkSQL Hands on Experience in developing, testing and maintaining applications on AWS Cloud. Strong hold on AWS Data Analytics Technology Stack (Glue, S3, Lambda, Lake for...

Join us as a Data Engineer - Pyspark,SQL at Barclays, where you'll spearhead the evolution of our digital landscape, driving innovation and excellence. You'll harness cutting-edge technology to revolutionise our digital offerings, ensuring unparalleled customer experiences. As a part of team of developers, you will deliver technology stack, using strong analytical and problem solving skills to understand the business requirements and deliver quality solutions. To be successful as a Data Engineer - Pyspark,SQL you should have experience with: Hands on experience in Pyspark and strong knowledge on Dataframes, RDD and SparkSQL Hands on experience in Pyspark performance optimization techniques ....

Job Description: As a Spark Expert, you will be expected to demonstrate expert proficiency in Spark and possess the ability to design and implement efficient data processing workflows. Your experience with Spark SQL and DataFrames will be crucial, as well as your good exposure to Big Data architectures and a solid understanding of the Big Data ecosystem. Additionally, your experience with framework building on Hadoop and strong DB knowledge with SQL tuning experience will be valuable. It would be beneficial if you have experience with Python APIs and exposure to Kafka. Key Responsibilities: - Demonstrate expert proficiency in Spark - Design and implement efficient data processing workflows -...

Role Overview: As a Senior Azure Data Engineer, you will utilize Azure technologies to facilitate data transformation, analytics, and machine learning. Your primary responsibility will involve designing scalable Databricks data pipelines using PySpark to convert raw data into actionable insights. Additionally, you will be tasked with constructing, deploying, and managing machine learning models using MLlib or TensorFlow, all while optimizing cloud data integration from various Azure sources such as Blob Storage, Data Lake, and SQL/NoSQL databases. Your role will also encompass executing large-scale data processing using Spark Pools and fine-tuning configurations for efficiency. Key Responsib...

When you join BairesDev, you become part of a team that has been at the forefront of technology projects for over 15 years. We are dedicated to delivering cutting-edge solutions to industry giants like Google and the most innovative startups in Silicon Valley. Our diverse team of over 4,000 professionals, who represent the world's Top 1% of tech talent, works on roles that have a significant global impact. When you apply for the position of Apache Spark Developer at BairesDev, you are embarking on a journey that goes beyond the ordinary. We are committed to matching your passions and skills with our job opportunities, guiding you towards exceptional career growth and success. Role Overview: ...

Onsite Opportunity Sr. Data Engineer / Lead Data Engineer Client: MNC Experience:8+years Location: Ras Al-Khaimah, UAE Job Mode: Permanent Salary Range: 15K AED to 20K AED Note: Selected candidates must hold a valid passport. Selected candidates should be open to travel to Ras Al-Khaimah, UAE. Candidates must possess strong communication skills. Key skills: Python, Data Engineer, Databricks, SQL, Shell Scripting, Snowflake, PySpark, dataframes, Azure Data Factory, Apache Airflow, and Databricks workflows, FastAPI, Azure etc.. Job Description: We are looking for a skilled Full Stack Developer with hands-on experience in building scalable web applications using Python (FastAPI/Flask/Django). Y...

As a Data Engineer specializing in Pyspark and SQL at Barclays, your main role will involve spearheading the evolution of the digital landscape, driving innovation and excellence within the company. You will be responsible for harnessing cutting-edge technology to revolutionize digital offerings, ensuring unparalleled customer experiences. Working as part of a team of developers, your primary focus will be delivering a technology stack using your strong analytical and problem-solving skills to understand business requirements and deliver quality solutions. Key Responsibilities: - Hands-on experience in Pyspark with a strong knowledge of Dataframes, RDD, and SparkSQL. - Proficiency in Pyspark...

Join us as a Data Engineer responsible for supporting the successful delivery of Location Strategy projects to plan, budget, agreed quality, and governance standards. You'll spearhead the evolution of our digital landscape, driving innovation and excellence. You will harness cutting-edge technology to revolutionize our digital offerings, ensuring unparalleled customer experiences. To be successful as a Data Engineer, you should have experience with: - Hands-on experience in PySpark and strong knowledge of Dataframes, RDD, and SparkSQL. - Hands-on experience in developing, testing, and maintaining applications on AWS Cloud. - Strong hold on AWS Data Analytics Technology Stack (Glue, S3, Lambd...

We are searching for a proficient Python Developer to become a valuable member of our product development team. Your primary focus will involve automating data ingestion, processing, and validation workflows utilizing PySpark to create robust, scalable, and efficient data pipelines. In this role, you will collaborate closely with data engineers, analysts, and stakeholders to provide impactful data solutions within a dynamic work environment. Your responsibilities will include collaborating with data engineers and stakeholders to outline requirements and implement automated solutions. You will design, develop, and upkeep scalable and efficient PySpark-based automation for data ingestion, proc...

The ideal candidate will be responsible for designing and implementing streaming data pipelines that integrate Kafka with Databricks using Structured Streaming. You will also be tasked with architecting and maintaining the Medallion Architecture, which consists of well-defined Bronze, Silver, and Gold layers. Additionally, you will need to implement efficient data ingestion processes using Databricks Autoloader for high-throughput data loads. You will work with large volumes of structured and unstructured data to ensure high availability and performance, applying performance tuning techniques like partitioning, caching, and cluster resource optimization. Collaboration with cross-functional t...

You have an exciting opportunity to join a dynamic and high-impact firm as a Python, PySpark, and SQL Developer with 8-12 years of relevant experience. In this role, you will be responsible for working on development activities, collaborating with cross-functional teams, and designing scalable data pipelines using Python and PySpark. You will also be involved in implementing ETL processes, developing Power BI reports and dashboards, and optimizing data pipelines for performance and reliability. The ideal candidate should have 8+ years of experience in Spark, Scala, and PySpark for big data processing. Proficiency in Python programming for data manipulation and analysis is essential, along wi...

NTT DATA is looking for an Azure Databricks Developer to join their team in Bengaluru, Karnataka, India. As an Azure Databricks Developer, you will be responsible for advanced experience with Databricks, including Databricks SQL, DataFrames, and Spark using PySpark or Scala. You should have a deep understanding of Spark architecture and optimization strategies, such as tuning Spark SQL configurations, managing data partitions, handling data skew, and leveraging broadcast joins. Your role will also involve proficiency in building and optimizing large-scale data pipelines for ETL/ELT processes using Databricks and familiarity with Delta Lake and data lake architectures. Strong programming skil...

You are a PySpark Developer with over 7 years of experience and expertise in Reltio MDM. As a part of the Data Engineering team, your primary responsibility will be to design, build, and optimize scalable data pipelines, ensuring seamless integration with Reltio's cloud-native MDM platform. This is an immediate requirement, and we are looking for candidates who can join us promptly. Your key responsibilities will include designing, developing, and maintaining scalable data pipelines using PySpark in distributed computing environments like AWS EMR and Databricks. You will also be responsible for integrating and synchronizing data between enterprise systems and the Reltio MDM platform, impleme...

About the Company At Tide, we are dedicated to creating a business management platform that aims to streamline operations for small businesses, enabling them to save valuable time and resources. Our services include offering business accounts, banking solutions, as well as a range of integrated administrative tools spanning from invoicing to accounting. Established in 2017, Tide has garnered a user base of over 1 million small businesses globally, catering to SMEs in the UK, India, and Germany. Headquartered in central London, we also have offices in Sofia, Hyderabad, Delhi, Berlin, and Belgrade, with a team of more than 2,000 employees. Tide is on a trajectory of rapid growth, continuously ...

Join us as a Data Engineer responsible for supporting the successful delivery of Location Strategy projects to plan, budget, agreed quality and governance standards. You'll spearhead the evolution of our digital landscape, driving innovation and excellence. You will harness cutting-edge technology to revolutionise our digital offerings, ensuring unparalleled customer experiences. To be successful as a Data Engineer, you should have experience with hands-on experience in pyspark and strong knowledge of Dataframes, RDD, and SparkSQL. Additionally, experience in developing, testing, and maintaining applications on AWS Cloud is crucial. A strong hold on the AWS Data Analytics Technology Stack (G...

Wissen Technology is hiring for Python+ Pyspark About Wissen Technology: At Wissen Technology, we deliver niche, custom-built products that solve complex business challenges across industries worldwide. Founded in 2015, our core philosophy is built around a strong product engineering mindsetensuring every solution is architected and delivered right the first time. Today, Wissen Technology has a global footprint with 2000+ employees across offices in the US, UK, UAE, India, and Australia. Our commitment to excellence translates into delivering 2X impact compared to traditional service providers. How do we achieve this Through a combination of deep domain knowledge, cutting-edge technology exp...