47 Dataframes Jobs

Role & responsibilities : Data Pipeline Development & Management Design and implement robust, scalable ETL/ELT pipelines using Databricks and Apache Spark Process large volumes of structured and unstructured data Develop and maintain data workflows using Databricks workflows, Apache Airflow, or similar orchestration tools Optimize data processing jobs for performance, cost efficiency, and reliability Implement incremental data processing patterns and change data capture (CDC) mechanisms Databricks Platform Engineering Build and maintain Delta Lake tables and implement medallion architecture (bronze, silver, gold layers) Develop streaming data pipelines using Structured Streaming and Delta Li...

As a Software Developer at Revrr Technologies, you will be responsible for designing, developing, and maintaining high-performance, scalable data processing applications using Scala and Apache Spark. You will collaborate with cross-functional teams to define, design, and ship new features. Your role will also involve optimizing and refactoring existing applications to improve efficiency, reliability, and performance. Ensuring the quality of code through unit testing, integration testing, and code reviews will be a key part of your responsibilities. Additionally, you will troubleshoot and resolve complex issues in both production and non-production environments. As a mentor, you will work wit...

. Responsibilities: *Design, develop, and maintain data pipelines using Spark, Python, Scala, and Java. *Write efficient and optimized SQL queries for data extraction, transformation, and loading (ETL) processes. *Work with Data Frames to manipulate and analyse large datasets. *Implement data storage and processing solutions using cloud technologies (preferably AWS or GCP). *Build and maintain real-time data streaming pipelines using MSK/Kafka. *Utilize S3 for data storage and retrieval. *Work with data lake technologies like Iceberg. *Ensure data quality, integrity, and security. *Collaborate with data scientists and other engineers to understand data requirements and deliver solutions. *Pa...

Join us as a Data Engineer - PySpark and AWS responsible for supporting the successful delivery of Location Strategy projects to plan, budget, agreed quality and governance standards. You'll spearhead the evolution of our digital landscape, driving innovation and excellence. You will harness cutting-edge technology to revolutionise our digital offerings, ensuring unparalleled customer experiences. To be successful as a Data Engineer - PySpark and AWS you should have experience with: Hands on experience in pyspark and strong knowledge on Dataframes, RDD and SparkSQL Hands on Experience in developing, testing and maintaining applications on AWS Cloud. Strong hold on AWS Data Analytics Technolo...

Responsibilities: * Design, develop, and maintain data pipelines using Spark, Scala, and Java. * Write efficient and optimized SQL queries for data extraction, transformation, and loading (ETL) processes. * Work with DataFrames to manipulate and analyze large datasets. * Implement data storage and processing solutions using cloud technologies (preferably AWS or GCP). * Build and maintain real-time data streaming pipelines using MSK/Kafka. * Utilize S3 for data storage and retrieval. * Work with data lake technologies like Iceberg. * Ensure data quality, integrity, and security. * Collaborate with data scientists and other engineers to understand data requirements and deliver solutions. * Par...

At Enkefalos Technologies LLP., you will be part of a supportive and inclusive environment where innovation thrives. You will collaborate with industry experts who are passionate about AI and next-generation tech solutions. Working here offers opportunities for career growth, continuous learning, and involvement in exciting projects that challenge you to push boundaries. If you are ready to start a rewarding career in AI and tech, explore our current job opening for the position of Databricks Engineer - Spark / PySpark. Join our team that is driving change through advanced GenAI solutions and together, we can shape the future of industries worldwide. **Role Overview:** As a Databricks Engine...

Job Description: As a Spark Expert, you will be required to demonstrate expert proficiency in Spark and possess the ability to design and implement efficient data processing workflows. Your experience with Spark SQL and DataFrames, along with a good exposure to Big Data architectures and understanding of the Big Data ecosystem, will be crucial. Additionally, your expertise in framework building on Hadoop, strong DB knowledge with SQL tuning experience, and familiarity with Python APIs and exposure to Kafka will be valuable assets. Key Responsibilities: - Demonstrate expert proficiency in Spark - Design and implement efficient data processing workflows - Utilize Spark SQL and DataFrames effec...

As a member of the IT Services Organization, your role will involve developing and maintaining applications using Python and Azure services. You will collaborate with cross-functional teams to define, design, and ship new features. Utilizing frameworks such as Django and Flask for application development will be a key part of your responsibilities. Additionally, working with data processing tools including DataFrames, PySpark, and SQL, as well as managing data storage solutions using ADLS and Unity Catalog, will be essential tasks. Implementing Agile methodologies in project management and staying updated with advancements in Generative AI to integrate relevant technologies are crucial aspec...

Job Description: Purpose of the role To enable data-driven strategic and operational decision making through extracting actionable insights from large datasets, performing statistical and advanced analytics to uncover trends and patterns, and presenting findings through clear visualisations and reports. Accountabilities Investigation and analysis of data issues related to quality, lineage, controls, and authoritative source identification, documenting data sources, methodologies, and quality findings with recommendations for improvement. Designing and building data pipelines to automate data movement and processing. Apply advanced analytical techniques to large datasets to uncover trends and...

Skills : Data Engineer / Data Analyst Related. Position : BA4 / AVP (Developer / Senior Developer / Technical Lead / Team Lead / Individual Contributor) . Work Experience : 5.00 Years to 15.00 Years Work Location : only Pune Kharadi & Bangalore Location Job Type : Permanent Employee (Direct Payroll) This is for TOP Investment Bank Opening in Pune & Bangalore Location - If you are Interested then please share me your updated word format CV with Mention All the details. Below mentioned One details is Mandatory for Uploading your resume : Aadhar Number : Driving Licence : Passport Number : Current Location : Preferred Location: Total Experience: Relevant Experience : Primary Active Personal Ema...

Azure Data Factory: - Develop Azure Data Factory Objects - ADF pipeline, configuration, parameters, variables, Integration services runtime - Hands-on knowledge of ADF activities(such as Copy, SP, lkp etc) and DataFlows - ADF data Ingestion and Integration with other services Azure Databricks: - Experience in Big Data components such as Kafka, Spark SQL, Dataframes, HIVE DB etc implemented using Azure Data Bricks would be preferred. - Azure Databricks integration with other services - Read and write data in Azure Databricks - Best practices in Azure Databricks Synapse Analytics: - Import data into Azure Synapse Analytics with and without using PolyBase - Implement a Data Warehouse with Azure...

Azure Data Factory: - Develop Azure Data Factory Objects - ADF pipeline, configuration, parameters, variables, Integration services runtime - Hands-on knowledge of ADF activities(such as Copy, SP, lkp etc) and DataFlows - ADF data Ingestion and Integration with other services Azure Databricks: - Experience in Big Data components such as Kafka, Spark SQL, Dataframes, HIVE DB etc implemented using Azure Data Bricks would be preferred. - Azure Databricks integration with other services - Read and write data in Azure Databricks - Best practices in Azure Databricks Synapse Analytics: - Import data into Azure Synapse Analytics with and without using PolyBase - Implement a Data Warehouse with Azure...

As a Data Engineer, your role will involve designing and implementing streaming data pipelines that integrate Kafka with Databricks using Structured Streaming. You will be responsible for architecting and maintaining the Medallion Architecture with well-defined Bronze, Silver, and Gold layers. Your tasks will include implementing efficient ingestion using Databricks Autoloader for high-throughput data loads and working with large volumes of structured and unstructured data to ensure high availability and performance. Additionally, you will apply performance tuning techniques such as partitioning, caching, and cluster resource optimization. Collaboration with cross-functional teams including ...

Location - Chennai, Bengaluru, Hyderabad, Pune Job Description : Python Developer Experience : 12+ Location : Any / Remote Overview: We are looking for a Python Developer with strong experience in Pandas and Object-Oriented Programming (OOP) to build data processing workflows, automation scripts, and backend components. Key Responsibilities Develop and maintain Python applications using OOP principles. Build data pipelines and perform data manipulation using Pandas. Implement ETL logic, data cleansing, and transformation using DataFrames. Integrate Python applications with SQL/NoSQL databases. Write reusable, efficient, and well-structured code. Create unit tests, documentation, and support ...

As a Data Engineer at our company, you will have the following responsibilities: - Review and understand business requirements to ensure development tasks are completed within the provided timeline and are thoroughly tested with minimal defects. - Collaborate with a software development team to implement best practices and optimize the performance of Data applications. - Engage with various teams across the company and interact with customers to define, design, and showcase new concepts and solutions. - Conduct research on new Big Data technologies, evaluating their maturity and alignment with business and technology strategies. - Work closely with other developers to meet client needs effic...

Join us as a Service Operations Manager at Barclays, where you'll take part in the evolution of our digital landscape, driving innovation and excellence. You'll harness cutting-edge technology to revolutionise our digital offerings, ensuring unparalleled customer experiences. As a part of the Service Operations team, you will deliver technology stack, using strong analytical and problem solving skills to understand the business requirements and deliver quality solutions. You'll be working on complex technical problems that will involve detailed analytical skills and analysis. This will be done in conjunction with fellow engineers, business analysts and business stakeholders. To be successful...

Role & responsibilities : Data Pipeline Development & Management Design and implement robust, scalable ETL/ELT pipelines using Databricks and Apache Spark Process large volumes of structured and unstructured data Develop and maintain data workflows using Databricks workflows, Apache Airflow, or similar orchestration tools Optimize data processing jobs for performance, cost efficiency, and reliability Implement incremental data processing patterns and change data capture (CDC) mechanisms Databricks Platform Engineering Build and maintain Delta Lake tables and implement medallion architecture (bronze, silver, gold layers) Develop streaming data pipelines using Structured Streaming and Delta Li...

As a Data Engineer - Pyspark at Barclays, you will be spearheading the evolution of the digital landscape, driving innovation and excellence. You will be utilizing cutting-edge technology to revolutionize digital offerings, ensuring unparalleled customer experiences. As part of a team of developers, you will be responsible for delivering the technology stack, using strong analytical and problem-solving skills to understand business requirements and deliver quality solutions. **Key Responsibilities:** - Hands-on experience in Pyspark and strong knowledge of Dataframes, RDD, and SparkSQL. - Hands-on experience in Pyspark performance optimization techniques. - Hands-on experience in developing,...

As a candidate for this role, you would be responsible for the following: - Understanding of Spark core concepts like RDDs, DataFrames, DataSets, SparkSQL, and Spark Streaming. - Experience with Spark optimization techniques. - Deep knowledge of Delta Lake features like time travel, schema evolution, data partitioning. - Ability to design and implement data pipelines using Spark and Delta Lake as the data storage layer. - Proficiency in Python/Scala/Java for Spark development and integrate with ETL process. - Knowledge of data ingestion techniques from various sources (flat files, CSV, API, database). - Understanding of data quality best practices and data validation techniques. Qualificatio...

As an experienced candidate with over 8 years of overall experience and 5+ years of relevant experience, your role will involve the following key responsibilities: - Strong understanding of Spark core including RDDs, DataFrames, DataSets, SparkSQL, and Spark Streaming - Proficiency in Delta Lake features such as time travel, schema evolution, and data partitioning - Designing and building data pipelines using Spark and Delta Lake - Solid experience in Python/Scala/Java for Spark development - Knowledge of data ingestion from files, APIs, and databases - Familiarity with data validation and quality best practices - Working knowledge of data warehouse concepts and data modeling - Hands-on expe...

As a skilled PySpark Developer with hands-on experience in Reltio MDM, your role will involve designing and implementing scalable data processing solutions using PySpark and integrating with Reltio's cloud-native MDM platform. Key Responsibilities: - Develop and maintain data pipelines using PySpark in distributed computing environments such as AWS EMR and Databricks. - Integrate and synchronize data between enterprise systems and the Reltio MDM platform. - Design and implement data transformation, cleansing, and enrichment processes. - Collaborate with data architects, business analysts, and Reltio solution architects to ensure high-quality data modeling. - Work on API-based integration bet...

Join us as a Data Engineer at Barclays, where you will spearhead the evolution of our infrastructure and deployment pipelines, driving innovation and operational excellence. You will harness cutting-edge technology to build and manage robust, scalable and secure infrastructure, ensuring seamless delivery of our digital solutions. To be successful as a Data Engineer, you should have experience with: Hands on experience in pyspark and strong knowledge on Dataframes, RDD and SparkSQL. Hands on Experience in developing, testing and maintaining applications on AWS Cloud. Strong hold on AWS Data Analytics Technology Stack (Glue, S3, Lambda, Lake formation, Athena). Design and implement scalable an...

Role Overview: You will be responsible for developing Python back end solutions with GenAI exposure. Your key responsibilities will include: - Demonstrating strong experience in Python programming, specifically with Django and Flasks frameworks - Utilizing dataframes, Pyspark, SQL, and Databricks SQL for effective data processing - Working with ADLS Unity Catalog for data storage and management - Applying Agile methodology to software development projects - Leveraging GenAI for enhanced AI capabilities - Operating in an Azure DevOps environment for efficient project management Qualification Required: You should possess the following qualifications: - 5+ years of experience in Python developm...

As an FDR/CVR Data Analyst, you will play a critical role in ensuring the safety and efficiency of aircraft operations by performing in-depth analysis of flight data. You will work with large datasets from flight recorders to monitor aircraft health and support return-to-service processes. Collaborating with various teams, you will interpret data, develop reports, and ensure compliance with industry standards. Additionally, you will contribute to the development of new data analysis methodologies and tools, provide technical expertise, and maintain thorough documentation. - Perform detailed readouts and analysis of FDR and CVR data to support aircraft health monitoring and return-to-service ...

As a Data Engineer specializing in Spark & Delta Lake, your role will involve understanding key Spark concepts such as RDDs, DataFrames, DataSets, SparkSQL, and Spark Streaming. You will also be expected to have experience in optimizing Spark performance and a deep understanding of Delta Lake features like time travel, schema evolution, and data partitioning. Your responsibilities will include designing and implementing data pipelines using Spark and Delta Lake as the data storage layer. Proficiency in Python, Scala, or Java for Spark development and integration with ETL processes is essential. Additionally, you should possess knowledge of data ingestion techniques from various sources such ...