170 Spark Sql Jobs - Page 5

You are an experienced PySpark ETL Lead responsible for driving data integration efforts in analytics and data warehousing projects. Your role includes developing and managing PySpark scripts, creating ETL workflows, and ensuring efficient data processing and integration across systems. You should be strong in PySpark and Python for ETL and data processing, with experience in ETL pipeline design, data integration, and data warehousing. Proficiency in SQL and working with large datasets is required, along with familiarity with workflow schedulers such as Airflow and Cron. Hands-on experience with Big Data tools like Hive, HDFS, and Spark SQL is essential. It would be beneficial if you have ex...

As a Senior Azure Data Engineer, your responsibilities will include: Building scalable data pipelines using Databricks and PySpark Transforming raw data into usable business insights Integrating Azure services like Blob Storage, Data Lake, and Synapse Analytics Deploying and maintaining machine learning models using MLlib or TensorFlow Executing large-scale Spark jobs with performance tuning on Spark Pools Leveraging Databricks Notebooks and managing workflows with MLflow Qualifications: Bachelors/Masters in Computer Science, Data Science, or equivalent 7+ years in Data Engineering, with 3+ years in Azure Databricks Strong hands-on in: PySpark, Spark SQL, RDDs, Pandas, NumPy, Delta Lake Azur...

We are seeking an experienced PySpark Developer / Data Engineer to design, develop, and optimize big data processing pipelines using Apache Spark and Python (PySpark). The ideal candidate should have expertise in distributed computing, ETL workflows, data lake architectures, and cloud-based big data solutions. Key Responsibilities: Develop and optimize ETL/ELT data pipelines using PySpark on distributed computing platforms (Hadoop, Databricks, EMR, HDInsight). Work with structured and unstructured data to perform data transformation, cleansing, and aggregation. Implement data lake and data warehouse solutions on AWS (S3, Glue, Redshift), Azure (ADLS, Synapse), or GCP (BigQuery, Dataflow). Op...

About the Roe: Grade Leve (for interna use): 11 The Team: Our team is responsibe for the design, architecture, and deveopment of our cient facing appications using a variety of toos that are reguary updated as new technoogies emerge. You wi have the opportunity every day to work with peope from a wide variety of backgrounds and wi be abe to deveop a cose team dynamic with coworkers from around the gobe. The Impact: The work you do wi be used every singe day, its the essentia code you write that provides the data and anaytics required for crucia, daiy decisions in the capita and commodities markets. Whats in it for you: Buid a career with a goba company. Work on code that fues the goba financ...

You are a PySpark Developer with over 7 years of experience and expertise in Reltio MDM. As a part of the Data Engineering team, your primary responsibility will be to design, build, and optimize scalable data pipelines, ensuring seamless integration with Reltio's cloud-native MDM platform. This is an immediate requirement, and we are looking for candidates who can join us promptly. Your key responsibilities will include designing, developing, and maintaining scalable data pipelines using PySpark in distributed computing environments like AWS EMR and Databricks. You will also be responsible for integrating and synchronizing data between enterprise systems and the Reltio MDM platform, impleme...

Role & responsibilities: Key Responsibilities: Design, develop, and optimize Java-based backend services (Spring Boot / Microservices) for API integrations. Develop and maintain Spark SQL queries and data processing pipelines for large-scale data ingestion. Build Spark batch and streaming jobs to land raw data from multiple vendor APIs into data lakes or warehouses. Implement robust error handling, logging, and monitoring for data pipelines. Collaborate with cross-functional teams across geographies to define integration requirements and deliverables. Troubleshoot and optimize Spark SQL for performance and cost efficiency. Participate in Agile ceremonies, daily standups, and client discussio...

Required Skills: Design and develop data pipelines using Azure Data Factory, Spark SQL, PySpark, and Python Implement ETL/ELT processes for both structured and unstructured data Optimize queries and data models in Azure Synapse Analytics and OneLake Build and maintain Lakehouse and Datawarehouse architectures using Microsoft Fabric Develop interactive dashboards and reports in Power BI (DirectLake mode) Integrate Microsoft Purview for data governance and compliance Work with DevOps teams to automate deployment of Fabric assets via Azure DevOps and CI/CD pipelines Ensure data security and performance tuning for large-scale datasets Role & responsibilities

As a software developer, you will be working in a constantly evolving environment driven by technological advances and the strategic direction of the organization you are employed by. Your primary responsibilities will include creating, maintaining, auditing, and enhancing systems to meet specific needs, often based on recommendations from systems analysts or architects. You will be tasked with testing both hardware and software systems to identify and resolve system faults. Additionally, you will be involved in writing diagnostic programs and designing and developing code for operating systems and software to ensure optimal efficiency. In situations where necessary, you will also provide re...

As a candidate for the role, you are expected to have a thorough understanding of collaboration in software development teams and shared software repositories and their concepts. Additionally, you should possess experience with Jira for effective project management. You should also have practical experience with at least one of the following tools for reporting and analysis: SSIS, SSAS, SSRS, or Power BI. Your expertise in using these tools will be crucial for generating insights and reports. In terms of Azure Data Factory and Synapse pipelines development, you are required to have a comprehensive understanding of how Data Factory pipelines function and their integration with other Azure ser...

As a Principal Architect - Data & Cloud at Quantiphi, you will bring your 14-20 years of experience in Technical, Solutioning, and Analytical roles to lead the way in architecting, designing, and implementing end-to-end data pipelines and data integration solutions for structured and unstructured data sources and targets. With a focus on Cloud platforms such as GCP, AWS, and Azure, you will be responsible for building and managing Data Lakes, Data Warehouse, Data Integration, Data Migration, and Business Intelligence/Artificial Intelligence solutions. Your role will involve understanding business requirements and translating them into functional and non-functional areas, defining boundaries ...

About the Company At Tide, we are dedicated to creating a business management platform that aims to streamline operations for small businesses, enabling them to save valuable time and resources. Our services include offering business accounts, banking solutions, as well as a range of integrated administrative tools spanning from invoicing to accounting. Established in 2017, Tide has garnered a user base of over 1 million small businesses globally, catering to SMEs in the UK, India, and Germany. Headquartered in central London, we also have offices in Sofia, Hyderabad, Delhi, Berlin, and Belgrade, with a team of more than 2,000 employees. Tide is on a trajectory of rapid growth, continuously ...

As a Spark and Scala Developer at Infosys, you will play a crucial role in facilitating digital transformation for our clients within a global delivery model. Your responsibilities will include conducting independent research on technologies, providing recommendations for suitable solutions, and contributing to technology-specific best practices and standards. It will be essential for you to effectively interact with key stakeholders and utilize your technical expertise across various stages of the Software Development Life Cycle. As part of our learning culture, teamwork and collaboration are highly encouraged, excellence is acknowledged, and diversity is both respected and valued. Required...

As a Senior AWS Data Engineer Cloud Data Platform at Teamware Solutions, a division of Quantum Leap Consulting Pvt. Ltd, located in Bangalore, you will be responsible for end-to-end implementation of Cloud data engineering solutions like Enterprise Data lake and Data hub in AWS. Working onsite in an office environment for 5 days a week, you will collaborate with the Offshore Manager and Onsite Business Analyst to understand the requirements and deliver scalable, distributed, cloud-based enterprise data solutions. You should have a strong background in AWS cloud technology, with 4-8 years of hands-on experience. Proficiency in architecting and delivering highly scalable solutions is a must, a...

As a Senior Data Engineer, you will be an integral part of our team, contributing to the design, implementation, and maintenance of scalable data pipelines utilizing cutting-edge tools like Airflow, PySpark, and Databricks. Your responsibilities will extend to supporting machine learning models in a production environment, crafting data products from vast datasets, and engaging with diverse teams to grasp data requirements effectively. Your primary duties will include designing and deploying robust data pipelines by leveraging Airflow, PySpark, and Databricks. You will also be tasked with ensuring the reliability and performance of machine learning models in production, creating data product...

Join us as a Principal Engineer in a challenging role where you will be responsible for designing and engineering software with a primary focus on customer or user experience. You will actively contribute to our architecture, design, and engineering center of excellence, collaborating to enhance the bank's overall software engineering capability. This role offers valuable stakeholder exposure, allowing you to build and leverage relationships while honing your technical talents. The position is available at the vice president level. As a Principal Engineer, your primary responsibilities will include creating exceptional customer outcomes through innovative engineering solutions to both existi...

NTT DATA is looking for a Databricks Engineer to join their team in Bangalore, Karnataka, India. As a Databricks Engineer, you will be responsible for various tasks related to data extraction, ETL pipeline modernization, job design, development, automation, metadata management, documentation, testing, collaboration, performance tuning, security, governance, and compliance. Your primary job duties will include extracting and analyzing data from SQL Server and Teradata sources, translating legacy SQL/DataStage transformations into Databricks-native code, building and orchestrating jobs within Databricks using tools like Databricks Workflows, Delta Lake, and Auto Loader, generating and maintain...

Join us as a Principal Engineer. This challenging role will involve designing and engineering software with a primary focus on customer or user experience. You will actively contribute to our architecture, design, and engineering center of excellence, collaborating to enhance the bank's software engineering capability. This role, offered at the vice president level, provides valuable stakeholder exposure, the opportunity to build and leverage relationships, and a chance to refine your technical skills. As a Principal Engineer, your responsibilities will include creating exceptional customer outcomes through innovative engineering solutions for both existing and new challenges. You will work ...

As a Senior Data Engineer, you will be responsible for designing and developing scalable data pipelines and notebooks using Microsoft Fabric or Synapse Analytics. You should have a research-oriented mindset in Data Projects, thinking outside of the box and focusing on future needs. Your role will involve building and managing Lakehouses and Data Warehouses using Fabrics OneLake architecture, integrating data from diverse sources into Fabric, and collaborating with BI developers for seamless integration with Power BI and other reporting tools. Additionally, you will be required to implement data governance, security, and compliance within the Fabric ecosystem, optimize data storage and proces...

Are you intellectually curious and passionate about promoting solutions across organizational boundaries Join the Consumer & Community Banking (CCB) Stress Testing Transformation team for a dynamic opportunity to design and build creative solutions for the future of stress testing and annual CCAR exercises. As a Senior Associate in the Stress Testing Transformation Solution team, you will be a strategic thinker who is passionate about designing and building creative solutions for the future of Stress Testing. You will spend your time solving complex problems, demonstrating strategic thought leadership, and designing the way our stakeholders operate. By leveraging a deep understanding of CCB ...

As a Senior Engineer at Impetus Technologies, you will play a crucial role in designing, developing, and deploying scalable data processing applications using Java and Big Data technologies. Your responsibilities will include collaborating with cross-functional teams, mentoring junior engineers, and contributing to architectural decisions to enhance system performance and scalability. Your key responsibilities will revolve around designing and maintaining high-performance applications, implementing data ingestion and processing workflows using frameworks like Hadoop and Spark, and optimizing existing applications for improved performance and reliability. You will also be actively involved in...

Join us as a Senior Automation QA at Barclays, where you will be responsible for supporting the successful delivery of location strategy projects to plan, budget, agreed quality and governance standards. Spearhead the evolution of the digital landscape, driving innovation and excellence by harnessing cutting-edge technology to revolutionize digital offerings, ensuring unparalleled customer experiences. To be successful in this role, you should have experience with Spark SQL, Python/Pyspark scripting, and ETL concepts. Additionally, skills in AWS exposure and Jupyter Notebook are highly valued. Your key critical skills relevant for success in this role may be assessed, such as risk and contro...

As a Senior Data Engineering Architect at Iris Software, you will play a crucial role in leading enterprise-level data engineering projects on public cloud platforms like AWS, Azure, or GCP. Your responsibilities will include engaging with client managers to understand their business needs, conceptualizing solution options, and finalizing strategies with stakeholders. You will also be involved in team building, delivering Proof of Concepts (PoCs), and enhancing competencies within the organization. Your role will focus on building competencies in Data & Analytics, including Data Engineering, Analytics, Data Science, AI/ML, and Data Governance. Staying updated with the latest tools, best prac...

We are looking for a highly skilled and experienced Senior Python & ML Engineer with expertise in PySpark, machine learning, and large language models (LLMs). You will play a key role in designing, developing, and implementing scalable data pipelines, machine learning models, and LLM-powered applications. In this role, you will need to have a solid understanding of Python's ecosystem, distributed computing using PySpark, and practical experience in AI optimization. Your responsibilities will include designing and maintaining robust data pipelines with PySpark, optimizing PySpark jobs for efficiency on large datasets, and ensuring data integrity throughout the pipeline. You will also be invol...

BizViz is a company that offers a comprehensive view of a business's data, catering to various industries and meeting the diverse needs of business executives. With a dedicated team of over 50 professionals working on the BizViz platform for several years, the company aims to develop technological solutions that provide our clients with a competitive advantage. At BizViz, we are committed to the success of our customers, striving to create applications that align with their unique visions and requirements. We steer clear of generic ERP templates, offering businesses a more tailored solution. As a Big Data Engineer at BizViz, you will join a small, agile team of data engineers focused on buil...

As a Lead database engine developer, you will play a crucial role in enhancing our database engine to operate at an exabyte scale. Our analytical database engine processes trillions of data points daily, enabling rapid queries with an impressive 60 ms response time at P50. Your technical expertise and leadership will be pivotal in ensuring that our system seamlessly manages exabytes of data on a daily basis. Your responsibilities will include developing and executing innovative technical strategies for our database engine that align with Newrelic's business objectives. You will focus on optimizing scalability and performance to handle exabyte-scale data while maintaining exceptional query pe...