98 Spark Sql Jobs - Page 2

We have an exciting and rewarding opportunity for you to take your software engineering career to the next level. As a Software Engineer III at JPMorgan Chase within the Corporate Technology Consumer and Community Banking Risk Technology team, you serve as a seasoned member of an agile team to design and deliver trusted market-leading technology products in a secure, stable, and scalable way. You are responsible for carrying out critical technology solutions across multiple technical areas within various business functions in support of the firm's business objectives. You will be responsible for executing software solutions, design, development, and technical troubleshooting with the ability to think beyond routine or conventional approaches to build solutions or break down technical problems. Additionally, you will work with product managers, data scientists, ML engineers, and other stakeholders to understand requirements and develop API/services for model deployment, ensuring scalability, reliability, and efficiency. Building applications to automate manual steps in MLOPs pipeline, executing POC for innovative ideas to solve complex business problems, and staying informed about the latest trends and advancements in the latest LLM/GenAI research are also part of your responsibilities. Furthermore, you will add to the team culture of diversity, equity, inclusion, and respect. Required qualifications, capabilities, and skills include formal training or certification on software engineering concepts and 3+ years of applied experience, proficiency in programming languages like Python for model development and Python for BigData (i.e., Pyspark, Spark SQL on EMR experience). Experience in web applications/APIs with AWS Lambda, cloud gateway, experimentation, and integration with OpenAI API, building API and restful services using Flask/Django/FastAPI, solid understanding of agile methodologies like CI/CD, Application Resiliency, and Security, experience with cloud computing platforms (e.g., AWS, Azure, or Google Cloud Platform), containerization technologies (e.g., Docker and Kubernetes), and microservices design, implementation, and performance optimization, as well as understanding the fundamentals of machine learning and LLMs, and experience in applied AI/ML engineering, with a track record of deploying business-critical machine learning models in production. Preferred qualifications, capabilities, and skills include familiarity with modern front-end technologies and the financial services industries.,

You will be responsible for designing, developing, and maintaining graph database schemas using Neo4j. Your role will involve optimizing Neo4j queries to enhance performance and efficiency. Additionally, you will utilize Python, PySpark, or Spark SQL for data transformation and analysis. Conducting User Acceptance Testing (UAT) to ensure data accuracy and overall system functionality will be a part of your duties. You will also be involved in developing and managing scalable data pipelines using Databricks and Azure Data Factory (ADF) for efficient data pipeline management. Working with Azure cloud services and having familiarity with Azure data engineering components will be essential for cloud integration tasks. Desired Skills: - Strong experience with Neo4j and Cypher query language - Proficiency in Python and/or PySpark - Hands-on experience with Databricks and Azure Data Factory - Familiarity with data engineering tools and best practices - Good understanding of database performance tuning - Ability to work effectively in fast-paced, client-driven environments If you possess skills in Azure, data engineering tools, Neo4j, PySpark, Azure Data Factory, Spark SQL, Databricks, cloud services, database performance tuning, Cypher query language, and Python, this role could be a great fit for you.,

NTT DATA is looking for a Sr. Data Modeler to join their team in Bangalore, Karntaka (IN-KA), India. As a Sr. Data Modeler, your primary responsibility will be to design and implement dimensional (star/snowflake) and 3NF data models that are optimized for analytical and reporting needs in Azure Synapse and Power BI. You will also be required to perform STM (source to target mapping) from the data source to multiple layers in the data lake. Additionally, you will analyze and optimize Spark SQL queries and collaborate with cross-functional teams to ensure that the data models align with business requirements. The ideal candidate should have at least 7 years of experience in SQL and PySpark. Hands-on experience with Azure Synapse, ADLS, Delta format, and metadata-driven data pipelines is essential. You should also be experienced in implementing dimensional (star/snowflake) and 3NF data models, as well as in PySpark and Spark SQL, including query optimization and performance tuning. Experience in writing complex SQL, performing Source-to-Target Mapping (STM), and familiarity with CI/CD practices in Git and Azure DevOps are also required. In this role, you will be responsible for maintaining version control and CI/CD pipelines in Git and Azure DevOps, as well as integrating Azure Purview to enable access controls and implementing row-level security. Strong problem-solving and analytical skills for debugging and optimizing data pipelines in Azure Synapse are essential. If you are a passionate and innovative individual looking to be part of a forward-thinking organization, apply now to join NTT DATA and be a part of their inclusive and adaptable team dedicated to long-term success and digital transformation. Visit us at us.nttdata.com.,

You are an experienced PySpark ETL Lead responsible for driving data integration efforts in analytics and data warehousing projects. Your role includes developing and managing PySpark scripts, creating ETL workflows, and ensuring efficient data processing and integration across systems. You should be strong in PySpark and Python for ETL and data processing, with experience in ETL pipeline design, data integration, and data warehousing. Proficiency in SQL and working with large datasets is required, along with familiarity with workflow schedulers such as Airflow and Cron. Hands-on experience with Big Data tools like Hive, HDFS, and Spark SQL is essential. It would be beneficial if you have experience with cloud platforms like AWS, Azure, or GCP. Your responsibilities will involve leading ETL development using PySpark, designing and scheduling data workflows, optimizing data processing performance, and collaborating with cross-functional teams. If you have a passion for data integration, a knack for optimizing processes, and enjoy working in a collaborative environment, this role is perfect for you. Join us and be part of our dynamic team driving impactful data initiatives.,

As a Senior Azure Data Engineer, your responsibilities will include: Building scalable data pipelines using Databricks and PySpark Transforming raw data into usable business insights Integrating Azure services like Blob Storage, Data Lake, and Synapse Analytics Deploying and maintaining machine learning models using MLlib or TensorFlow Executing large-scale Spark jobs with performance tuning on Spark Pools Leveraging Databricks Notebooks and managing workflows with MLflow Qualifications: Bachelors/Masters in Computer Science, Data Science, or equivalent 7+ years in Data Engineering, with 3+ years in Azure Databricks Strong hands-on in: PySpark, Spark SQL, RDDs, Pandas, NumPy, Delta Lake Azure ecosystem: Data Lake, Blob Storage, Synapse Analytics Location: Remote- Bengaluru,Hyderabad,Delhi / NCR,Chennai,Pune,Kolkata,Ahmedabad,Mumbai

We are seeking an experienced PySpark Developer / Data Engineer to design, develop, and optimize big data processing pipelines using Apache Spark and Python (PySpark). The ideal candidate should have expertise in distributed computing, ETL workflows, data lake architectures, and cloud-based big data solutions. Key Responsibilities: Develop and optimize ETL/ELT data pipelines using PySpark on distributed computing platforms (Hadoop, Databricks, EMR, HDInsight). Work with structured and unstructured data to perform data transformation, cleansing, and aggregation. Implement data lake and data warehouse solutions on AWS (S3, Glue, Redshift), Azure (ADLS, Synapse), or GCP (BigQuery, Dataflow). Optimize PySpark jobs for performance tuning, partitioning, and caching strategies. Design and implement real-time and batch data processing solutions. Integrate data pipelines with Kafka, Delta Lake, Iceberg, or Hudi for streaming and incremental updates. Ensure data security, governance, and compliance with industry best practices. Work with data scientists and analysts to prepare and process large-scale datasets for machine learning models. Collaborate with DevOps teams to deploy, monitor, and scale PySpark jobs using CI/CD pipelines, Kubernetes, and containerization. Perform unit testing and validation to ensure data integrity and reliability. Required Skills & Qualifications: 6+ years of experience in big data processing, ETL, and data engineering. Strong hands-on experience with PySpark (Apache Spark with Python). Expertise in SQL, DataFrame API, and RDD transformations. Experience with big data platforms (Hadoop, Hive, HDFS, Spark SQL). Knowledge of cloud data processing services (AWS Glue, EMR, Databricks, Azure Synapse, GCP Dataflow). Proficiency in writing optimized queries, partitioning, and indexing for performance tuning. Experience with workflow orchestration tools like Airflow, Oozie, or Prefect. Familiarity with containerization and deployment using Docker, Kubernetes, and CI/CD pipelines. Strong understanding of data governance, security, and compliance (GDPR, HIPAA, CCPA, etc.). Excellent problem-solving, debugging, and performance optimization skills.

About the Roe: Grade Leve (for interna use): 11 The Team: Our team is responsibe for the design, architecture, and deveopment of our cient facing appications using a variety of toos that are reguary updated as new technoogies emerge. You wi have the opportunity every day to work with peope from a wide variety of backgrounds and wi be abe to deveop a cose team dynamic with coworkers from around the gobe. The Impact: The work you do wi be used every singe day, its the essentia code you write that provides the data and anaytics required for crucia, daiy decisions in the capita and commodities markets. Whats in it for you: Buid a career with a goba company. Work on code that fues the goba financia markets. Grow and improve your skis by working on enterprise eve products and new technoogies. Responsibiities: Sove probems, anayze and isoate issues.Provide technica guidance and mentoring to the team and hep them adopt change as new processes are introduced.Champion best practices and serve as a subject matter authority.Deveop soutions to deveop/support key business needs.Engineer components and common services based on standard deveopment modes, anguages and toosProduce system design documents and ead technica wakthroughsProduce high quaity codeCoaborate effectivey with technica and non-technica partnersAs a team-member shoud continuousy improve the architecture Basic Quaifications: 10-12 years of experience designing/buiding data-intensive soutions using distributed computing.Proven experience in impementing and maintaining enterprise search soutions in arge-scae environments.Experience working with business stakehoders and users, providing research direction and soution design and writing robust maintainabe architectures and APIs.Experience deveoping and depoying Search soutions in a pubic coud such as AWS.Proficient programming skis at a high-eve anguages -Java, Scaa, PythonSoid knowedge of at east one machine earning research frameworksFamiiarity with containerization, scripting, coud patforms, and CI/CD.5+ years experience with Python, Java, Kubernetes, and data and workfow orchestration toos4+ years experience with Easticsearch, SQL, NoSQL,Apache spark, Fink, Databricks and Mfow.Prior experience with operationaizing data-driven pipeines for arge scae batch and stream processing anaytics soutionsGood to have experience with contributing to GitHub and open source initiatives or in research projects and/or participation in Kagge competitionsAbiity to quicky, efficienty, and effectivey define and prototype soutions with continua iteration within aggressive product deadines.Demonstrate strong communication and documentation skis for both technica and non-technica audiences. Preferred Quaifications: Search TechnoogiesQuery and Indexing content for Apache Sor, Eastic Search, etc.Proficiency in search query anguages (e.g., Lucene Query Syntax) and experience with data indexing and retrieva.Experience with machine earning modes and NLP techniques for search reevance and ranking.Famiiarity with vector search techniques and embedding modes (e.g., BERT, Word2Vec).Experience with reevance tuning using A/B testing frameworks.Big Data TechnoogiesApache Spark, Spark SQL, Hadoop, Hive, AirfowData Science Search TechnoogiesPersonaization and Recommendation modes, Learn to Rank (LTR)Preferred LanguagesPython, JavaDatabase TechnoogiesMS SQL Server patform, stored procedure programming experience using Transact SQL.Abiity to ead, train and mentor. About S&P Goba Market Inteigence At S&P Goba Market Inteigence, a division of S&P Goba we understand the importance of accurate, deep and insightfu information. Our team of experts deivers unrivaed insights and eading data and technoogy soutions, partnering with customers to expand their perspective, operate with confidence, andmake decisions with conviction.For more information, visit . Whats In It For You Our Purpose: Progress is not a sef-starter. It requires a catayst to be set in motion. Information, imagination, peope, technoogythe right combination can unock possibiity and change the word.Our word is in transition and getting more compex by the day. We push past expected observations and seek out new eves of understanding so that we can hep companies, governments and individuas make an impact on tomorrow. At S&P Goba we transform data into Essentia Inteigence, pinpointing risks and opening possibiities. We Acceerate Progress.

You are a PySpark Developer with over 7 years of experience and expertise in Reltio MDM. As a part of the Data Engineering team, your primary responsibility will be to design, build, and optimize scalable data pipelines, ensuring seamless integration with Reltio's cloud-native MDM platform. This is an immediate requirement, and we are looking for candidates who can join us promptly. Your key responsibilities will include designing, developing, and maintaining scalable data pipelines using PySpark in distributed computing environments like AWS EMR and Databricks. You will also be responsible for integrating and synchronizing data between enterprise systems and the Reltio MDM platform, implementing data transformation processes, and collaborating with various stakeholders for effective data modeling. Additionally, you will work on API-based integrations between Reltio and other applications, optimize PySpark jobs for performance and cost-efficiency, and ensure data quality and governance across workflows. To excel in this role, you should possess at least 7 years of hands-on experience in PySpark development and distributed data processing. Strong expertise in Apache Spark, DataFrames, and Spark SQL is essential, along with proven experience in Reltio MDM, REST APIs, and working with JSON data formats. Experience with cloud platforms, particularly AWS (S3, Lambda, Step Functions, EMR), data warehousing concepts, ETL workflows, and data modeling is required. Familiarity with CI/CD pipelines, Git, and version control is also beneficial. Strong problem-solving, analytical, and communication skills are key attributes for this role. This opportunity offers you the chance to work on cutting-edge data engineering projects and gain exposure to Reltio MDM, a leading cloud-native MDM platform. You will have the flexibility to work from any location across India (PAN India), making it a convenient and enriching work environment for you.,

Role & responsibilities: Key Responsibilities: Design, develop, and optimize Java-based backend services (Spring Boot / Microservices) for API integrations. Develop and maintain Spark SQL queries and data processing pipelines for large-scale data ingestion. Build Spark batch and streaming jobs to land raw data from multiple vendor APIs into data lakes or warehouses. Implement robust error handling, logging, and monitoring for data pipelines. Collaborate with cross-functional teams across geographies to define integration requirements and deliverables. Troubleshoot and optimize Spark SQL for performance and cost efficiency. Participate in Agile ceremonies, daily standups, and client discussions. EXPERTISE AND QUALIFICATIONS Required Skills: 4 to 8 years of relevant experience. Core Java (Java 8 or above) with proven API development experience. Apache Spark (Core, SQL, DataFrame APIs) for large-scale data processing. Spark SQL strong ability to write and optimize queries for complex joins, aggregations, and transformations. Experience with API integration (RESTful APIs, authentication, payload handling, and rate limiting). Hands-on with data ingestion frameworks and ETL concepts. Experience with MySQL or other RDBMS for relational data management. Proficiency in Git for version control. Strong debugging, performance tuning, and problem-solving skills. Ability to work with minimal supervision in a short-term, delivery-focused engagement. Nice to Have: Experience with Apache Kafka for real-time streaming integrations. Familiarity with AWS data services (S3, EMR, Glue). Exposure to NoSQL databases like Cassandra or MongoDB.

Required Skills: Design and develop data pipelines using Azure Data Factory, Spark SQL, PySpark, and Python Implement ETL/ELT processes for both structured and unstructured data Optimize queries and data models in Azure Synapse Analytics and OneLake Build and maintain Lakehouse and Datawarehouse architectures using Microsoft Fabric Develop interactive dashboards and reports in Power BI (DirectLake mode) Integrate Microsoft Purview for data governance and compliance Work with DevOps teams to automate deployment of Fabric assets via Azure DevOps and CI/CD pipelines Ensure data security and performance tuning for large-scale datasets Role & responsibilities

As a software developer, you will be working in a constantly evolving environment driven by technological advances and the strategic direction of the organization you are employed by. Your primary responsibilities will include creating, maintaining, auditing, and enhancing systems to meet specific needs, often based on recommendations from systems analysts or architects. You will be tasked with testing both hardware and software systems to identify and resolve system faults. Additionally, you will be involved in writing diagnostic programs and designing and developing code for operating systems and software to ensure optimal efficiency. In situations where necessary, you will also provide recommendations for future developments. Joining us offers numerous benefits, including the opportunity to work on challenging projects and solve complex technical problems. You can expect rapid career growth and the chance to assume leadership roles. Our mentorship program allows you to learn from experienced mentors and industry experts, while our global opportunities enable you to collaborate with clients from around the world and gain international experience. We offer competitive compensation packages and benefits to our employees. If you are passionate about technology and interested in working on innovative projects with a skilled team, pursuing a career as an Infosys Power Programmer could be an excellent choice for you. To be considered for this role, you must possess the following mandatory skills: - Proficiency in AWS Glue, AWS Redshift/Spectrum, S3, API Gateway, Athena, Step, and Lambda functions. - Experience with Extract Transform Load (ETL) and Extract Load & Transform (ELT) data integration patterns. - Expertise in designing and constructing data pipelines. - Development experience in one or more object-oriented programming languages, preferably Python. In terms of job specifications, we are looking for candidates who meet the following criteria: - At least 5 years of hands-on experience in developing, testing, deploying, and debugging Spark Jobs using Scala in the Hadoop Platform. - Profound knowledge of Spark Core and working with RDDs and Spark SQL. - Familiarity with Spark Optimization Techniques and Best Practices. - Strong understanding of Scala Functional Programming concepts like Try, Option, Future, and Collections. - Proficiency in Scala Object-Oriented Programming covering Classes, Traits, Objects (Singleton and Companion), and Case Classes. - Sound knowledge of Scala Language Features including the Type System and Implicit/Givens. - Hands-on experience working in the Hadoop Environment (HDFS/Hive), AWS S3, EMR. - Proficiency in Python programming. - Working experience with Workflow Orchestration tools such as Airflow and Oozie. - Experience with API calls in Scala. - Familiarity and exposure to file formats like Apache AVRO, Parquet, and JSON. - Desirable knowledge of Protocol Buffers and Geospatial data analytics. - Ability to write test cases using frameworks like scalatest. - Good understanding of Build Tools such as Gradle & SBT. - Experience using GIT, resolving conflicts, and working with branches. - Preferred experience in workflow systems like Airflow. - Strong programming skills focusing on data structures and algorithms. - Excellent analytical and communication skills. Candidates applying for this position should have: - 7-10 years of industry experience. - A BE/B.Tech in Computer Science or an equivalent qualification.,

As a candidate for the role, you are expected to have a thorough understanding of collaboration in software development teams and shared software repositories and their concepts. Additionally, you should possess experience with Jira for effective project management. You should also have practical experience with at least one of the following tools for reporting and analysis: SSIS, SSAS, SSRS, or Power BI. Your expertise in using these tools will be crucial for generating insights and reports. In terms of Azure Data Factory and Synapse pipelines development, you are required to have a comprehensive understanding of how Data Factory pipelines function and their integration with other Azure services. Your ability to design and implement pipelines, as well as utilize Databricks and/or Synapse Delta Lake for BI purposes, will be essential. Furthermore, your role will involve Spark development, where you will need practical experience in Spark and understanding of its interaction within Synapse and Databricks Notebooks. Knowledge of Spark design, data structures, mpp flows, and expertise in Spark SQL usage within Databricks or Synapse Notebooks is also expected from you. Having an understanding of Kafka and its role in streaming is essential, along with knowledge of integrating Kafka with Spark and other persistence services. This is a permanent position requiring a minimum of 7 years of experience in Azure cloud and Git. Your expertise in these areas will be crucial for success in this role.,

As a Principal Architect - Data & Cloud at Quantiphi, you will bring your 14-20 years of experience in Technical, Solutioning, and Analytical roles to lead the way in architecting, designing, and implementing end-to-end data pipelines and data integration solutions for structured and unstructured data sources and targets. With a focus on Cloud platforms such as GCP, AWS, and Azure, you will be responsible for building and managing Data Lakes, Data Warehouse, Data Integration, Data Migration, and Business Intelligence/Artificial Intelligence solutions. Your role will involve understanding business requirements and translating them into functional and non-functional areas, defining boundaries in terms of Availability, Scalability, Performance, Security, and Resilience. You will leverage your expertise in various Data Integration and ETL technologies on Cloud, including Spark, Pyspark/Scala, Dataflow, DataProc, and more. Additionally, you will have the opportunity to work with traditional ETL tools like Informatica, DataStage, OWB, Talend, and others. Your deep knowledge of Cloud and On-Premise Databases such as Cloud SQL, Cloud Spanner, Big Table, RDS, Aurora, and others will be instrumental in architecting scalable data warehouse solutions on Cloud platforms like Big Query or Redshift. Moreover, your exposure to No-SQL databases and experience with data integration, storage, and data pipeline tool sets will be crucial in designing optimized data analytics solutions. Being a thought leader in architecture design and development of cloud data analytics solutions, you will collaborate with internal and external stakeholders to present solutions, support sales teams in building proposals, and lead discovery workshops with potential customers globally. Your role will also involve mentoring young talent, contributing to building Assets and Accelerators, and ensuring the successful delivery of projects on parameters of Schedule, Quality, and Customer Satisfaction. The position offers the experience of working in a high-growth startup in the AI, Decision Science, and Big Data Domain, along with the opportunity to be part of a diverse and proactive team that constantly raises the bar in translating data into tangible business value for clients. Additionally, flexible remote working options are available to foster productivity and work-life balance. If you are passionate about innovation, excellence, and growth, and enjoy working with a dynamic team of tech enthusiasts, Quantiphi is the place for you to shape your career in Data & Cloud architecture. Join us on our journey of digital transformation and be a part of creating impactful solutions that drive business success.,

About the Company At Tide, we are dedicated to creating a business management platform that aims to streamline operations for small businesses, enabling them to save valuable time and resources. Our services include offering business accounts, banking solutions, as well as a range of integrated administrative tools spanning from invoicing to accounting. Established in 2017, Tide has garnered a user base of over 1 million small businesses globally, catering to SMEs in the UK, India, and Germany. Headquartered in central London, we also have offices in Sofia, Hyderabad, Delhi, Berlin, and Belgrade, with a team of more than 2,000 employees. Tide is on a trajectory of rapid growth, continuously venturing into new markets and products, and continuously seeking individuals who are enthusiastic and motivated to join us in our mission to empower small businesses by aiding them in saving time and resources. About the Role We are in search of an experienced Senior Data Engineer with exceptional skills in PySpark to join our ML/Data engineering team. This team's responsibilities encompass feature development, data quality assessments, deployment, and integration of ML models with backend services, and enhancing the overall Tide platform. As a Senior Data Engineer, you will play a crucial role in designing, developing, and optimizing our upcoming data pipelines and platforms. Your tasks will involve working with extensive datasets, addressing intricate data challenges, and contributing to the creation of robust, scalable, and efficient data solutions that drive business value. This position presents an exciting opportunity for individuals who are passionate about big data technologies, performance optimization, and constructing resilient data infrastructure. As a Data Engineer, You Will: - Focus on Performance Optimization: Identify and resolve complex performance bottlenecks in PySpark jobs and Spark clusters, utilizing Spark UI, query plans, and advanced optimization techniques. - Lead Design & Development: Spearhead the design and implementation of scalable, fault-tolerant ETL/ELT pipelines using PySpark for batch and real-time data processing. - Collaborate on Data Modeling: Work alongside data scientists, analysts, and product teams to design efficient data models for analytical and operational use cases. - Ensure Data Quality & Governance: Implement strong data quality checks, monitoring, and alerting mechanisms to maintain data accuracy, consistency, and reliability. - Contribute to Architectural Decisions: Aid in shaping the data architecture strategy, assess new technologies, and implement best practices to enhance the data platform's capabilities. - Uphold Best Practices: Promote engineering best practices, participate in code reviews, and mentor junior data engineers. - Foster Collaboration: Work closely with cross-functional teams to deliver impactful data solutions. Qualifications: - Possess 8+ years of professional experience in data engineering, with a minimum of 4+ years focusing on PySpark development in a production environment. - Demonstrate expert-level proficiency in PySpark, including Spark SQL, DataFrames, RDDs, and understanding Spark's architecture. - Showcase hands-on experience in optimizing PySpark performance, debugging slow jobs, and handling common issues in large datasets. - Exhibit strong programming skills in Python, proficiency in SQL, and familiarity with data warehousing concepts. - Prior experience with distributed data storage solutions and version control systems. - Strong problem-solving abilities, attention to detail, and excellent communication skills. - Hold a Bachelor's or Master's degree in Computer Science, Engineering, or a related field. What We Offer: - Competitive salary - Health and life insurance for self and family - OPD benefits - Mental well-being support - Learning and development budget - WFH setup allowance - Generous leave policy - Stock options Tide Ways of Working: At Tide, we embrace a flexible workplace model that accommodates both in-person and remote work to cater to the diverse needs of our teams. While we support remote work, we believe in the importance of face-to-face interactions to foster collaboration and team spirit, making our offices hubs for innovation and community building. Tide is a Place for Everyone: We promote a transparent and inclusive environment where every voice is valued and heard. Your personal data will be handled by Tide for recruitment purposes in accordance with our Recruitment Privacy Notice.,

As a Spark and Scala Developer at Infosys, you will play a crucial role in facilitating digital transformation for our clients within a global delivery model. Your responsibilities will include conducting independent research on technologies, providing recommendations for suitable solutions, and contributing to technology-specific best practices and standards. It will be essential for you to effectively interact with key stakeholders and utilize your technical expertise across various stages of the Software Development Life Cycle. As part of our learning culture, teamwork and collaboration are highly encouraged, excellence is acknowledged, and diversity is both respected and valued. Required Qualifications: - Must be located within commuting distance of Raleigh, NC, Charlotte, NC, or Richardson, TX, or be open to relocating to these areas. - A Bachelor's degree or foreign equivalent from an accredited institution is required. Alternatively, three years of progressive experience in the specialty can be considered in place of each year of education. - All candidates authorized to work in the United States are welcome to apply. - Minimum of 4 years of experience in Information Technology. - Profound understanding of distributed computing principles and big data technologies. - At least 3 years of hands-on experience working with Apache Spark, Scala, Spark SQL, and Starburst. - Knowledge of data serialization formats like Parquet, Avro, or ORC. - Familiarity with data processing and transformation techniques. Preferred Qualifications: - Hands-on experience with data lakes, data warehouses, and ETL processes. - Solid comprehension of Agile software development frameworks. - Previous experience in the Banking domain. - Exceptional communication and analytical skills. - Ability to collaborate in teams within a diverse, multi-stakeholder environment involving Business and Technology teams. - Willingness and experience to work in a global delivery environment. This role may involve prolonged periods of sitting and computer work. Effective communication via telephone, email, or face-to-face interactions is essential. Travel might be necessary based on job requirements. About Us: Infosys is a renowned global leader in next-generation digital services and consulting. We assist clients in over 50 countries in navigating their digital transformation journey. With more than four decades of experience in managing the systems and operations of global enterprises, we expertly guide our clients through their digital evolution. By empowering enterprises with an AI-powered core to prioritize change execution and delivering agile digital solutions at scale, we aim to achieve exceptional levels of performance and customer satisfaction. Our commitment to continuous improvement is driven by an always-on learning agenda, enabling the transfer of digital skills, expertise, and innovative ideas from our thriving innovation ecosystem.,

As a Senior AWS Data Engineer Cloud Data Platform at Teamware Solutions, a division of Quantum Leap Consulting Pvt. Ltd, located in Bangalore, you will be responsible for end-to-end implementation of Cloud data engineering solutions like Enterprise Data lake and Data hub in AWS. Working onsite in an office environment for 5 days a week, you will collaborate with the Offshore Manager and Onsite Business Analyst to understand the requirements and deliver scalable, distributed, cloud-based enterprise data solutions. You should have a strong background in AWS cloud technology, with 4-8 years of hands-on experience. Proficiency in architecting and delivering highly scalable solutions is a must, along with expertise in Cloud data engineering solutions, Lambda or Kappa Architectures, Data Management concepts, and Data Modelling. You should be proficient in AWS services such as EMR, Glue, S3, Redshift, and DynamoDB, as well as have experience in Big Data frameworks like Hadoop and Spark. Additionally, you must have hands-on experience with AWS compute and storage services, AWS Streaming Services, troubleshooting and performance tuning in Spark framework, and knowledge of Application DevOps tools like Git and CI/CD Frameworks. Familiarity with AWS CloudWatch, Cloud Trail, Account Config, Config Rules, security, key management, data migration processes, and analytical skills is required. Good communication and presentation skills are essential for this role. Desired skills include experience in building stream-processing systems, Big Data ML toolkits, Python, Offshore/Onsite Engagements, flow tools like Airflow, Nifi or Luigi, and AWS services like STEP & Lambda. A professional background in BE/B.Tech/MCA/M.Sc/M.E/M.Tech/MBA is preferred, and an AWS certified Data Engineer certification is recommended. If you are interested in this position and meet the qualifications mentioned above, please send your resume to netra.s@twsol.com.,

As a Senior Data Engineer, you will be an integral part of our team, contributing to the design, implementation, and maintenance of scalable data pipelines utilizing cutting-edge tools like Airflow, PySpark, and Databricks. Your responsibilities will extend to supporting machine learning models in a production environment, crafting data products from vast datasets, and engaging with diverse teams to grasp data requirements effectively. Your primary duties will include designing and deploying robust data pipelines by leveraging Airflow, PySpark, and Databricks. You will also be tasked with ensuring the reliability and performance of machine learning models in production, creating data products that align with business objectives, and collaborating closely with cross-functional teams to address their data-related needs. Furthermore, you will play a key role in implementing process monitoring mechanisms to uphold data quality and reliability, including support for ACID transactions. Additionally, you will be expected to provide technical guidance and assistance to your colleagues whenever necessary. To qualify for this role, you should possess a Bachelor's or Master's degree in Computer Science, Engineering, or a related field, along with a minimum of 5 years of experience in a data engineering capacity. Your background should demonstrate a successful history of delivering scalable and dependable data solutions. Proficiency in Databricks, AWS, and Airflow ETL architectures, particularly in managing data streams from real-time consumer applications, is essential. Moreover, you should exhibit a strong command of Python, PySpark, and Spark SQL for data processing and analysis, coupled with a solid understanding of SQL databases and data modeling principles. Familiarity with Databricks Lakeview and other data visualization tools for monitoring and reporting purposes will be advantageous. Your ability to work autonomously and collaboratively within a team environment, combined with exceptional problem-solving and communication skills, will be critical for excelling in this role.,

Join us as a Principal Engineer in a challenging role where you will be responsible for designing and engineering software with a primary focus on customer or user experience. You will actively contribute to our architecture, design, and engineering center of excellence, collaborating to enhance the bank's overall software engineering capability. This role offers valuable stakeholder exposure, allowing you to build and leverage relationships while honing your technical talents. The position is available at the vice president level. As a Principal Engineer, your primary responsibilities will include creating exceptional customer outcomes through innovative engineering solutions to both existing and new challenges. You will design technology solutions that are customer-centric, innovative, high performance, secure, and robust. You will work closely with software engineers to produce and prototype innovative ideas, engage with domain and enterprise architects to validate and incorporate relevant architectures, and lead functional engineering teams. Additionally, you will manage end-to-end product implementations, drive demos, and engage stakeholders across platforms. Your role will involve designing and developing software with a focus on automating build, test, and deployment activities, while promoting the discipline of software engineering throughout the business. In addition to the above responsibilities, you will define, create, and oversee engineering and design solutions with a focus on end-to-end automation, simplification, resilience, security, performance, scalability, and reusability. You will collaborate with software engineers within a platform or feature team to design and engineer complex software, scripts, and tools that enable the delivery of bank platforms, applications, and services. You will also be responsible for defining architecture models and roadmaps for application and software components, driving common usability across products and domains. Furthermore, you will design, test, and implement working code, apply Agile methods to software development, and utilize DevOps techniques. To excel in this role, you should have significant experience in software engineering, software or database design, architecture, and working within a DevOps and Agile framework. You should possess an expert understanding of market trends, technologies, and tools, along with experience in implementing programming best practices related to scalability, automation, virtualization, optimization, availability, and performance. Additionally, hands-on experience with gathering business requirements, translating them into technical user stories, and leading functional solution design within the banking domain and CRM (MS Dynamics) is essential. Proficiency in PowerApps, D365 (including Custom Pages), Power BI (SQL, DAX, Power Query, Data Modeling, RLS, Azure, Lakehouse, Python, Spark SQL), and designing or implementing APIs is required. You should also demonstrate the ability to quickly understand and translate product and business requirements into technical solutions.,

NTT DATA is looking for a Databricks Engineer to join their team in Bangalore, Karnataka, India. As a Databricks Engineer, you will be responsible for various tasks related to data extraction, ETL pipeline modernization, job design, development, automation, metadata management, documentation, testing, collaboration, performance tuning, security, governance, and compliance. Your primary job duties will include extracting and analyzing data from SQL Server and Teradata sources, translating legacy SQL/DataStage transformations into Databricks-native code, building and orchestrating jobs within Databricks using tools like Databricks Workflows, Delta Lake, and Auto Loader, generating and maintaining data flow diagrams and job documentation, designing and executing unit tests and integration tests for data pipelines, optimizing data ingestion and transformation for performance and cost efficiency, ensuring compliance with data governance policies, and implementing access control via Unity Catalog. To be successful in this role, you must have a strong understanding of ETL/ELT principles and data pipelines, proficiency with Databricks platform and PySpark or Spark SQL, advanced SQL skills, familiarity with Teradata and SQL Server environments, ability to read and understand data models, schemas, and ERDs, basic proficiency with Git for code versioning, ability to write and validate unit/integration tests, strong communication skills, and an awareness of security and governance principles. NTT DATA is a global innovator of business and technology services, serving 75% of the Fortune Global 100. They are committed to helping clients innovate, optimize, and transform for long-term success. As a Global Top Employer, NTT DATA has diverse experts in more than 50 countries and a robust partner ecosystem. Their services include business and technology consulting, data and artificial intelligence, industry solutions, as well as the development, implementation, and management of applications, infrastructure, and connectivity. NTT DATA is a leading provider of digital and AI infrastructure and is part of the NTT Group, investing in R&D to help organizations and society move confidently into the digital future. Visit us at us.nttdata.com.,

Join us as a Principal Engineer. This challenging role will involve designing and engineering software with a primary focus on customer or user experience. You will actively contribute to our architecture, design, and engineering center of excellence, collaborating to enhance the bank's software engineering capability. This role, offered at the vice president level, provides valuable stakeholder exposure, the opportunity to build and leverage relationships, and a chance to refine your technical skills. As a Principal Engineer, your responsibilities will include creating exceptional customer outcomes through innovative engineering solutions for both existing and new challenges. You will work with software engineers to produce and prototype innovative ideas, collaborate with domain and enterprise architects to validate and incorporate relevant architectures, and lead functional engineering teams. Your role will involve managing end-to-end product implementations, driving demos and stakeholder engagement across platforms, and focusing on automating build, test, and deployment activities. Additionally, you will play a key part in developing the discipline of software engineering within the organization. You will also be responsible for defining, creating, and overseeing engineering and design solutions with a strong emphasis on end-to-end automation, simplification, resilience, security, performance, scalability, and reusability. Working within a platform or feature team, you will collaborate with software engineers to design and engineer complex software, scripts, and tools that enable the delivery of bank platforms, applications, and services. Your role will involve defining and developing architecture models and roadmaps for application and software components, ensuring they meet business and technical requirements, and driving consistent usability across products and domains. You will design, test, and implement working code while applying Agile methods and DevOps techniques to software development. The skills required for this role include significant experience in software engineering, software or database design and architecture, and working within a DevOps and Agile framework. You should possess an expert understanding of the latest market trends, technologies, and tools, along with demonstrable experience in implementing programming best practices, particularly related to scalability, automation, virtualization, optimization, availability, and performance. Additionally, you should have strong experience in gathering business requirements, translating them into technical user stories, and leading functional solution design, especially within the banking domain and CRM (MS Dynamics). Proficiency in PowerApps, D365 (including Custom Pages), and frontend configuration, as well as familiarity with Power BI (SQL, DAX, Power Query, Data Modeling, RLS, Azure, Lakehouse, Python, Spark SQL) is required. A background in designing or implementing APIs and the ability to quickly understand and translate product and business requirements into technical solutions are also essential for this role.,

As a Senior Data Engineer, you will be responsible for designing and developing scalable data pipelines and notebooks using Microsoft Fabric or Synapse Analytics. You should have a research-oriented mindset in Data Projects, thinking outside of the box and focusing on future needs. Your role will involve building and managing Lakehouses and Data Warehouses using Fabrics OneLake architecture, integrating data from diverse sources into Fabric, and collaborating with BI developers for seamless integration with Power BI and other reporting tools. Additionally, you will be required to implement data governance, security, and compliance within the Fabric ecosystem, optimize data storage and processing for performance and cost-efficiency, monitor and troubleshoot data workflows to ensure high data quality and reliability, and document architecture, data models, and processes. It is essential to have experience in automated functional testing along with development. Key Skills required for this role include Pyspark, Data Modelling, Spark SQL, and proficiency in Microsoft Fabric, including an understanding of Shortcuts, Mirroring, Data flows, and all the features. Familiarity with Data Ingestion Design Patterns is also a desired skill for this position.,

Are you intellectually curious and passionate about promoting solutions across organizational boundaries Join the Consumer & Community Banking (CCB) Stress Testing Transformation team for a dynamic opportunity to design and build creative solutions for the future of stress testing and annual CCAR exercises. As a Senior Associate in the Stress Testing Transformation Solution team, you will be a strategic thinker who is passionate about designing and building creative solutions for the future of Stress Testing. You will spend your time solving complex problems, demonstrating strategic thought leadership, and designing the way our stakeholders operate. By leveraging a deep understanding of CCB Stress Testing processes and extensive Finance domain knowledge, you will build scalable solutions that optimize process efficiencies, use data assets effectively, and advance platform capabilities. Responsibilities: - Collaborate with cross-functional teams to lead the design and implementation of end-to-end solutions for Stress Testing, addressing business problems with various technical solutions. - Provide expertise in process re-engineering and guidance based on the roadmap for large-scale Stress Testing transformation initiatives. - Assess, challenge, and provide solutions for Stress Testing processes, focusing on data sources, with the ability to influence and drive the roadmap. - Evaluate, recommend, and develop solutions and architecture, including integration with APIs, Python, AI/ML technology, and other enterprise applications. - Leverage data and best-in-class tools to improve processes and controls, enable cross-business applications, and embrace a consistent framework. - Simplify complex issues into manageable steps and achievements. - Eliminate manual reporting, reengineer processes, and increase the ability to generate insights faster through an integrated data and platform approach. Required Qualifications: - Bachelor's degree in engineering or a related field. - Experience with business intelligence, analytics, and data wrangling tools such as Alteryx, SAS, or Python. - Experience with relational databases, optimizing SQL to extract and summarize large datasets, report creation, and ad-hoc analyses. - Experience with Hive, Spark SQL, Impala, or other big-data query tools. - Ability to understand the underlying business context beyond raw data and identify business opportunities hidden in data. - Collaborative skills to work with global teams in a fast-paced, results-driven environment. - Strong problem-solving and analytical skills with a transformation mindset. Preferred Qualifications: - Experience with Databricks, SQL, Python, or other data platforms. - 8+ years of experience in Analytics Solution and Data Analytics, preferably related to the financial services domain.,

As a Senior Engineer at Impetus Technologies, you will play a crucial role in designing, developing, and deploying scalable data processing applications using Java and Big Data technologies. Your responsibilities will include collaborating with cross-functional teams, mentoring junior engineers, and contributing to architectural decisions to enhance system performance and scalability. Your key responsibilities will revolve around designing and maintaining high-performance applications, implementing data ingestion and processing workflows using frameworks like Hadoop and Spark, and optimizing existing applications for improved performance and reliability. You will also be actively involved in mentoring junior engineers, participating in code reviews, and staying updated with the latest technology trends in Java and Big Data. To excel in this role, you should possess a strong proficiency in Java programming language, hands-on experience with Big Data technologies such as Apache Hadoop and Apache Spark, and an understanding of distributed computing concepts. Additionally, you should have experience with data processing frameworks and databases, strong problem-solving skills, and excellent communication and teamwork abilities. In this role, you will collaborate with a diverse team of skilled engineers, data scientists, and product managers who are passionate about technology and innovation. The team environment encourages knowledge sharing, continuous learning, and regular technical workshops to enhance your skills and keep you updated with industry trends. Overall, as a Senior Engineer at Impetus Technologies, you will be responsible for designing and developing scalable Java applications for Big Data processing, ensuring code quality and performance, and troubleshooting and optimizing existing systems to enhance performance and scalability. Qualifications: - Strong proficiency in Java programming language - Hands-on experience with Big Data technologies such as Hadoop, Spark, and Kafka - Understanding of distributed computing concepts - Experience with data processing frameworks and databases - Strong problem-solving skills - Knowledge of version control systems and CI/CD pipelines - Excellent communication and teamwork abilities - Bachelor's or master's degree in Computer Science, Engineering, or related field preferred Experience: 7 to 10 years Job Reference Number: 13131,

Join us as a Senior Automation QA at Barclays, where you will be responsible for supporting the successful delivery of location strategy projects to plan, budget, agreed quality and governance standards. Spearhead the evolution of the digital landscape, driving innovation and excellence by harnessing cutting-edge technology to revolutionize digital offerings, ensuring unparalleled customer experiences. To be successful in this role, you should have experience with Spark SQL, Python/Pyspark scripting, and ETL concepts. Additionally, skills in AWS exposure and Jupyter Notebook are highly valued. Your key critical skills relevant for success in this role may be assessed, such as risk and controls, change and transformation, business acumen, strategic thinking, and digital and technology, along with job-specific technical skills. This role is based out of Pune. Purpose of the role: Design, develop, and execute testing strategies to validate functionality, performance, and user experience. Collaborate with cross-functional teams to identify and resolve defects, continuously improve testing processes and methodologies to ensure software quality and reliability. Accountabilities: - Develop and implement comprehensive test plans and strategies to validate software functionality and ensure compliance with established quality standards. - Create and execute automated test scripts, leveraging testing frameworks and tools for early detection of defects. - Collaborate with cross-functional teams to analyze requirements, participate in design discussions, and contribute to the development of acceptance criteria. - Conduct root cause analysis for identified defects, working closely with developers for defect resolution. - Participate in code reviews, promote a culture of code quality and knowledge sharing. - Stay informed of industry technology trends and innovations, contribute to the organization's technology communities. As an Assistant Vice President, you are expected to advise and influence decision-making, contribute to policy development, and ensure operational effectiveness. Lead a team performing complex tasks, set objectives, coach employees, and demonstrate leadership behaviours to create an environment for colleagues to thrive. For an individual contributor, lead collaborative assignments, guide team members, identify new directions for assignments, consult on complex issues, and contribute to risk mitigation and policy development. All colleagues are expected to demonstrate the Barclays Values of Respect, Integrity, Service, Excellence, and Stewardship, along with the Barclays Mindset to Empower, Challenge, and Drive.,

As a Senior Data Engineering Architect at Iris Software, you will play a crucial role in leading enterprise-level data engineering projects on public cloud platforms like AWS, Azure, or GCP. Your responsibilities will include engaging with client managers to understand their business needs, conceptualizing solution options, and finalizing strategies with stakeholders. You will also be involved in team building, delivering Proof of Concepts (PoCs), and enhancing competencies within the organization. Your role will focus on building competencies in Data & Analytics, including Data Engineering, Analytics, Data Science, AI/ML, and Data Governance. Staying updated with the latest tools, best practices, and trends in the Data and Analytics field will be essential to drive innovation and excellence in your work. To excel in this position, you should hold a Bachelor's or Master's degree in a Software discipline and have extensive experience in Data architecture and implementing large-scale Data Lake/Data Warehousing solutions. Your background in Data Engineering should demonstrate leadership in solutioning, architecture, and successful project delivery. Strong communication skills in English, both written and verbal, are essential for effective collaboration with clients and team members. Proficiency in tools such as AWS Glue, Redshift, Azure Data Lake, Databricks, Snowflake, and databases, along with programming skills in Spark, Spark SQL, PySpark, and Python, are mandatory competencies for this role. Joining Iris Software offers a range of perks and benefits designed to support your financial, health, and overall well-being. From comprehensive health insurance and competitive salaries to flexible work arrangements and continuous learning opportunities, we are dedicated to providing a supportive and rewarding work environment where your success and happiness are valued. If you are inspired to grow your career in Data Engineering and thrive in a culture that values talent and personal growth, Iris Software is the place for you. Be part of a dynamic team where you can be valued, inspired, and encouraged to be your best professional and personal self.,