12395 Big Data Jobs - Page 39

Design & Develop Transformations aspects using the ELT framework to modernize the Ingestion pipelines and build data transformations at scale Work closely with other data teams & Data Science teams and participate in the development of Ingestion pipelines Provide technical expertise in the areas of design and implementation of Ratings Data Ingestion pipelines with modern AWS cloud and other technologies such as S3, Hive, Databricks, Scala, Python, and Spark Build and maintain a data environment for speed, accuracy, consistency, and up time Work closely with other data teams & Data Science teams and participate in the development of Ingestion pipelines Ensure data governance principles are ad...

We are seeking a big data engineer with advance Java experience to join our modernization initiative team. The candidate will be responsible for implementing and maintaining high volume online and offline batch and transactional client facing applications in cloud and On-prem. Candidate will also responsible for the creation and maintenance of our Code & database infrastructure, including collecting and maintaining data, ensuring the integrity of our data, and training data models Minimum 8 to 10 years of experience in Spring boot, Advanced Java and Spark. Bachelors degree or higher in computer science, data science, or a related field. Hands-on experience with data cleaning, visualisation, ...

Location - Hyderabad/Bangalore/Gurugram/Pune/Mumbai Work type: Remote but physical onboarding will be conducted & candidate needs to visit office in 3 to 6 month for conference Key Points Must Have Advanced Java proficient- Microservices Spring boot Writing SQL Queries (proficient) AWS 4-5 Yrs in Java, - Spark Good to have Unix Shell scripting JD Minimum of 8 years of experience in building complex Data Platforms and Data Engineering solutions. Minimum of 6 years of hands on experience in architecture and development of data solutions in AWS environment using AWS Services Experience with big data technologies such as: Spark, EMR, Hadoop, Hive, Experience programming with at least one modern ...

We are seeking a big data engineer with advance Java experience to join our modernization initiative team. The candidate will be responsible for implementing and maintaining high volume online and offline batch and transactional client facing applications in cloud and On-prem. Candidate will also responsible for the creation and maintenance of our Code & database infrastructure, including collecting and maintaining data, ensuring the integrity of our data, and training data models Minimum 8 to 10 years of experience in Spring boot, Advanced Java and Spark. Bachelors degree or higher in computer science, data science, or a related field. Hands-on experience with data cleaning, visualisation, ...

Key Responsibility Area: ( Specifies Key Result Areas for the Incumbent ) o Engage with clients to precisely identify and fulfill their data engineering needs. o Lead and manage special projects to meet strategic goals. o Develop advanced models with Snowflake and SSIS to enhance data accuracy and utility. o Continuously refine data processing rules and procedures for optimal results. o Design and implement scalable data architectures using Snowflake. o Maintain and enhance data pipelines, integrating new data sources and APIs as needed. o Monitor and ensure high data quality across systems for reliable decision-making. o Utilize SSIS for efficient data extraction, transformation, and loadin...

Role Overview : Looking for experienced Python Developers to work on big data solutions using PySpark and Hadoop in a client-facing environment. Key Responsibilities : Develop and optimize data processing pipelines using Python and PySpark. Work with Hadoop ecosystems for large-scale data processing. Collaborate with cross-functional teams to implement scalable solutions. Ensure performance tuning and best coding practices. Required Skills : 5-7 years (SSE) / 79 years (TL) of experience in Python development. Strong expertise in PySpark and Hadoop. Hands-on experience in big data processing and analytics.

This is a remote position. We are seeking a big data engineer with advance Java experience to join our modernization initiative team. The candidate will be responsible for implementing and maintaining high volume online and offline batch and transactional client facing applications in cloud and On-prem. Candidate will also responsible for the creation and maintenance of our Code & database infrastructure, including collecting and maintaining data, ensuring the integrity of our data, and training data models Minimum 8 to 10 years of experience in Spring boot, Advanced Java and Spark. Bachelors degree or higher in computer science, data science, or a related field. Hands-on experience with dat...

US Shift Key Responsibility Area: ( Specifies Key Result Areas for the Incumbent ) o Engage with clients to precisely identify and fulfill their data engineering needs. o Lead and manage special projects to meet strategic goals. o Develop advanced models with Snowflake and SSIS to enhance data accuracy and utility. o Continuously refine data processing rules and procedures for optimal results. o Design and implement scalable data architectures using Snowflake. o Maintain and enhance data pipelines, integrating new data sources and APIs as needed. o Monitor and ensure high data quality across systems for reliable decision-making. o Utilize SSIS for efficient data extraction, transformation, a...

8+ years of Test Data Management experience on tools like Informatica, Delphix, Broadcom, etc. Good understanding of data masking, scrubbing, and obfuscation techniques. Experience with CA TDM and Broadcom TDM (Informatica, Data masking, integration). Ability to write queries, macros, and scripts using tools like SQL, VB, QTP, Selenium, and XML. Good understanding of various database technologies (DB2, Teradata, Oracle 10g/11g, SQL Server, Big Data/Hadoop). Automate test data refresh and creation processes. Automation experience using Selenium, Excel Macros, or similar tools. Proficient in MS Word, Excel, Outlook, and PowerPoint. Desired Skills Delphi Informatica JMETER Selenium Test Automat...

Desired Skills: Azure Big Data Desired Competencies (Technical/Behavioral ): Experience with creating Data Models, Data Lineages, and Data Flow Diagrams. Experience performing Data Migration using native Azure tools, third-party applications, or custom-built data pipelines. Design and architect transactional and analytical data systems using MPP as well as transactional databases. Expertise in building Data Pipelines using Azure Data Factory and Python. Experience managing data platforms such as Data Lakes and Data Warehouses using Databricks, Azure Synapse, and/or Snowflake. Experience developing and managing Azure Data Services like ADF, Databricks, and SQL databases. Hands-on experience m...

Job Description Experience in PySpark, Hive, and HBase. Proficiency in integrating, transforming, and consolidating data from various structured and unstructured data sources. Good experience in SQL or native SQL query languages. Good experience in PySpark, Hive, HBase, and Hadoop. Exposure to basic issues in working within Big Data clusters. Desired Skills Big Data Hadoop Python Spark Apache Spark Qualifications Bachelor of Technology.

5+ years of experience in technical presales, solution architect, or similar customer-facing role. Previous experience with Databricks or Spark-based technologies is a plus. Experience with machine learning frameworks and pipelines, particularly in Databricks. Experience with AI/ML model deployment. Desired Skills - AWS, Big Data, Machine Learning, Spark Qualifications Bachelor of Engineering.

Designing solutions: Expertise in creating detailed technical solutions to meet business goals. Design and implement high-quality, scalable, and secure software architectures using microservices. Design the technical architecture for GenAI solutions, including data pipelines, model selection, training infrastructure, and deployment strategies. Oversee the design, implementation, and maintenance of the CI/CD pipelines for GenAI solutions. Utilize cloud services effectively to build, deploy, and manage applications. Develop and maintain data architectures, ensuring efficient handling of big data. Leverage data warehousing and advanced analytics solutions. Design and build scalable cloud infras...

Required Technical Skill Set: Bigdata, Hadoop, Apache, Spark, Hive, Scala, Kafka Must-Have: Prior experience of working with HDFS Hive/HQL Shell Scripting Spark Arcadia Kafka Daily status reporting Attending scrum calls Good to Have: Good communication skills Good in troubleshooting Agile knowledge is a must Good team player Minimum Qualification: 15 years of full-time education Minimum percentile of 50% in 10th, 12th, UG & PG (if applicable)

Role Overview Seeking a Data Engineer (PySpark) to build big data pipelines and handle ETL development. Key Responsibilities Develop scalable ETL pipelines using PySpark . Work with Databricks, Hadoop, Hive, Spark SQL . Integrate data into data lakes and warehouses. Troubleshoot data quality and performance issues. Skills & Qualifications Expertise in Python, PySpark, SQL. Experience with cloud (AWS/Azure/GCP).

5-6 years of total experience in data engineering or big data development. 2-3 years hands-on experience with Databricks and Apache Spark. Proficient in AWS cloud services (S3, Glue, Lambda, EMR, Redshift, CloudWatch, IAM). Strong programming skills in PySpark, Python, and optionally Scala. Solid understanding of data lakes, lakehouses, and Delta Lake concepts. Experience in SQL development and performance tuning. Familiarity with Airflow, dbt, or similar orchestration tools is a plus. Experience in CI/CD tools like Jenkins, GitHub Actions, or CodePipeline. Knowledge of data security, governance, and compliance frameworks. Qualifications: Bachelor of Engineering

Overall minimum 6+ years of post-college working experience as a developer in Big Data, Hadoop, or Data Warehousing Tools, with Cloud Computing as an added skill. Minimum 3+ years of experience with Hadoop, Hive SQL, Spark, and Big Data ecosystem tools. Preferable to have good technical knowledge of Cloud Computing, AWS, or Azure Cloud Services. Strong conceptual and creative problem-solving skills, ability to work with considerable ambiguity, and ability to learn new and complex concepts quickly. Experience working with teams in a complex organization involving multiple reporting lines. The candidate should be able to tweak queries and work on performance enhancement. The candidate will be ...

We are looking for a skilled Java/Spark, Scala professional with 5 to 10 years of experience. The job location is flexible. Roles and Responsibility Design, develop, and implement scalable data processing systems using Java/Spark. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data pipelines and architectures. Troubleshoot and optimize system performance issues. Participate in code reviews and ensure high-quality code standards. Stay updated with industry trends and emerging technologies in Java/Spark. Job Requirements Strong proficiency in Java/Spark and Scala programming languages. Experience with big data processin...

Spark/Scala - Job Description Qualifications: Bachelor of Engineering Desired Skills: Scala, Spark, Unix Must-Have Skills Minimum 4+ years of experience in development of Spark Scala. Experience in designing and developing solutions for Big Data using Hadoop ecosystem technologies such as HDFS, Spark, Hive, Parquet File format, YARN, MapReduce, Sqoop. Good experience in writing and optimizing Spark jobs, Spark SQL, and handling both batch and streaming data processing. Experience in writing and optimizing complex Hive and SQL queries to process large datasets, including proficiency with UDFs, tables, joins, and views. Experience in debugging Spark code. Working knowledge of basic UNIX comman...

Key Technical Skills: AWS tools (Redshift, Glue, EMR, Athena, StepFunction, CloudFormation, CloudWatch) Programming languages: SQL, Python, Pyspark Data Pipelining skills Data blending Big data technologies: Hadoop, Hive Data Ingestion and Integration: Ingest data from various sources such as databases, files, APIs, and streaming platforms into Azure. For example, use Azure Data Factory to orchestrate the extraction, transformation, and loading (ETL) of data from an on-premises SQL Server database to Azure Blob Storage. Data Transformation and Processing: Perform data transformation and processing tasks to prepare the data for analysis and reporting. Azure Databricks or Azure Synapse Analyti...

Hands-on development experience in programming languages such as Java and Scala using Maven, Apache Spark Frameworks , and Unix Shell scripting. Comfortable with Unix File System as well as HDFS commands. Experience with query languages such as Oracle SQL, Hive SQL, Spark SQL, Impala, and HBase DB. Should be flexible. Good communication and customer management skills. Desired Candidate Profile Qualifications : Undergraduate

Strong proficiency in Python and SQL for data manipulation and processing. Experience with data warehouse solutions such as Snowflake , BigQuery , and Databricks . Ability to design and implement efficient data models for data lakes and warehouses. Familiarity with CI/CD pipelines and automation tools to streamline data engineering workflows. Deep understanding of principles in data warehousing and cloud architecture for building efficient and scalable data systems. Experience with Apache Airflow and/or AWS MWAA. Experience with Snowflakes distinctive features, including multi-cluster architecture and shareable data features. Expertise in distributed processing frameworks like Apache Spark o...

Role Overview: We are looking for a skilled Database & Data Engineer to design, build, and maintain robust data solutions on AWS . The ideal candidate will have hands-on experience with RDS (Oracle & PostgreSQL) , data lakes , and streaming platforms , ensuring efficient data flow and high availability across systems. Key Responsibilities: Develop and maintain data pipelines using Apache NiFi and Spark for batch and streaming data processing. Manage AWS RDS instances for Oracle and PostgreSQL , including configuration, performance tuning, and troubleshooting. Implement and optimize data lakes on AWS S3 using Apache Iceberg for structured and semi-structured data. Build and manage real-time d...

Design, develop, and deploy end-to-end data engineering solutions on the Databricks platform. Build scalable, reliable, and optimized data pipelines that support large-scale analytics, machine learning, and real-time data processing. Utilize Databricks to integrate, transform, and analyze big data from various sources. Desired Skills AWS, Big Data, Spark, Apache, Data Modeling Qualifications Bachelor of Engineering

Need to work as a developer in Cloudera Hadoop. Work on Hadoop, Python, PySpark, Hive SQLs, Bigdata Eco System Tools. Experience in working with teams in a complex organization involving multiple reporting lines. The candidate should have strong functional and technical knowledge to deliver what is required and should be well acquainted with Banking terminologies. The candidate should have strong DevOps and Agile Development Framework knowledge. Create Python/PySpark jobs for data transformation and aggregation. Qualifications Bachelor of Engineering