9747 Big Data Jobs - Page 38

Design, develop, and maintain scalable big data systems and pipelines. Implement data processing frameworks and optimize large datasets using tools like Hadoop, Spark, and Hive. - Develop and maintain ETL processes and ensure data is available and accurate for downstream applications.

1. Experience specifically in defining solution, performing POCs, Application design and development, integration migration. 2. Understanding of Google Cloud Platform technologies in the big data and data warehousing space BigQuery, pub-sub, Dataflow, Data Catalog, Composer Airflow, Complex SQL, Stored Procedures) with experience on delivering data architect solutions using BigQuery, Google data flow, Google pub sub, google cloud SQL, google compute engine, etc. Should have at least 5+ year of experience in GCP Google Cloud Platform or similar Cloud Technologies. 3. Experience in Looker with design, development, configuration setup, dashboarding and reporting techniques. 4. Experience in Dev...

Job Description: Need to work as a developer in Bigdata, Hadoop or Data Warehousing Tools and Cloud Computing ? Work on Hadoop, Hive SQL?s, Spark, Bigdata Eco System Tools.? Experience in working with teams in a complex organization involving multiple reporting lines.? The candidate should have strong functional and technical knowledge to deliver what is required and he/she should be well acquainted with Banking terminologies. ? The candidate should have strong DevOps and Agile Development Framework knowledge.? Create Scala/Spark jobs for data transformation and aggregation? Experience with stream-processing systems like Storm, Spark-Streaming, Flink" Essential Skills: "? Working experience ...

Job Description: Role descriptions / Expectations from the Role Experience in other messaging systems such as IBM MQ or Rabbit MQ. Experience in the Atlassian stack tools Jira, Confluence and Bitbucket (GIT). Experience in Jenkins automation development. Experience in the scripting languages shell and python. Experience in the operating systems Linux (RedHat and/or SuSE) and Windows Server. Experience in search engines such as Elasticsearch. Job description: must have • Cluster Management: Install, configure, and maintain Confluent and AWS MSK Kafka clusters. Manage large-scale, multi-node Kafka cluster environments (on-prem and cloud). Troubleshoot and resolve issues related to Kafka broker...

Responsibilities Job Responsibilities: Collaborate closely with global product and engineering teams to design AI and LLM solutions and support business objectives. Design, develop and deliver AI/ML models, leveraging cutting edge technologies to drive innovation. Analyze user behavior on existing solutions and enhance them through an AI/ML driven approach. Basic Qualifications Bachelor or above degree in Computer Science or Engineering 6+ years of working experience in Natural Language Processing and machine learning. Deep knowledge in NLP techniques, including embeddings, attention mechanisms, sequence-to-sequence models, generative AI, etc. Rich experience in designing and implementing NL...

Design, develop, and maintain a robust, end-to-end data analytics pipeline for real-time data processing and visualization Take ownership of the entire project lifecycle, from architectural design and component integration to deployment, security, and presentation Work on data analytics platform (on GKE/GDC air gapped) to provide real-time hot spot prediction and ride recommendations for taxi drivers The application will use the NYC taxi dataset and showcase the capabilities of Kafka, Spark, HBase, Solr, Trino, Apache Airflow, Apache Superset and Apache Ranger Requirements Must Have: Big Data Ecosystem: Apache Kafka, Apache Spark, HBase, Apache Solr, and TrinoTools & Orchestration: Apache Ai...

Job Summary: We are looking for an experienced GCP Data Engineer to design, build, and optimize data pipelines and solutions on the Google Cloud Platform (GCP) . The ideal candidate will have deep expertise in BigQuery, Dataflow, Dataproc , and other GCP data services, along with strong programming and analytical skills. Key Responsibilities: Design and develop scalable data pipelines and ETL workflows using GCP services such as BigQuery, Dataflow, Dataproc, Composer, and Pub/Sub . Build and maintain data lakes, data warehouses, and analytics solutions on GCP. Collaborate with data scientists, analysts, and stakeholders to deliver clean, reliable, and structured datasets . Implement data ing...

Job Summary: We are looking for an experienced GCP Data Engineer to design, build, and optimize data pipelines and solutions on the Google Cloud Platform (GCP) . The ideal candidate will have deep expertise in BigQuery, Dataflow, Dataproc , and other GCP data services, along with strong programming and analytical skills. Key Responsibilities: Design and develop scalable data pipelines and ETL workflows using GCP services such as BigQuery, Dataflow, Dataproc, Composer, and Pub/Sub . Build and maintain data lakes, data warehouses, and analytics solutions on GCP. Collaborate with data scientists, analysts, and stakeholders to deliver clean, reliable, and structured datasets . Implement data ing...

Overview Prodapt is looking for senior data engineers having 3+ years of experience having hands on expertise on GCP, BQ, Data Pipeline creation & ETL. Responsibilities Understanding the requirements Performing in-depth analysis on data Setting up secure data pipelines from ingestion to delivery Individual contributor Requirements 3- 5 years of experience in Data Engineering with strong ability to work independently, communicate effectively, and deliver high-quality solutions. Should have strong SQL programming skills. Experience in cloud-based systems (GCP, BQ) is must. Performs data analysis required to troubleshoot data related issues and assist in the resolution of data issues. Expertise...

Job Summary: We are looking for a Big Data Engineer with strong expertise in Apache Spark and Scala to design, develop, and optimize large-scale data processing solutions. The ideal candidate will have hands-on experience with distributed data systems, ETL development, and modern data engineering tools. Key Responsibilities: Design and develop data ingestion, transformation, and processing pipelines using Apache Spark (Core, SQL, Streaming) and Scala . Build scalable and reliable big data solutions on distributed platforms such as Hadoop, Databricks, or EMR . Optimize Spark jobs for performance and scalability . Work closely with data architects, analysts, and business teams to define data r...

Job Summary: We are seeking an experienced ML Data Scientist to design, develop, and deploy machine learning models that solve complex business problems and deliver measurable outcomes. The ideal candidate will possess strong analytical skills, deep understanding of statistical and ML techniques, and hands-on experience in model development and deployment in production environments. Key Responsibilities: Understand business objectives and translate them into data-driven solutions . Design, develop, and implement machine learning models, algorithms, and predictive analytics . Perform data cleaning, preprocessing, feature engineering , and exploratory data analysis (EDA). Evaluate model perfor...

Job Summary: We are looking for an experienced GCP Data Engineer to design, build, and optimize data pipelines and solutions on the Google Cloud Platform (GCP) . The ideal candidate will have deep expertise in BigQuery, Dataflow, Dataproc , and other GCP data services, along with strong programming and analytical skills. Key Responsibilities: Design and develop scalable data pipelines and ETL workflows using GCP services such as BigQuery, Dataflow, Dataproc, Composer, and Pub/Sub . Build and maintain data lakes, data warehouses, and analytics solutions on GCP. Collaborate with data scientists, analysts, and stakeholders to deliver clean, reliable, and structured datasets . Implement data ing...

Job Summary: The Principal Digital Architect will provide strategic leadership for the overall architecture of the Manufacturing and Supply Digital Platform by designing platform capabilities that anticipate evolving business requirements and use cases. The role will collaborate with peer architects and Enterprise Capability Owners to deliver diverse use cases thru applications, analytics, agents, and business intelligence. The position holds accountability for developing an enterprise-wide data architecture, creating reusable data components that can be assembled into datasets for business applications, analytics, business intelligence, agents, and digital twin solutions. Additionally, the ...

Job Summary: The Principal Digital Architect will provide strategic leadership for the overall architecture of the Manufacturing and Supply Digital Platform by designing platform capabilities that anticipate evolving business requirements and use cases. The role will collaborate with peer architects and Enterprise Capability Owners to deliver diverse use cases thru applications, analytics, agents, and business intelligence. The position holds accountability for developing an enterprise-wide data architecture, creating reusable data components that can be assembled into datasets for business applications, analytics, business intelligence, agents, and digital twin solutions. Additionally, the ...

Job Summary We’re hiring a Software Engineer with expertise in AI/ML platforms and cloud-native microservices. The role involves designing scalable AI infrastructure, building intelligent agent systems, and optimizing machine learning workflows. Key skills include Python, TensorFlow, PyTorch, Kubernetes, and experience with tools like Pinecone and FAISS. You’ll work on CI/CD pipelines, RAG systems, and integrate AI models into real-world applications. Ideal candidates are analytical, innovative, and passionate about building intelligent automation across domains like robotics and data analysis. Job Requirements • Creative and analytical problem-solving abilities. • Understanding of computer ...

We are seeking a Consulting Lead-Databricks with strong experience in Databricks, Python, and PySpark, along with hands-on knowledge of Unity Catalog (UC) and Lakehouse architecture. In this role, youll design and optimize large-scale data pipelines, implement robust governance and security through UC, and build scalable solutions in a modern cloud data environment. The ideal candidate must have 5+ years in data engineering, expertise with any one of the cloud platforms (Azure/AWS/GCP), and proven ability to deliver high-performing ETL/ELT pipelines that support business-critical analytics. Qualification Hands-on expertise in Databricks, Python, PySpark, and Delta Lake/Lakehouse architecture...

Role & responsibilities Work as a project manager to Lead the design and evolution of our large-scale data lakehouse architecture, ensuring it is scalable, reliable, and cost-effective. Provide technical leadership and mentorship to data engineers and analysts. Collaborate closely with Software Engineers, and business stakeholders to understand requirements and deliver effective data solutions. Experience in data modelling, data mapping, data profiling and meta data management. Architect and tune high-performance data processing pipelines. Identify and resolve complex performance issues in distributed computing environments involving Spark execution, Hive query optimization, and Iceberg meta...

Overview Job Summary: We are looking for a skilled GCP Data Engineer with 3 to 6 years of experience to join our dynamic team. The ideal candidate will have a strong background in data engineering, cloud technologies, and experience with Google Cloud Platform (GCP) services. You will be responsible for designing, building, and maintaining scalable data pipelines and frameworks that support our data analytics and machine learning initiatives. Key Responsibilities: Design and implement data pipelines using GCP services such as BigQuery, Dataflow, and Pub/Sub. Collaborate with data scientists and analysts to understand data needs and deliver appropriate solutions. Optimize data storage and proc...

We are seeking a skilled Data Engineer with strong experience in SQL, PySpark, and cloud technologies to join our dynamic team. The ideal candidate will have a solid background in designing and implementing data engineering pipelines, with a focus on performance, scalability, and reliability. You will work closely with other data engineers, data scientists, and stakeholders to develop, maintain, and optimize data infrastructure. Key Responsibilities : SQL Development : - Write medium-complexity SQL queries to extract, transform, and load data efficiently. - Optimize SQL queries for performance and scalability. - Collaborate with team members to understand data requirements and translate them...

Spark, Hive, Scala, SQL ,Python scripting ,data visualization Key Responsibilities: Leadership and Team Management: Lead and mentor a team of data engineers, fostering a culture of collaboration and continuous improvement. Oversee the planning and execution of data projects, ensuring alignment with business objectives and timelines. Provide technical guidance and expertise to the team, promoting best practices in data engineering. Data Architecture and Development: Design, develop, and maintain scalable data pipelines and ETL processes to ingest, process, and store large volumes of data. Architect and implement robust data models and schemas to support analytics and business intelligence nee...

Strong implementation knowledge of Moodle 4.0+, Should have a proven track record in developing Plugins and themes. Good in building responsive layouts . Capability in writing test cases. It is good to have exposure to third-party API integration. (SIS Banner, Salesforce, etc...). Understanding of GCP cloud services (Good to have)

Overview Prodapt is looking for Lead MLOPS Engineers having experience of 4 to 8 years from chennai or bangalore locations. Responsibilities Requirements B.S./B.E/B.Tech/MTech in computer science with 4 to 8 years of experience or equivalent Experience with Big Data Platform and data processing frameworks (Hadoop, Hive, Map Reduce, Spark, etc.) and professional skill in writing optimal production grade SQL queries. Experience on deploying shallow learning and deep learning models, built using TensorFlow, PyTorch, H2O, SciKit-Learn, Keras, Spark MLlib, XGBoost, LightGBM etc. frameworks. Experience with cloud technologies specially with respect to Data and ML Engineering (GCP/AWS/Azure) Experi...

The Principal Digital Architect will provide strategic leadership for the overall architecture of the Manufacturing and Supply Digital Platform by designing platform capabilities that anticipate evolving business requirements and use cases. The role will collaborate with peer architects and Enterprise Capability Owners to deliver diverse use cases thru applications, analytics, agents, and business intelligence. The position holds accountability for developing an enterprise-wide data architecture, creating reusable data components that can be assembled into datasets for business applications, analytics, business intelligence, agents, and digital twin solutions. Additionally, the role will lev...

What will your role look like Design, develop, and maintain scalable data pipelines and applications using Python and PySpark. Build end-to-end data solutions from scratch, ensuring clean, modular, and reusable code. Write and optimize complex SQL queries for large-scale data processing, transformation, and validation. Develop dynamic, reusable frameworks and implement design patterns for scalability and performance. Refactor legacy codebases to enhance performance, maintainability, and reliability. Collaborate with cross-functional teams including data engineers, analysts, and stakeholders to translate business requirements into technical solutions. Utilize Databricks and Azure Data Factory...

We are looking for a highly skilled Data Engineer with expertise in AWS to join our team. The ideal candidate will have a strong background in designing and implementing data pipelines using AWS services such as S3, Lambda, and EMR. Roles and Responsibility Design and implement scalable data pipelines using AWS services like S3, Lambda, and EMR. Develop and maintain large-scale data warehouses and data lakes on AWS. Collaborate with cross-functional teams to identify and prioritize project requirements. Ensure data quality and integrity by implementing data validation and testing procedures. Optimize data processing workflows for performance and efficiency. Troubleshoot and resolve technical...