654 Apache Pig Jobs - Page 7

Role Description As a Technical Lead - Cloud Data Platform (AWS) at Incedo, you will be responsible for designing, deploying and maintaining cloud-based data platforms on the AWS platform. You will work with data engineers, data scientists and business analysts to understand business requirements and design scalable, reliable and cost-effective solutions that meet those requirements. Roles & Responsibilities: Designing, developing and deploying cloud-based data platforms using Amazon Web Services (AWS) Integrating and processing large amounts of structured and unstructured data from various sources Implementing and optimizing ETL processes and data pipelines Developing and maintaining securi...

We are looking for a skilled Hadoop Developer with 3 to 6 years of experience to join our team in Chennai. The ideal candidate will have expertise in Scala, SQL, Unix, and Hive & Spark. Roles and Responsibility Design, develop, and implement scalable data processing solutions using Hadoop technologies. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data pipelines using Scala and Spark. Troubleshoot and resolve complex technical issues related to Hadoop applications. Participate in code reviews and contribute to improving overall code quality. Stay up-to-date with the latest trends and technologies in Hadoop developmen...

Python, big data (Hive, spark or snowflake, Kafka), SQL/ SQLServer. Experience in Python for data processing.

Immediate Openings onPyspark Experience : 5+ Skill:-Pyspark Location :- Bangalore Notice Period :- Immediate. Pyspark Experience in Cloud platform, e.g., AWS, GCP, Azure, etc. Experience in distributed technology tools, viz. SQL, Spark, Python, PySpark, Scala Performance Turing Optimize SQL, PySpark for performance Airflow workflow scheduling tool for creating data pipelines GitHub source control tool & experience with creating/ configuring Jenkins pipeline

Role Description As a Senior Cloud Data Platform (AWS) Specialist at Incedo, you will be responsible for designing, deploying and maintaining cloud-based data platforms on the AWS platform. You will work with data engineers, data scientists and business analysts to understand business requirements and design scalable, reliable and cost-effective solutions that meet those requirements. Roles & Responsibilities: Designing, developing and deploying cloud-based data platforms using Amazon Web Services (AWS) Integrating and processing large amounts of structured and unstructured data from various sources Implementing and optimizing ETL processes and data pipelines Developing and maintaining secur...

Role Description As a Technical Lead - Cloud Data Platform (GCP) at Incedo, you will be responsible for managing and optimizing the Google Cloud Platform environment, ensuring its performance, scalability, and security. You will work closely with data analysts and data scientists to develop data pipelines and run data science experiments. You will be skilled in cloud computing platforms such as AWS or Azure and have experience with big data technologies such as Hadoop or Spark. You will be responsible for configuring and optimizing the GCP environment, ensuring that data pipelines are efficient and accurate, and troubleshooting any issues that arise. You will also work with the security team...

Looking to onboard a skilled Hadoop Admin with 5-12 years of experience to join our team in Bangalore. The ideal candidate will have a strong background in Big Data technologies and programming languages such as Python, Scala. Roles and Responsibility Design, develop, and implement scalable data processing systems using Hadoop and Spark. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data pipelines and architectures. Troubleshoot and resolve complex technical issues related to data processing and storage. Ensure high availability and performance of data systems and applications. Participate in code reviews and contrib...

Design and implement an end-to-end data solution architecture tailored to enterprise analytics and operational needs. Build, maintain, and optimize data pipelines and transformations using Python, SQL, and Spark . Manage large-scale data storage and processing with Iceberg, Hadoop, and HDFS . Develop and maintain dbt models to ensure clean, reliable, and well-structured data. Implement robust data ingestion processes , integrating with third-party APIs and on-premise systems. Collaborate with cross-functional teams to align on business goals and technical requirements. Contribute to documentation and continuously improve engineering and data quality processes. Qualifications 4+ years of expe...

Skill:Data Engineer Notice Period:Immediate. Employment Type: Contract Job Description : Experience in Cloud platform, e.g., AWS, GCP, Azure, etc. Experience in distributed technology tools, viz. SQL, Spark, Python, PySpark, Scala Performance Turing Optimize SQL, PySpark for performance Airflow workflow scheduling tool for creating data pipelines GitHub source control tool & experience with creating/ configuring Jenkins pipeline Experience in EMR/ EC2, Databricks etc. DWH tools incl. SQL database, Presto, and Snowflake Streaming, Serverless Architecture .

Big data engineer/Developer Spark-Scala HQL, Hive Control-m Jinkins Git Technical analysis and up to some extent business analysis (knowledge about banking products, credit cards and its transactions) Notice Period: Immediate . Employment Type: Contract

We are looking for a skilled Hadoop Admin with 7 to 12 years of experience, located in Hyderabad, Bangalore, Chennai, and Bhubaneshwar. The ideal candidate will have hands-on experience in building EKS clusters, Spark on EKS, AWS (EMR, S3, RDS), Python as programming skills, Shell scripting, and any CI/CD. Roles and Responsibility Manage and maintain large-scale Hadoop clusters for high availability and performance. Design and implement scalable data processing pipelines using Spark and Airflow instances. Integrate Ranger and Hive Metastore services in EKS for secure data storage and access. Collaborate with cross-functional teams to identify and resolve technical issues. Develop and impleme...

We are looking for a skilled Hadoop professional with 7-14 years of experience to join our team in Pune and Hyderabad. The ideal candidate will have expertise in Hadoop, Spark, Kafka, Data Visualization, scripting languages in Python and R, SQL and NoSQL, Agile methodology, ETL / ELT data pipelines. Roles and Responsibility Design and develop scalable Hadoop applications using Spark and Kafka. Create data visualizations using tools like Splunk, Qlik, and Grafana. Develop and maintain ETL/ELT data pipelines using Agile methodology. Collaborate with cross-functional teams to identify and prioritize project requirements. Troubleshoot and resolve technical issues related to Hadoop applications. ...

We are looking for skilled Hadoop Developers with 5-10 years of experience to join our team in Bangalore, Kolkata, Hyderabad, and Pune. The ideal candidate should have strong proficiency in Hadoop, Scala, Spark, and SQL. Roles and Responsibility Design, develop, and implement scalable data processing systems using Hadoop. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data pipelines using Spark and Scala. Troubleshoot and resolve complex technical issues related to Hadoop. Participate in code reviews and ensure high-quality code standards. Stay updated with the latest trends and technologies in Hadoop development. Job...

We are looking for a skilled Data Engineer with 4 to 7 years of experience to join our team in Bangalore. The ideal candidate will have expertise in Python, SQL, AWS, Spark, and Big Data processing frameworks. Roles and Responsibility Design and develop scalable data pipelines using Python, Scala, or Java. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data systems using Hadoop and Big Data processing frameworks. Ensure data quality and integrity by implementing robust testing and validation procedures. Optimize data storage and retrieval processes for improved performance and efficiency. Participate in code reviews t...

We are looking for a skilled Data Engineer with 4 to 7 years of experience to join our team in Bangalore, Hyderabad, Chennai, and Pune. The ideal candidate will have expertise in Python, Pyspark, SQL, Hadoop, and NoSQL databases. Roles and Responsibility Design and develop scalable data pipelines using Azure ML, Azure Data Factory, and MLOps. Collaborate with cross-functional teams to integrate data from various sources into a unified platform. Develop and maintain large-scale data warehouses using Hadoop and NoSQL databases. Implement CI/CD pipelines for automated testing and deployment of data engineering projects. Troubleshoot and resolve complex technical issues related to data engineeri...

We are looking for a skilled Pyspark Data Engineer with 3 to 5 years of experience. The ideal candidate will have expertise in Hadoop, Spark/Pyspark, Hive, YARN, Scala development, and data modeling. This position is based in Pune and Hyderabad. Roles and Responsibility Design and develop scalable data pipelines using Pyspark and Spark. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data systems and architectures. Ensure data quality and integrity by implementing robust testing and validation procedures. Participate in code reviews and contribute to improving overall code quality. Troubleshoot and resolve complex tech...

We are looking for a skilled professional with 5-9 years of experience to join our team as an Hadoop-Spark specialist. The ideal candidate will have a strong background in big data processing and analytics. Roles and Responsibility Design, develop, and implement scalable big data solutions using Hadoop and Spark. Collaborate with cross-functional teams to identify business requirements and develop technical solutions. Develop and maintain large-scale data pipelines and architectures. Troubleshoot and optimize performance issues in big data systems. Ensure data quality and integrity through data validation and testing procedures. Stay updated with industry trends and emerging technologies in ...

We are looking for skilled Hadoop Developers with 7-10 years of experience to join our team in Bangalore and Hyderabad. Roles and Responsibility Design, develop, and implement scalable data processing systems using Hadoop and Spark. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data pipelines and architectures. Troubleshoot and optimize system performance issues. Ensure data quality and integrity through data validation and testing procedures. Participate in code reviews and contribute to improving overall code quality. Job Requirements Strong proficiency in Hadoop, Scala, Spark, and SQL. Experience working with big ...

We are looking for a skilled Pyspark Data Engineer with 5 to 7 years of experience, located in Pune and Hyderabad. The ideal candidate will have expertise in Pyspark, Scala, Hive, Airflow, Control M, CICD, Python, Hadoop, Data Modelling, and experience with Jira, Confluence, GitHub, or other similar Agile Scrum technologies. Roles and Responsibility Design and develop scalable data pipelines using Pyspark and Airflow. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data systems using Hadoop and Data Modelling techniques. Troubleshoot and resolve complex technical issues related to data processing and analysis. Implemen...

Looking for a skilled professional with expertise in Spark and Delta Lake to join our team. The ideal candidate will have 7-12 years of experience. Roles and Responsibility Design and implement data pipelines using Spark and Delta Lake as the data storage layer. Develop and optimize Spark applications using Python, Scala, or Java. Work with large datasets to ensure data quality and integrity. Implement data partitioning and schema evolution techniques using Delta Lake. Collaborate with cross-functional teams to integrate Spark applications with ETL processes. Troubleshoot and resolve issues related to Spark application performance and data quality. Job Requirements Strong understanding of Sp...

We are looking for a skilled Databricks & AWS Lead with 7-14 years of experience to join our team in Pune, Mumbai, Chennai, Hyderabad, Bangalore, and Noida. The ideal candidate will have expertise in Databricks and AWS, Unity Catalog, Spark, SQL, and Python. Roles and Responsibility Design and implement data pipelines using Databricks and AWS. Develop and maintain big data technologies and data warehousing solutions. Create CI/CD pipelines using Jenkins or GitLab CI/CD. Collaborate with cross-functional teams to identify and prioritize project requirements. Troubleshoot and resolve technical issues related to Databricks and AWS. Ensure data quality and integrity by implementing data validati...

We are looking for a highly skilled MLOps professional with 0 to 30 days of experience to join our team in Bangalore, Hyderabad, Pune, Chennai, and Kolkata. The ideal candidate will have expertise in MLOps, including Azure/AWS/GCP, Pyspark, and Bigdata. Roles and Responsibility Design and implement scalable data pipelines using MLOps tools. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale machine learning models and algorithms. Ensure seamless integration of MLOps solutions with existing systems and infrastructure. Troubleshoot and resolve complex technical issues related to MLOps. Stay up-to-date with industry trends a...

We are looking for a skilled Hadoop Admin with 4 to 9 years of experience. The ideal candidate will have expertise in Hadoop administration, CDP distribution, Dataflow, and Big data platform. Roles and Responsibility Manage and maintain large-scale Hadoop clusters for high availability and performance. Design and implement data pipelines using Kafka and Nifi profiles. Develop and execute Linux shell scripts for automation and monitoring. Collaborate with cross-functional teams to resolve technical issues and improve processes. Ensure compliance with security best practices and industry standards. Troubleshoot and optimize HBase databases for improved query performance. Job Requirements Stron...

We are looking for a skilled Lead Data Engineer with 5 to 10 years of experience to join our team in Bangalore. The ideal candidate will have expertise in Data Engineering, Python, SQL, AWS, Spark, Hadoop, and Big Data processing frameworks. Roles and Responsibility Design and develop large-scale data pipelines using various data engineering tools and technologies. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain complex data architectures and data warehouses. Ensure data quality and integrity by implementing robust testing and validation procedures. Optimize data processing workflows for improved performance and efficiency. Mentor...

What youll do: Undertake primary ownership in driving self and team effort across all phases of a project lifecycle. Translate business requirements into technical terms and drive team effort to design, build and manage technology solutions that solve business problems. Apply appropriate development methodologies (e.g. agile, waterfall) and best practices (e.g.: mid-development client reviews, embedded QA procedures, unit testing) to ensure successful and timely completion. Partner with Project lead/ Program lead in delivering projects and assist in project management responsibility like - project plans, people management, staffing and risk mitigation. Collaborate with other team members acr...