755 Apache Pig Jobs - Page 3

We are seeking an experienced and highly skilled Senior Data Engineer to join our organization. The successful candidate will demonstrate deep technical expertise in Snowflake, strong analytical and problem-solving abilities, and the capability to manage client interactions with professionalism and clarity. This role requires a proactive individual who can deliver robust data solutions, collaborate effectively across teams, and adapt to evolving technologies. Design, develop, and maintain efficient, scalable data pipelines and solutions using Snowflake. Collaborate with clients and internal stakeholders to understand business requirements and translate them into effective technical solutions...

Role: Databricks Technical Lead/ Data Architect Experience: 8-13 years Location: Bangalore, Chennai, Delhi, Pune, Kolkata About Tredence: Tredence focuses on last-mile delivery of powerful insights into profitable actions by uniting its strengths in business analytics, data science and software engineering. The largest companies across industries are engaging with us and deploying their prediction and optimization solutions at scale. Head quartered in the San Francisco Bay Area, we serve clients in the US, Canada, Europe, and South East Asia. We are seeking an experienced data scientist who apart from the required mathematical and statistical expertise also possesses the natural curiosity an...

Our Purpose Title and Summary Consultant, Advisors & Consulting Services, Performance Analytics Consultant Performance Analytics Advisors & Consulting Services Services within Mastercard is responsible for acquiring, engaging, and retaining customers by managing fraud and risk, enhancing cybersecurity, and improving the digital payments experience. We provide value-added services and leverage expertise, data-driven insights, and execution. Our Advisors & Consulting Services team combines traditional management consulting with Mastercards rich data assets, proprietary platforms, and technologies to provide clients with powerful strategic insights and recommendations. Our teams work with a div...

About The Role Project Role : Application Lead Project Role Description : Lead the effort to design, build and configure applications, acting as the primary point of contact. Must have skills : AWS BigData Good to have skills : NA Minimum 5 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As an Application Lead, you will lead the effort to design, build, and configure applications, acting as the primary point of contact. Your typical day will involve collaborating with various teams to ensure project milestones are met, facilitating discussions to address challenges, and guiding your team in implementing effective solutions. You will also en...

About The Role Project Role : Application Lead Project Role Description : Lead the effort to design, build and configure applications, acting as the primary point of contact. Must have skills : Snowflake Data Warehouse Good to have skills : NA Minimum 5 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As an Application Lead, you will lead the effort to design, build, and configure applications, acting as the primary point of contact. Your typical day will involve collaborating with various teams to ensure that application development aligns with business objectives, overseeing project timelines, and facilitating communication among stakehold...

About The Role Project Role : Application Lead Project Role Description : Lead the effort to design, build and configure applications, acting as the primary point of contact. Must have skills : AWS BigData Good to have skills : NA Minimum 7.5 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As an Application Lead, you will lead the effort to design, build, and configure applications, acting as the primary point of contact. Your typical day will involve collaborating with various teams to ensure project milestones are met, facilitating discussions to address challenges, and guiding your team in implementing effective solutions. You will also ...

As an Application Developer, you will lead IBM into the future by translating system requirements into the design and development of customized systems in an agile environment. The success of IBM is in your hands as you transform vital business needs into code and drive innovation. Your work will power IBM and its clients globally, collaborating and integrating code into enterprise systems. You will have access to the latest education, tools and technology, and a limitless career path with the world’s technology leader. Come to IBM and make a global impact Responsibilities: Responsible to manage end to end feature development and resolve challenges faced in implementing the same Learn new te...

Experience writing custom Kafka Connectors (source and sink). Understanding of Kafka Connect framework, including connector lifecycle, configuration, and error handling. Ability to implement authentication mechanisms within connector code: -OAuth2, Basic Auth, API Keys depending on the target system. Experience managing secrets securely (e.g., using environment variables, vaults etc..). Strong command of Git for code versioning, branching strategies, and pull requests (optional) Deep understanding of Kafka architecture: topics, partitions, producers, consumers, brokers. Experience with KSQL for stream processing Ability to connect Kafka with external systems (e.g., databases, REST APIs, clou...

Experience writing custom Kafka Connectors (source and sink). Understanding of Kafka Connect framework, including connector lifecycle, configuration, and error handling. Ability to implement authentication mechanisms within connector code: -OAuth2, Basic Auth, API Keys depending on the target system. Experience managing secrets securely (e.g., using environment variables, vaults etc..). Strong command of Git for code versioning, branching strategies, and pull requests (optional) Deep understanding of Kafka architecture: topics, partitions, producers, consumers, brokers. Experience with KSQL for stream processing Ability to connect Kafka with external systems (e.g., databases, REST APIs, clou...

Myridius x Aethereus is looking for Data Engineer to join our dynamic team and embark on a rewarding career journey Liaising with coworkers and clients to elucidate the requirements for each task. Conceptualizing and generating infrastructure that allows big data to be accessed and analyzed. Reformulating existing frameworks to optimize their functioning. Testing such structures to ensure that they are fit for use. Preparing raw data for manipulation by data scientists. Detecting and correcting errors in your work. Ensuring that your work remains backed up and readily accessible to relevant coworkers. Remaining up-to-date with industry standards and technological advancements that will impro...

Responsibilities What you'll do Engineer, test, document and manage GCP Dataproc, DataFlow and VertexAI services used in high-performance data processing pipelines and Machine Learning. Help developers optimize data processing jobs using Spark, Python, and Java. Collaborate with development teams to integrate data processing pipelines with other cloud services and applications. Utilize Terraform and Tekton for infrastructure as code (IaC) and CI/CD pipelines, ensuring efficient deployment and management. Good to have Experience with Spark for large-scale data processing. Solid understanding and experience with GitHub for version control and collaboration. Experience with Terraform for infras...

Skills : Spark, Scala, Unix, Shell Scripting, GCP - BigQuery, BigTable, DataProc, AirflowManual testing All ETL jobs are written in Scala, preferably Scala with Spark will help supporting and migrating to Lumi also helps.(No python for this Roles and Responsibilities Skills : Spark, Scala, Unix, Shell Scripting, GCP - BigQuery, BigTable, DataProc, AirflowManual testing All ETL jobs are written in Scala, preferably Scala with Spark will help supporting and migrating to Lumi also helps.(No python for this

SUMMARY Working knowledge on big data technologies should be more than 6. Increase the no. of years for Sr profiles Excellent working knowledge on SQL Proficient with spark and any one or both programming language e.g. Scala and/or Python Demonstrated working knowledge with kafka, Airflow, DBT, Flink, Apache Iceberg, Datadog Experience with snowflake will be a big positive Experience with CI/CD framework implementation in conjunction with any popular code repository like Github

Education : Bachelors or Masters in Computer Science or an equivalent area Skill-sets At least 5+ years of experience working in search infrastructure software engineering role. Proven leadership experience (e.g. as tech lead, lead architect or similar positions). Experience with Large, Distributed infrastructures. Especially the Hadoop stack and NoSQL Experience with Cassandra, Chef, AWS, and Jenkins a plus (not all are required). Ability to deal with large datasets and optimize query performance Solid understanding of the full web technology stack (e.g. from browser to data source). Solid understanding of computer science principles (OOP, Design patterns, algorithms, data structures etc.) ...

Analyze existing Hadoop, Pig, and Spark scripts from Dataproc and refactor them into Databricks-native PySpark. Implement data ingestion and transformation pipelines using Delta Lake best practices. Apply conversion rules and templates for automated code migration and testing. Conduct data validation between legacy and migrated environments (schema, count, and data-level checks). Collaborate on developing AI-driven tools for code conversion, dependency extraction, and error remediation. Ensure best practices for code versioning, error handling, and performance optimization. Participate in UAT, troubleshooting, and post-migration validation activities. Technical Skills Core: Python, PySpark, ...

Lead design and execution of Dataproc Databricks PySpark migration roadmap. Define modernization strategy , including data ingestion, transformation, orchestration, and governance. Architect scalable Delta Lake and Unity Catalog -based solutions. Manage and guide teams on code conversion, dependency mapping, and data validation. Collaborate with platform, infra, and DevOps teams to optimize compute costs and performance. Own the automation & GenAI acceleration layer , integrating code parsers, lineage tools, and validation utilities. Conduct performance benchmarking, cost optimization, and platform tuning (Photon, Auto-scaling, Delta Caching). Mentor senior and mid-level developers, ensuring...

JD Candidate should have hands on experience on Hadoop and can have little experience on snowflake. 5 7 years of experience with the technical analysis and design, development and implementation of data warehousing / Data Lake solutions. Well versed and Hand on in Hadoop, Hive Query language (HQL), Python and SQL Hadoop, SQL, PL/SQL and unix shell scripting Strong UNIX Shell scripting experience to support data warehousing solutions. Process oriented, focused on standardization, streamlining, and implementation of best practices delivery approach. Excellent problem solving and analytical skills. Excellent verbal and written communication skills.

Data Engineer (Hadoop, Spark/Scala, HBase, Kafka) Responsibilities: Gather and understand data requirements, build complex data pipelines, and work to achieve high quality data ingestion goals. Drive the development of Hadoop based data warehouses, data ingestion, and data profiling activities. Analyzing, re architecting, and re platforming on premise datalake if needed Implementation and maintenance of standard data/technology deployment workflow to ensure that all deliverables/enhancements are delivered in a disciplined and robust manner. Highly self motivated and able to work independently as well as in a team environment. Required Skills: Proficiency with Spark Scala, Python language (ma...

Description: Experience in following technologies: Hadoop Spark Hive Pig SQL query experience in data application development at scale in the Hadoop ecosystem. Bigdata Spark kafka Java/python/scala Hadoop hive design and architecture problem solving skills Roles and responsibilities Drive the technical design of large scale data platforms utilizing modern and open source technologies. Set standards for data engineering functions; design templates for the data management program which are scalable repeatable and simple Serve as a consultant on all aspects of data engineering analytics and data projects Be able to consult on complex data engineering efforts and lead project teams through the s...

Location Chennai and Hyderbad preferred but customer is willing to take resources from Hyderabad Experience 5 to 8 yrs ( U3 ). Exp - 5- 10 Yrs Location - Hyderabad / Chennai Location Proven experience as a development data engineer or similar role, with ETL background. Experience with data integration / ETL best practices and data quality principles. Play a crucial role in ensuring the quality and reliability of the data by designing, implementing, and executing comprehensive testing. By going over the User Stories build the comprehensive code base and business rules for testing and validation of the data. Knowledge of continuous integration and continuous deployment (CI/CD) pipelines. Famil...

Description Detailed JD RTIM Pega CDH 8.8 Multi App Infinity 24.1 Java Restful API oAuth 1. Understanding the NBA requirements and the complete CDH architecture 2. Review of the Conceptual Design Detailed Design and estimations. 3. Reviewing and contributing to the deployment activities and practices 4. Contributing to overall technical solution and putting it to practise. 5. Contributing to the requirement discussion with the Subject matter expertise in relation to Pega CDH 6. Experience in Pega CDH v8.8 multi app or 24.1 and retail banking domain is preferred. 7. Conducting peer code reviews 8. Excellent communications skills Named Job Posting? (if Yes - needs to be approved by SCSC) Addit...

Description Overall 10+ years of experience in Python and Shell Knowledge of distributed systems like Hadoop and Spark as well as cloud computing platforms such as Azure and AWS Named Job Posting? (if Yes - needs to be approved by SCSC) Additional Details Global Grade B Level To Be Defined Named Job Posting? (if Yes - needs to be approved by SCSC) No Remote work possibility No Global Role Family To be defined Local Role Name To be defined Local Skills Ruby;automation;Python Languages RequiredENGLISH Role Rarity To Be Defined

EXP 4 to 6 yrs Location Any PSL Location Rate below 14$ JD - DBT/AWS Glue/Python/Pyspark Hands-on experience in data engineering, with expertise in DBT/AWS Glue/Python/Pyspark. Strong knowledge of data engineering concepts, data pipelines, ETL/ELT processes, and cloud data environments (AWS) Technology DBT, AWS Glue, Athena, SQL, Spark, PySpark Good understanding of Spark internals and how it works. Goot skills in PySpark Good understanding of DBT basically should be to understand DBT limitations and when it will end-up in model explosion Good hands-on experience in AWS Glue AWS expertise should know different services and should know how to configure them and infra-as-code experience Basic ...

Role Purpose The purpose of this role is to design, test and maintain software programs for operating systems or applications which needs to be deployed at a client end and ensure its meet 100% quality assurance parameters Do 1. Instrumental in understanding the requirements and design of the product/ software Develop software solutions by studying information needs, studying systems flow, data usage and work processes Investigating problem areas followed by the software development life cycle Facilitate root cause analysis of the system issues and problem statement Identify ideas to improve system performance and impact availability Analyze client requirements and convert requirements to fe...

6-7+ years of Experience in Hadoop or any Cloud Bigdata components (specific to the Data Engineering role), Hadoop hands on (airflow, oozie, hive, hdfs, sqoop, pig, map, reduce ) 4+ years of Exp in Spark (Spark Batch, Streaming, Mlibetc). Candidates should possess proficiency in utilizing the Apache Spark framework. 6-7+ years of experience in Python programming language. 4+ years of experience with pyspark data transformation(json,csv,rdbms,stream) pipeline design , development and deployment with kubernate/onprem platform (not cloud based). 2+ years of experience in designing and implementing data workflows with Apache Airflow. Kafka or equivalent Cloud Bigdata components (specific to the ...