453 Data Engineer Jobs - Page 12

Key Skill: Pyspark, Cloudera Data Platfrorm, Big data Hadoop, Hive, Kafka Responsibilities: Data Pipeline Development: Design, develop, and maintain highly scalable and optimized ETL pipelines using PySpark on the Cloudera Data Platform, ensuring data integrity and accuracy. Data Ingestion: Implement and manage data ingestion processes from a variety of sources (e.g., relational databases, APIs, file systems) to the data lake or data warehouse on CDP. Data Transformation and Processing: Use PySpark to process, cleanse, and transform large datasets into meaningful formats that support analytical needs and business requirements. Performance Optimization: Conduct performance tuning of PySpark code and Cloudera components, optimizing resource utilization and reducing runtime of ETL processes. Data Quality and Validation: Implement data quality checks, monitoring, and validation routines to ensure data accuracy and reliability throughout the pipeline. Automation and Orchestration: Automate data workflows using tools like Apache Oozie, Airflow, or similar orchestration tools within the Cloudera ecosystem. Candidate’s Profile & Technical Competency: BE/B Tech, MCA/BCA with 4+ years of experience as a Data Engineer, with a strong focus on PySpark and the Cloudera Data Platform PySpark: Advanced proficiency in PySpark, including working with RDDs, DataFrames, and optimization techniques. Cloudera Data Platform : Strong experience with Cloudera Data Platform (CDP) components, including Cloudera Manager, Hive, Impala, HDFS, and HBase. Data Warehousing: Knowledge of data warehousing concepts, ETL best practices, and experience with SQL-based tools (e.g., Hive, Impala). Big Data Technologies : Familiarity with Hadoop, Kafka, and other distributed computing tools. Orchestration and Scheduling: Experience with Apache Oozie, Airflow, or similar orchestration frameworks. Scripting and Automation : Strong scripting skills in Linux. Ready for 6 months contract role at Chennai Can you join 15 days

Senior Data Engineer Aveva PI Specialist Location: Hyderabad Experience: 8+ years Job Type: Full-Time Industry: Pharmaceuticals / Biotech / Manufacturing Work Mode: Hybrid Job Summary: We are looking for a Senior Data Engineer with deep expertise in Aveva PI solutions to join our organization. This critical role involves leading the implementation and governance of Aveva PI across multiple manufacturing sites while driving its strategic adoption within our Center of Excellence (CoE) . The ideal candidate will bring a balance of strong technical skills, industrial domain knowledge, and experience in data governance to optimize real-time data solutions. Key Responsibilities: Lead end-to-end implementation of Aveva PI across manufacturing plants. Install, configure, and validate PI Servers and data connectivity (e.g., OPC, PI Cloud Connect, PI WEB API, RDBMS, UFL ). Design and build AF structure, Event Frames (EF), PI Analytics, Notifications , and develop data architecture for collection, aggregation, and visualization. Drive the strategic vision and adoption of Aveva PI through the CoE. Establish and maintain governance frameworks and compliance standards for Aveva PI usage. Collaborate with cross-functional teams to gather requirements and implement robust system architectures. Develop and maintain technical documentation , best practices, SOPs, and training resources. Ensure high availability and performance of Aveva PI systems through proactive monitoring and support. Lead cross-functional forums to promote knowledge sharing , innovation, and continuous improvement. Required Skills & Experience: 8+ years of hands-on experience with Aveva PI, including full-cycle implementations. In-depth knowledge of PI System components : PI Server, PI AF, PI Vision, PI Interfaces, PI Analytics. Solid understanding of industrial automation , process data integration , and ERP/MES system interactions . Experience with GMP environments , including creation of qualification and compliance documentation. Strong scripting and data skills: SQL, Python or similar. Familiarity with cloud technologies and data lake integration with PI data. Proficiency in data governance and OT systems best practices . Excellent communication and leadership skills to guide stakeholders and lead forums. Experience in Agile delivery environments and working in or establishing Centers of Excellence (CoE) . Preferred Industry Experience: Pharmaceuticals Biotech Chemicals / Manufacturing Why Join Us? Be a key part of a strategic digital transformation initiative Work with cutting-edge PI and cloud technologies Lead innovation in real-time industrial data systems Opportunity to shape and grow a Center of Excellence Apply Now on minal_mohurle@persolapac.com to drive operational excellence through data! CONFIDENTIAL NOTE: By submitting your resume or personal data, you acknowledge reading and agreeing to our Privacy Policy . You hereby provide voluntary consent to the collection, use, processing, and disclosure of your data by us and our affiliates, in line with the Privacy Policy . and applicable laws. If you wish to withdraw your consent or have any concerns, you may submit a request to our designated consent manager, as outlined in our Privacy Policy . We prioritize your privacy. SECURITY NOTE: We at PERSOLKELLY India or our representatives, do not ask job seekers for fees, personal banking information, or payments through unofficial channels. Official communications will only come from @persolkelly.com. Report any suspicious activity to [Contactus_in@persolkelly.com]Contactus_in@persolkelly.com . Click here to find out how you can safeguard yourself from job scams..

Key Skills: Python, Data Engineer, SQL Roles & Responsibilities: Design, develop, test, deploy, maintain and improve software. Develop flowcharts, layouts and documentation to identify requirements & solutions. Write well designed & high-quality testable code. Produce specifications and determine operational feasibility. Integrate software components into fully functional platform. Apply pro-actively & perform hands-on design and implementation of best practice CI/CD. Coaching & mentoring of other Service Team members. Develop/contribute to software verification plans and quality assurance procedures. Document and maintain software functionality. Troubleshoot, debug and upgrade existing systems, including participating in DR tests. Deploy programs and evaluate customer feedback. Contribute to team estimation for delivery and expectation management for scope. Comply with industry standards and regulatory requirements. Ensuring throughout project life cycle, the data component of the solution i.e. data management (data generation, standardisation, Integration, transformation, migration, delivery, etc) is aligned with the Data Architecture pattern/ principles/policies and working closely with the solution designers. Identification and escalation of any potential gaps in the target data architecture, including likely impact of gaps. Responsible for ensuring the Data design will fit for business purpose and deliver expected benefits with minimal risk/data impact to the other business unit across the enterprise, who are users of the specific data. In addition, the data engineer to ensure the design caters for potential future usage of data by customers across the enterprise. Experience Requirments: 3-8 years technical experience (within financial services industry preferred). Technical Domain experience (Subject Matter Expertise in Technology or Tools). Solid experience, knowledge and skills in Data Engineering, BI/software development such as ELT/ETL, data extraction and manipulation in Data Lake/Data Warehouse/Lake House environment. Hands on programming experience in writing Python, SQL, Unix Shell scripts, in a complex enterprise environment. Experience in configuration management using Ansible/Jenkins/GIT. Hands on experience working on relational Databases - Oracle or MS SQL or DB2 (SQL/PLSQL, Database design, Normalisation, Execution plan analysis, Index creation and maintenance, Stored Procedures). Knowledge of Data Modelling techniques and practices used for a Data Warehouse/Data Mart application. Quality engineering development experience (CI/CD - Jenkins). Experience with Source Control Tools - Github or BitBucket. Skilled in querying data from a range of data sources that store structured and unstructured data. Desired Experience: Hands on expertise in SQL and PL/SQL in Oracle pref. 19c. Hands on exposure on Bash/Shell scripting & Control M for job scheduling Experience with Source code control - GitHub, VSTS etc. Experience with UNIX command-line tools for complex file operations Understanding and ability to translate/physicalise Data Models (Star Schema, Data Vault 2.0 etc). Data integration experience in complex multi asset/multi-platform/ distributed technology environments involving data migration strategy, decommission plans, etc. Understanding of data management concepts, Data Architecture Principles, Data Controls, Data Policies and Standards and possible causes of Data risks/impacts in the organisation. Good to Have Exposure to Big Data (Hive-QL/DataBricks/Redshift SQL). Exposure on python scripting. Exposure to Cloud/AWS service including S3, Glue, Athena, Data Pipeline, etc. Knowledge or understanding of Power BI. Essential Capabilities Enthusiasm for technology, keeping up with latest trends. Ability to articulate complex technical issues and desired outcomes of system enhancements. Proven analytical skills and evidence-based decision making. Excellent problem solving, troubleshooting & documentation skills. Strong written and verbal communication skills. Excellent collaboration and interpersonal skills. Strong delivery focus with an active approach to quality and auditability. Ability to work under pressure and excel within a fast-paced environment Ability to self-manage tasks. Agile software development practices. Qualification: UG / PG in Computer Science.

• Act as Data domain expert for Snowflake in a collaborative environment to provide demonstrated understanding of data management best practices and patterns. • Design and implement robust data architectures to meet and support business requirements leveraging Snowflake platform capabilities. • Develop and enforce data modelling standards and best practices for Snowflake environments. • Develop, optimize, and maintain Snowflake data warehouses. • Leverage Snowflake features such as clustering, materialized views, and semi structured data processing to enhance data solutions. • Ensure data architecture solutions meet performance, security, and scalability requirements. • Stay current with the latest developments and features in Snowflake and related technologies, continually enhancing our data capabilities. • Collaborate with cross-functional teams to gather business requirements, translate them into effective data solutions in Snowflake and provide data-driven insights. • Stay updated with the latest trends and advancements in data architecture and Snowflake technologies. • Provide mentorship and guidance to junior data engineers and architects. • Troubleshoot and resolve data architecture-related issues effectively. Skills Requirement: • 5+ years of proven experience as a Data Engineer with 3+ years as Data Architect. • Proficiency in Snowflake with Hands-on experience with Snowflake features such as clustering, materialized views, and semi-structured data processing. • Experience in designing and building manual or auto ingestion data pipeline using Snowpipe. • Design and Develop automated monitoring processes on Snowflake using combination of Python, PySpark, Bash with SnowSQL. • SnowSQL Experience in developing stored Procedures writing Queries to analyse and transform data • Working experience on ETL tools like Fivetran, DBT labs, MuleSoft • Expertise in Snowflake concepts like setting up Resource monitors, RBAC controls, scalable virtual warehouse, SQL performance tuning, zero copy clone, time travel and automating them. • Excellent problem-solving skills and attention to detail. • Effective communication and collaboration abilities. • Relevant certifications (e.g., SnowPro Core / Advanced) are a must have. • Must have expertise in AWS. Azure, Salesforce Platform as a Service (PAAS) model and its integration with Snowflake to load/unload data. • Strong communication and exceptional team player with effective problem-solving skills Educational Qualification Required: • Masters degree in Business Management (MBA / PGDM) / Bachelor's degree in computer science, Information Technology, or related field.

Job Title: Senior Data Engineer ADF | Snowflake | DBT | Databricks Experience: 5 to 8 Years Locations: Pune / Hyderabad / Gurgaon / Bangalore (Hybrid) Job Type: Full Time, Permanent Job Description: We are hiring for a Senior Data Engineer role with strong expertise in Azure Data Factory (ADF) , Snowflake , DBT , and Azure Databricks . The ideal candidate will be responsible for designing, building, and maintaining scalable cloud-based data pipelines and enabling high-quality data delivery for analytics and reporting. Key Responsibilities Build and manage ETL/ELT pipelines using ADF, Snowflake, DBT, and Databricks Create parameterized, reusable components within ADF pipelines Perform data transformations and modeling in Snowflake using DBT Use Databricks for data processing using PySpark/SQL Collaborate with stakeholders to define and implement data solutions Optimize data workflows for performance, scalability , and cost-efficiency Ensure data quality, governance, and documentation standards Mandatory Skills Azure Data Factory (ADF) Snowflake DBT (Data Build Tool) Azure Databricks Strong SQL and data modeling experience Good-to-Have Skills Azure Data Lake, Azure Synapse, Blob Storage CI/CD using Azure DevOps or GitHub Python scripting, PySpark Power BI/Tableau integration Experience in metadata/data governance tools Role Requirements Education : Bachelors/Masters degree in Computer Science, Data Engineering, or related field Certifications : Azure or Snowflake certification is a plus Strong problem-solving and communication skills Keywords: Azure Data Factory, ADF, Snowflake, DBT, Azure Databricks, PySpark, SQL, Data Engineer, Azure Data Lake, ETL, ELT, Azure Synapse, Power BI, CI/CD

AZURE DATA ENGINEER Skills - Strong technical experience in Azure, SQL , Azure data factory, ETL, Databricks Graduation must Experience- 5-10 years CTC- Up to 14 - 20 LPA 21st June -F2F Interview only (Pune) Contact- 7742324144

Design, develop, and maintain scalable data pipelines and ETL processes. Build and optimize data architecture to ensure data quality and consistency. Integrate data from diverse internal and external sources. Collaborate with cross-functional teams. Required Candidate profile 2 years of professional experience in data engineering or software development. Solid understanding of SQL and proficiency in at least one programming language, such as Python, Java, or Scala.

Job Description: We are seeking a talented and experienced Data Scientist to join our dynamic team. The ideal candidate will have a strong background in data analysis, machine learning, statistical modeling, and artificial intelligence. Experience with Natural Language Processing (NLP) is desirable. Experience delivering products that incorporate AI/ML, familiarity with Cloud Services such as AWS highly desirable. Key Responsibilities: Clean, prepare, and explore data to find trends and patterns Build, validate, and implement AI/ML models Extensively document all aspects of the work including data analysis, model development, results Collaborate with other team members teams to incorporate AI/ML models into software applications Stay updated with the latest advancements in AI/ML domain and incorporate into day-to-day work Required Skills/Qualifications: 3-5 years of experience in AI/ML related work Extensive experience in Python Familiarity with Statistical models such as Linear/Logistic regression, Bayesian Models, Classification/Clustering models, Time Series analysis Experience with deep learning models such as CNNs, RNNs, LSTM, Transformers Experience with machine learning frameworks such as TensorFlow, PyTorch, Scikit- learn, Keras Experience with GenAI, LLMs, RAG architecture would be a plus Familiarity with cloud services such as AWS, Azure Familiarity with version control systems (e.g., Git), JIRA, Confluence Familiarity with MLOPs concepts, AI/ML pipeline tooling such as Kedro Knowledge of CI/CD pipelines and DevOps practices Experience delivering customer facing AI Solutions delivered as SaaS would be a plus Bachelors degree in Computer Science, Engineering, or a related field, or equivalent practical experience. Strong problem-solving skills and attention to detail Excellent verbal and written communication and teamwork skills Benefits: Competitive salary and benefits package Opportunity to work on cutting-edge technologies and innovative projects Collaborative and inclusive work environment Professional development and growth opportunities

•Strong experience as an AWS/Azure/GCP Data Engineer & must have AWS/Azure/GCP Databricks experience. •Expert proficiency in Spark Scala, Python, spark,ADF & SQL •Design & develop applications on Databricks. NP-Immediate Email- sachin@assertivebs.com

About Client Hiring for One of the Most Prestigious Multinational Corporations Job Title: Senior GCP Data Engineer Experience: 8 to 13 years Key Responsibilities : Design, build, and maintain scalable and reliable data pipelines on Google Cloud Platform (GCP) . Develop ETL/ELT workflows using Cloud Dataflow , Apache Beam , Dataproc , BigQuery , and Cloud Composer (Airflow). Optimize performance of data processing and storage solutions (e.g., BigQuery, Cloud Storage). Collaborate with data analysts, data scientists, and business stakeholders to deliver data-driven insights. Design and implement data lake and data warehouse solutions following best practices. Ensure data quality, security, and governance across GCP environments. Implement CI/CD pipelines for data engineering workflows using tools like Cloud Build , GitLab CI , or Jenkins . Monitor and troubleshoot data jobs, ensuring reliability and timeliness of data delivery. Mentor junior engineers and participate in architectural design discussions. Technical Skills: Strong experience in Google Cloud Platform (GCP) data services: BigQuery , Dataflow , Dataproc , Pub/Sub , Cloud Storage , Cloud Functions Proficiency in Python and/or Java for data processing. Strong knowledge of SQL and performance tuning in large-scale environments. Hands-on experience with Apache Beam , Apache Spark , and Airflow . Solid understanding of data modeling , data warehousing , and streaming/batch processing . Experience with CI/CD , Git, and modern DevOps practices for data workflows. Familiarity with data security and compliance in cloud environments. NOTE : Only immediate and 15 days joiners Notice period : Only immediate and 15 days joiners Location: Pune, Chennai. Hyderabad, Kolkata Mode of Work : WFO(Work From Office) Thanks & Regards, SWETHA Black and White Business Solutions Pvt.Ltd. Bangalore,Karnataka,INDIA. Contact Number:8067432433 rathy@blackwhite.in |www.blackwhite.in

Role & responsibilities Description: 6+ years of experience in data engineering, specifically in cloud environments like AWS. Proficiency in Python and PySpark for data processing and transformation tasks. Solid experience with AWS Glue for ETL jobs and managing data workflows. Hands-on experience with AWS Data Pipeline (DPL) for workflow orchestration. Strong experience with AWS services such as S3, Lambda, Redshift, RDS, and EC2. Technical Skills : Deep understanding of ETL concepts and best practices.. Strong knowledge of SQL for querying and manipulating relational and semi-structured data. Experience with Data Warehousing and Big Data technologies, specifically within AWS. Additional Skills : Experience with AWS Lambda for serverless data processing and orchestration. Understanding of AWS Redshift for data warehousing and analytics. Familiarity with Data Lakes, Amazon EMR, and Kinesis for streaming data processing. Knowledge of data governance practices, including data lineage and auditing. Familiarity with CI/CD pipelines and Git for version control. Experience with Docker and containerization for building and deploying applications. Design and Build Data Pipelines: Design, implement, and optimize data pipelines on AWS using PySpark, AWS Glue, and AWS Data Pipeline to automate data integration, transformation, and storage processes. ETL Development: Develop and maintain Extract, Transform, and Load (ETL) processes using AWS Glue and PySpark to efficiently process large datasets. Data Workflow Automation: Build and manage automated data workflows using AWS Data Pipeline, ensuring seamless scheduling, monitoring, and management of data jobs. Data Integration: Work with different AWS data storage services (e.g., S3, Redshift, RDS) to ensure smooth integration and movement of data across platforms.Optimization and Scaling: Optimize and scale data pipelines for high performance and cost efficiency, utilizing AWS services like Lambda, S3, and EC2.

Role & responsibilities Strong, hands-on proficiency with Snowflake: In-depth knowledge of Snowflake architecture, features (e.g., Snowpipe, Tasks, Streams, Time Travel, Zero-Copy Cloning). Experience in designing and implementing Snowflake data models (schemas, tables, views). Expertise in writing and optimizing complex SQL queries in Snowflake. Experience with data loading and unloading techniques in Snowflake. Solid experience with AWS Cloud services: Proficiency in using AWS S3 for data storage, staging, and as a landing zone for Snowflake. Experience with other relevant AWS services (e.g., IAM for security, Lambda for serverless processing, Glue for ETL - if applicable). Strong experience in designing and building ETL/ELT data pipelines.

Key Skills: Data Engineer, Python. Roles and Responsibilities: Develop and maintain scalable data pipelines using Python and PySpark. Design and implement data lake and data warehouse solutions to support business intelligence and analytics needs. Work extensively on the Databricks platform for data processing and transformation. Write complex SQL queries and build efficient data models to support analytics and reporting. Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions. Ensure data quality, consistency, and reliability across various sources and destinations. Troubleshoot and resolve issues in data ingestion, transformation, and delivery processes. Lead and mentor junior data engineers, ensuring adherence to best practices and coding standards. Experience Requirement: 3-8 years of experience with data warehousing and data lake architectures. Extensive hands-on experience with the Databricks platform. Proven expertise in SQL and data modeling. Strong proficiency in Python and PySpark. Excellent problem-solving and analytical skills. Demonstrated experience in leading and mentoring teams. Education: Any Graduation.

Role & responsibilities Data engineer with architect level experience in ETL, AWS (Glue), Pyspark, Python etc Preferred candidate profile Immediate joiners who can work on Contract basis If you are interested please share your updated CV at pavan.teja@careernet.in

Experience Data Engineer ((Python, PySpark, ADB,ADF, Azure, Snowflake) Data science can also apply

Roles and Responsibilities: We are looking for an experienced AWS Cloud Data Engineer to join our Data Science & Analytics team to build, optimize, and maintain cloud-based data solutions. The ideal candidate will possess strong technical knowledge in data engineering on AWS, expertise in data integration, pipeline creation, performance optimization, and a strong understanding of DevOps methodologies. Design, develop, and deploy scalable, high-performance data pipelines on AWS and scalable AWS infrastructure solutions. Implement data solutions utilizing AWS services such as S3, Glue, Redshift, EMR, Athena, and Kinesis. Optimize data storage, processing, and query performance to ensure efficiency and reliability. Maintain and enhance ETL processes, including data extraction, transformation, and loading using AWS Glue and Lambda. Ensure data quality, security, compliance, and governance are integrated throughout data workflows. Collaborate closely with data scientists, analysts, and application developers to meet data needs. Monitor and troubleshoot data pipelines and infrastructure proactively. Document data architectures, processes, and standard operating procedures. Document cloud architectures, procedures, and processes clearly and comprehensively. Required Qualifications: Bachelors degree in Computer Science, IT, or a related technical field. 3-5+ years of experience working as a Data Engineer, particularly with AWS cloud infrastructure. AWS Certified Data Engineering - Specialty or similar certifications preferred. Proficiency in AWS data services including S3, Glue, Lambda, Redshift, Athena, Kinesis, and EMR. Strong expertise in building data pipelines using Python, PySpark, or SQL. Experience with big data technologies and frameworks (e.g., Hadoop, Spark). Demonstrable skills with infrastructure-as-code tools like Terraform or CloudFormation. Experience with containerization technologies like Docker and Kubernetes Preferred Qualifications: Familiarity with data lake architectures and Lakehouse implementations. Knowledge of container technologies (Docker, Kubernetes). Experience with data visualization and reporting tools (QuickSight, Tableau, Power BI). Understanding of DevOps methodologies, CI/CD pipelines, and Agile development practices. Competencies: Analytical mindset with a keen attention to detail. Strong problem-solving and troubleshooting capabilities. Excellent collaboration and communication skills. Proactive learner with a commitment to continuous professional development.

Roles and Responsibilities: We are looking for an experienced AWS Cloud Data Engineer to join our Data Science & Analytics team to build, optimize, and maintain cloud-based data solutions. The ideal candidate will possess strong technical knowledge in data engineering on AWS, expertise in data integration, pipeline creation, performance optimization, and a strong understanding of DevOps methodologies. Design, develop, and deploy scalable, high-performance data pipelines on AWS and scalable AWS infrastructure solutions. Implement data solutions utilizing AWS services such as S3, Glue, Redshift, EMR, Athena, and Kinesis. Optimize data storage, processing, and query performance to ensure efficiency and reliability. Maintain and enhance ETL processes, including data extraction, transformation, and loading using AWS Glue and Lambda. Ensure data quality, security, compliance, and governance are integrated throughout data workflows. Collaborate closely with data scientists, analysts, and application developers to meet data needs. Monitor and troubleshoot data pipelines and infrastructure proactively. Document data architectures, processes, and standard operating procedures. Document cloud architectures, procedures, and processes clearly and comprehensively. Required Qualifications: Bachelors degree in Computer Science, IT, or a related technical field. 3-5+ years of experience working as a Data Engineer, particularly with AWS cloud infrastructure. AWS Certified Data Engineering - Specialty or similar certifications preferred. Proficiency in AWS data services including S3, Glue, Lambda, Redshift, Athena, Kinesis, and EMR. Strong expertise in building data pipelines using Python, PySpark, or SQL. Experience with big data technologies and frameworks (e.g., Hadoop, Spark). Demonstrable skills with infrastructure-as-code tools like Terraform or CloudFormation. Experience with containerization technologies like Docker and Kubernetes Preferred Qualifications: Familiarity with data lake architectures and Lakehouse implementations. Knowledge of container technologies (Docker, Kubernetes). Experience with data visualization and reporting tools (QuickSight, Tableau, Power BI). Understanding of DevOps methodologies, CI/CD pipelines, and Agile development practices. Competencies: Analytical mindset with a keen attention to detail. Strong problem-solving and troubleshooting capabilities. Excellent collaboration and communication skills. Proactive learner with a commitment to continuous professional development.

Roles and Responsibilities: We are looking for an experienced AWS Cloud Data Engineer to join our Data Science & Analytics team to build, optimize, and maintain cloud-based data solutions. The ideal candidate will possess strong technical knowledge in data engineering on AWS, expertise in data integration, pipeline creation, performance optimization, and a strong understanding of DevOps methodologies. Design, develop, and deploy scalable, high-performance data pipelines on AWS and scalable AWS infrastructure solutions. Implement data solutions utilizing AWS services such as S3, Glue, Redshift, EMR, Athena, and Kinesis. Optimize data storage, processing, and query performance to ensure efficiency and reliability. Maintain and enhance ETL processes, including data extraction, transformation, and loading using AWS Glue and Lambda. Ensure data quality, security, compliance, and governance are integrated throughout data workflows. Collaborate closely with data scientists, analysts, and application developers to meet data needs. Monitor and troubleshoot data pipelines and infrastructure proactively. Document data architectures, processes, and standard operating procedures. Document cloud architectures, procedures, and processes clearly and comprehensively. Required Qualifications: Bachelors degree in Computer Science, IT, or a related technical field. 3-5+ years of experience working as a Data Engineer, particularly with AWS cloud infrastructure. AWS Certified Data Engineering - Specialty or similar certifications preferred. Proficiency in AWS data services including S3, Glue, Lambda, Redshift, Athena, Kinesis, and EMR. Strong expertise in building data pipelines using Python, PySpark, or SQL. Experience with big data technologies and frameworks (e.g., Hadoop, Spark). Demonstrable skills with infrastructure-as-code tools like Terraform or CloudFormation. Experience with containerization technologies like Docker and Kubernetes Preferred Qualifications: Familiarity with data lake architectures and Lakehouse implementations. Knowledge of container technologies (Docker, Kubernetes). Experience with data visualization and reporting tools (QuickSight, Tableau, Power BI). Understanding of DevOps methodologies, CI/CD pipelines, and Agile development practices. Competencies: Analytical mindset with a keen attention to detail. Strong problem-solving and troubleshooting capabilities. Excellent collaboration and communication skills. Proactive learner with a commitment to continuous professional development.

Skill required : Data Engineers- Azure Designation : Sr Analyst/ Consultant Job Location : Bengaluru Qualifications: BE/BTech Years of Experience : 4 - 11 Years OVERALL PURPOSE OF JOB Understand client requirements and build ETL solution using Azure Data Factory, Azure Databricks & PySpark . Build solution in such a way that it can absorb clients change request very easily. Find innovative ways to accomplish tasks and handle multiple projects simultaneously and independently. Works with Data & appropriate teams to effectively source required data. Identify data gaps and work with client teams to effectively communicate the findings to stakeholders/clients. Responsibilities : Develop ETL solution to populate Centralized Repository by integrating data from various data sources. Create Data Pipelines, Data Flow, Data Model according to the business requirement. Proficient in implementing all transformations according to business needs. Identify data gaps in data lake and work with relevant data/client teams to get necessary data required for dashboarding/reporting. Strong experience working on Azure data platform, Azure Data Factory, Azure Data Bricks. Strong experience working on ETL components and scripting languages like PySpark, Python . Experience in creating Pipelines, Alerts, email notifications, and scheduling jobs. Exposure on development/staging/production environments. Providing support in creating, monitoring and troubleshooting the scheduled jobs. Effectively work with client and handle client interactions. Skills Required: Bachelors' degree in Engineering or Science or equivalent graduates with at least 4-11 years of overall experience in data management including data integration, modeling & optimization. Minimum 4 years of experience working on Azure cloud, Azure Data Factory, Azure Databricks. Minimum 3-4 years of experience in PySpark, Python, etc. for data ETL . In-depth understanding of data warehouse, ETL concept and modeling principles. Strong ability to design, build and manage data. Strong understanding of Data integration. Strong Analytical and problem-solving skills. Strong Communication & client interaction skills. Ability to design database to store huge data necessary for reporting & dashboarding. Ability and willingness to acquire knowledge on the new technologies, good analytical and interpersonal skills with ability to interact with individuals at all levels.

Architect & Build Scalable Systems: Design and implement a petabyte-scale lakehouse Architectures to unify data lakes and warehouses. Real-Time Data Engineering: Develop and optimize streaming pipelines using Kafka, Pulsar, and Flink. Required Candidate profile Data engineering experience with large-scale systems• Expert proficiency in Java for data-intensive applications. Handson experience with lakehouse architectures, stream processing, & event streaming

Warm Greetings from SP Staffing!! Role: AWS Data Engineer Experience Required :4 to 6 yrs Work Location :Bangalore/Pune/Hyderabad/Chennai Required Skills, Pyspark AWS Glue Interested candidates can send resumes to nandhini.spstaffing@gmail.com

Education: Bachelors or masters degree in computer science, Information Technology, Engineering, or a related field. Experience: 6-10 years 8+ years of experience in data engineering or a related field. Strong hands-on experience with Azure Databricks , Spark , Python/Scala, CICD, Scripting for data processing. Experience working in multiple file formats like Parquet , Delta , and Iceberg . Knowledge of Kafka or similar streaming technologies for real-time data ingestion. Experience with data governance and data security in Azure. Proven track record of building large-scale data ingestion and ETL pipelines in cloud environments, specifically Azure. Deep understanding of Azure Data Services (e.g., Azure Blob Storage, Azure Data Lake, Azure SQL Data Warehouse, Event Hubs, Functions etc.). Familiarity with data lakes , data warehouses , and modern data architectures. Experience with CI/CD pipelines , version control (Git), Jenkins and agile methodologies. Understanding of cloud infrastructure and architecture principles (especially within Azure ). Technical Skills: Expert-level proficiency in Spark, SPARK Streaming , including optimization, debugging, and troubleshooting Spark jobs. Solid knowledge of Azure Databricks for scalable, distributed data processing. Strong coding skills in Python and Scala for data processing. Experience working with SQL , especially for large datasets. Knowledge of data formats like Iceberg , Parquet , ORC , and Delta Lake . Leadership Skills: Proven ability to lead and mentor a team of data engineers, ensuring adherence to best practices. Excellent communication skills, capable of interacting with both technical and non-technical stakeholders. Strong problem-solving, analytical, and troubleshooting abilities.

Role & responsibilities Designed and implemented end-to-end data pipeline using DBT, Snowflake Created and structure DBT models like staging, transformation, marts, YAML configurations for models and tests, dbt seeds. Hands-on experience on DBT Jinja templating, macro development, dbt jobs and snapshot management for Slowly changing dimensions. Develop python script for data cleaning, transformation and automation of repetitive task. Experienced in loading structured and semi-structured data from AWS S3 to Snowflake by designing file formats, configuring storage integration, and automating data loads using Snow pipe. Designed scalable incremental models for handling large datasets, reducing resource usage Preferred candidate profile Candidate must have 5+ Yrs experience. Early joiner, who can join within a month

Bachelors degree in computer science, engineering, or a related field. Master’s degree preferred. Data: 5+ years of experience with data analytics and data warehousing. Sound knowledge of data warehousing concepts. SQL: 5+ years of hands-on experience on SQL and query optimization for data pipelines. ELT/ETL: 5+ years of experience in Informatica/ 3+ years of experience in IICS/IDMC Migration Experience: Experience Informatica on prem to IICS/IDMC migration Cloud: 5+ years’ experience working in AWS cloud environment Python: 5+ years of hands-on experience of development with Python Workflow: 4+ years of experience in orchestration and scheduling tools (e.g. Apache Airflow) Advanced Data Processing: Experience using data processing technologies such as Apache Spark or Kafka Troubleshooting: Experience with troubleshooting and root cause analysis to determine and remediate potential issues Communication: Excellent communication, problem-solving and organizational and analytical skills Able to work independently and to provide leadership to small teams of developers. Reporting: Experience with data reporting (e.g. MicroStrategy, Tableau, Looker) and data cataloging tools (e.g. Alation) Experience in Design and Implementation of ETL solutions with effective design and optimized performance, ETL Development with industry standard recommendations for jobs recovery, fail over, logging, alerting mechanisms. Role & responsibilities Preferred candidate profile

Job Description Data Engineer We are seeking a highly skilled Data Engineer with extensive experience in Snowflake, Data Build Tool (dbt), Snaplogic, SQL Server, PostgreSQL, Azure Data Factory, and other ETL tools. The ideal candidate will have a strong ability to optimize SQL queries and a good working knowledge of Python. A positive attitude and excellent teamwork skills are essential. Role & responsibilities Data Pipeline Development: Design, develop, and maintain scalable data pipelines using Snowflake, DBT, Snaplogic, and ETL tools. SQL Optimization: Write and optimize complex SQL queries to ensure high performance and efficiency. Data Integration: Integrate data from various sources, ensuring consistency, accuracy, and reliability. Database Management: Manage and maintain SQL Server and PostgreSQL databases. ETL Processes: Develop and manage ETL processes to support data warehousing and analytics. Collaboration: Work closely with data analysts, data scientists, and business stakeholders to understand data requirements and deliver solutions. Documentation: Maintain comprehensive documentation of data models, data flows, and ETL processes. Troubleshooting: Identify and resolve data-related issues and discrepancies. Python Scripting: Utilize Python for data manipulation, automation, and integration tasks. Preferred candidate profile Proficiency in Snowflake, DBT, Snaplogic, SQL Server, PostgreSQL, and Azure Data Factory. Strong SQL skills with the ability to write and optimize complex queries. Knowledge of Python for data manipulation and automation. Knowledge of data governance frameworks and best practices Soft Skills: Excellent problem-solving and analytical skills. Strong communication and collaboration skills. Positive attitude and ability to work well in a team environment. Certifications: Relevant certifications (e.g., Snowflake, Azure) are a plus. Please forward your updated profiles to the below mentioned Email Address: divyateja.s@prudentconsulting.com