784 Etl Pipelines Jobs - Page 17

Job Responsibilities Design, build & maintain scalable data pipelines for ingestion, processing & storage. Collaborate with data scientists, analysts, and product teams to deliver high-quality data solutions. Optimize data systems for performance, reliability, scalability, and cost-efficiency . Implement data quality checks ensuring accuracy, completeness, and consistency. Work with structured & unstructured data from diverse sources. Develop & maintain data models, metadata, and documentation . Automate & monitor workflows using tools like Apache Airflow (or similar). Ensure data governance & security best practices are followed. Required Skills & Qualifications Bachelors/Master’s degree in...

Technical / Behavioral - You must have proven track record of working in collaborative teams to deliver high quality data solutions in a multi-developer agile environment following design & coding best practices. You must be an expert in using SQL queries with UNIX shell /Python scripting skills. You should have experience of using AWS services like EC2, S3, EMR to move data onto cloud platform and experience with Enterprise data lake strategy and use of Snowflake (AWS) is a big plus. You must have knowledge to develop ELT/ETL pipelines to move data to and from Snowflake data store using combination of Python and Snowflake SnowSQL is a big plus. You should have experience in SQL performance ...

Job Title : Senior Data Engineer Work location Pune Let me tell you about the role Data Engineer designs and builds scalable data management systems that support application simplification efforts. They develop and maintain databases and large-scale processing systems to enable efficient data collection, analysis, and integration. Key responsibilities include ensuring data accuracy for modeling and analytics, optimizing data pipelines for scalability, and collaborating with data scientists to drive data-driven decision-making. They play a crucial role in supporting the teams goals by enabling seamless access to reliable data for simplification initiatives. What you will deliver As part of a ...

As a skilled professional in the field of cloud computing and data engineering, you will be responsible for a variety of tasks related to managing data assets, developing ETL pipelines, and exploring machine learning capabilities. Your expertise in technologies such as Git, Kubernetes, Docker, and Helm will be essential in ensuring the smooth functioning of cloud infrastructure on platforms like AWS and Azure. Your role will involve the evaluation and analysis of data using tools like AWS QuickSight, as well as the deployment of machine learning models through AWS MLOps. Proficiency in AWS services such as Glue, QuickSight, SageMaker, Lambda, and Step Functions will be crucial for success in...

As a Lead Data Engineer with over 7 years of experience, you will be responsible for designing, developing, and maintaining data pipelines, ETL processes, and data warehouses. You will be based in Hyderabad and should be able to join immediately. Your primary skills should include proficiency in SQL, Python, PySpark, AWS, Airflow, Snowflake, and DBT. Additionally, you should have a Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or a related field. In this role, you will need a minimum of 4+ years of hands-on experience in data engineering, ETL, and data warehouse development. You should have expertise in ETL tools such as Informatica Power Center or IDMC...

Role & responsibilities We are seeking a skilled LEAD Data Engineer to join our dynamic team. The Data Engineer will be responsible for designing, developing, and maintaining our data pipelines, integrations, and data warehouse infrastructure. The successful candidate will work closely with data scientists, analysts, and business stakeholders to ensure that our data is accurate, secure, and accessible for all users. Responsibilities Design and build scalable data pipeline architecture that can handle large volumes of data Develop ELT/ETL pipelines to extract, load and transform data from various sources into our data warehouse Optimize and maintain the data infrastructure to ensure high avai...

Were seeking a Senior Data Engineer to join our client, a world-class education consulting firm, in building robust data solutions that drive innovation and scalability. In this role, youll design and optimize high-performance data pipelines, ensuring seamless data flow to support advanced analytics and business intelligence. Collaborating with cross-functional teams, youll enhance data models, improve accessibility, and empower data-driven decision-making across a global organization. Location: Bengaluru (Remote Work Available for the right candidate) Company: A Globally Renowned Education Consulting Leader Work Flexibility: While based in Bengaluru, this role offers the flexibility to work...

As a Data Engineer with 4 to 7 years of experience and a budget range of 25 to 30 LPA, you will be responsible for handling the flow and storage of data efficiently to ensure that the model has access to the right datasets. Your role will involve tasks such as Database Design, where you will create efficient database systems using SQL, NoSQL, Hadoop, and Spark to store large volumes of transactional or behavioral data. You will also be involved in ETL Pipelines, building pipelines to collect, transform, and load data from various sources into a format suitable for analysis. Data Integration will be another key aspect of your responsibilities, where you will integrate different data sources, ...

As a Senior Azure Data Engineer based in Chennai, India, you will play a crucial role in designing and implementing data pipelines using Azure Synapse to integrate data from various sources and file formats into SQL Server databases. Your responsibilities will include developing batch and real-time data pipelines to facilitate the transfer of data to data warehouses and data lakes. Collaborating with the Data Architect, you will work on new Data projects by constructing Data pipelines and managing Master Data. Your expertise in data analysis, extraction, cleansing, column-mapping, data transformations, and data modeling will be essential in meeting business requirements. You will be tasked w...

DESCRIPTION Are you excited about the digital media revolution and passionate about designing and delivering advanced analytics that directly influence the product decisions of Amazon's digital businesses. Do you see yourself as a champion of innovating on behalf of the customer by turning data insights into action The Amazon Digital Acceleration (DA) org is looking for an analytical and technically skilled data engineer to join our team. In this role, you will play a critical part in developing foundational analytical datasets spanning orders, subscriptions, discovery, promotions, pricing and royalties. Our mission is to enable digital clients to easily innovate with data on behalf of custo...

As a Microsoft Purview Data Governance Consultant / Sr. Consultant with 5-8 years of experience, you will be responsible for developing and deploying solutions with Microsoft Purview or similar data governance platforms. You must have a deep understanding of data governance principles, including metadata management, data cataloging, lineage tracking, and compliance frameworks. Your proficiency in Microsoft Azure, especially with services like Azure Data Factory, Azure Synapse Analytics, Azure SQL Database, and Azure Blob Storage, will be crucial for this role. It would be nice to have experience in data integration, ETL pipelines, and data modeling, as well as knowledge of security and compl...

You should have a minimum of 5 years of experience in SQL development with a strong skill in query optimization. Your responsibilities will include designing, developing, and maintaining SQL queries, stored procedures, and database structures. You must demonstrate expertise in building and managing ETL pipelines using Azure Data Factory (ADF) and integrating data from on-premise and cloud sources using AWS services. A good understanding of data warehousing concepts, data modeling, and transformation logic is required. You should be able to ensure data accuracy, consistency, and performance across multiple systems and possess strong troubleshooting skills to resolve data and performance-relat...

As a Data Product Manager at our company based in Mumbai, you will play a crucial role in driving the development of data-driven products that align with our business objectives. Your responsibilities will involve collaborating with data engineers, analysts, and stakeholders to build and scale data-powered solutions. By defining and executing the data product roadmap, you will ensure that our data strategy meets both company goals and customer needs. Additionally, you will work on developing product features that enhance data accessibility, analytics, and AI/ML adoption. It will be your responsibility to integrate data-driven insights into products and decision-making processes while ensurin...

As a candidate for the position, your main responsibility will involve designing and implementing intelligent Agentic AI systems using LLMs, vector databases, and orchestration frameworks. You will also be tasked with building, deploying, and maintaining ML workflows utilizing AWS SageMaker, Lambda, and EC2 with Docker. Another key aspect of the role will be developing and managing ETL pipelines through AWS Glue and integrating them with structured/unstructured data sources. Additionally, you will be expected to implement APIs and full-stack components to support ML agents, including visualization tools using Streamlit, as well as reverse-engineer existing codebases and APIs to integrate AI ...

Summary We are seeking an experienced Data Engineer (5years of relevant experience) with a strong technical background to design, develop, and support cutting-edge data-driven solutions. The ideal candidate will bring expertise in data engineering, cloud technologies, and enterprise-scale solution development, along with exceptional problem-solving, communication, and leadership skills. Key Responsibilities: Design and implement data engineering pipelines using PySpark, Palantir Foundry and Python. Leverage cloud technologies (AWS) to build scalable and secure solutions. Ensure compliance with architecture best practices and develop enterprise-scale solutions. Develop data-driven application...

Role & responsibilities We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. The ideal candidate will have a passion for building and optimizing robust data pipelines, managing real-time data flows, and developing solutions that drive data-driven decision-making. If you thrive in a fast-paced environment and are eager to tackle complex data challenges, we want to hear from you! Role and Responsibilities: Design, develop, and maintain scalable and high-performance ETL/ELT pipelines for batch and real-time data processing from diverse sources. Develop and implement APIs and webhooks for seamless and secure data ingestion and consumption by various internal and ...

Senior Azure Databricks Developer Weekend Virtual Drive on 30 Aug - Between 11am to 3pm Salary up to 15 LPA Exp: 7+ Years POSITION GENERAL DUTIES AND TASKS : Advanced experience with Databricks, including Databricks SQL, DataFrames, and Spark (PySpark or Scala). Deep understanding of Spark architecture and optimization strategies (e.g., tuning Spark SQL configurations, managing data partitions, handling data skew, and leveraging broadcast joins). Proficiency in building and optimizing large-scale data pipelines for ETL/ELT processes using Databricks. Familiarity with Delta Lake and data lake architectures. Strong programming skills in Python, SQL, or Scala. Experience with version control (e...

Role & responsibilities Design, develop, and maintain ETL pipelines to extract, transform, and load data from various sources into Oracle, Snowflake, or cloud data warehouses. Write and optimize PL/SQL scripts, stored procedures, and functions for data processing and performance tuning. Build and configure Talend jobs (or other ETL tools) for batch and real-time data integration. Develop and support data models, schemas, and warehouse structures aligned with business reporting needs. Collaborate with BI teams to enable dashboards and reports in Cognos and Power BI. Implement data quality checks, validations, and reconciliation mechanisms. Work with cloud data platforms (AWS, Azure, or GCP) f...

You will be joining Tietoevry Create as a Snowflake Developer in Bengaluru, India. Your primary responsibility will be to design, implement, and maintain data solutions using Snowflake's cloud data platform. Working closely with cross-functional teams, you will deliver high-quality, scalable data solutions that drive business value. With over 7 years of experience, you should excel in designing and developing Datawarehouse & Data integration projects at the SSE / TL level. It is essential to have experience working in an Azure environment and be proficient in developing ETL pipelines using Python and Snowflake SnowSQL. Your expertise in writing SQL queries against Snowflake and understanding...

You will be responsible for designing, implementing, and continuously expanding data pipelines through extraction, transformation, and loading activities. Your role will involve gathering requirements and business process knowledge to ensure data transformation aligns with the needs of end users. Additionally, you will maintain and enhance existing processes, focusing on scalability and maintainability of the data architecture. Collaborating with the business, you will design and deliver high-quality data solutions. As part of our commitment to diversity, equity, and inclusion (DEI), you will be joining a workplace that values these principles. At Indium, we prioritize DEI through dedicated ...

We are looking for a highly skilled AI Technical Delivery Lead (Associate Director) with expertise in machine learning, generative AI, and hands-on architecture. Your role will involve leading our AI initiatives, providing technical guidance, fostering innovation, and ensuring successful implementation of advanced AI solutions. You will be responsible for designing, developing, and deploying cutting-edge machine learning and generative AI solutions to address business challenges. As the AI Technical Delivery Lead, you will be required to architect robust and scalable AI systems, ensuring adherence to best practices in design, coding, testing, and maintenance. Collaborating with cross-functio...

You are an experienced and technically proficient SQL Server Warehouse Admin / Tech Lead responsible for managing, optimizing, and troubleshooting large-scale SQL Server-based data warehouse environments. Your role involves building and maintaining SQL Server high availability (HA) clusters, implementing disaster recovery (DR) solutions, leading development teams, and ensuring performance, stability, and scalability across systems. As a Technical Leader & Solution Delivery expert, you will develop, implement, and promote robust technical solutions using SQL Server technologies. You will design and maintain SQL Server HA clusters, lead and mentor team members, and ensure adherence to coding s...

Critical Skills To Possess Advanced working knowledge and experience with relational and non-relational databases. Advanced working knowledge and experience with API data providers Experience building and optimizing Big Data pipelines, architectures, and datasets. Strong analytic skills related to working with structured and unstructured datasets. Hands-on experience in Azure Databricks utilizing Spark to develop ETL pipelines. Strong proficiency in data analysis, manipulation, and statistical modeling using tools like Spark, Python, Scala, SQL, or similar languages. Strong experience in Azure Data Lake Storage Gen2, Azure Data Factory, Databricks, Event Hub, Azure Synapse. Familiarity with ...

DESCRIPTION We are seeking a highly skilled Data Engineer to join our FinTech ADA team, responsible for building and optimizing scalable data pipelines and platforms that power analytics, automation, and decision-making across Finance and Accounting domains. The ideal candidate will have strong expertise in AWS cloud technologies including Redshift, S3, AWS Glue, EMR, Kinesis, Firehose, Lambda, and IAM, along with hands-on experience designing secure, efficient, and resilient data architectures. You will work with large-scale structured and unstructured datasets, leveraging both relational and non-relational data stores (object storage, key-value/document databases, graph, and column-family ...

DESCRIPTION Are you interested in diving deep into the data and engineering the metrics model that drives continuous improvement for Amazon's eCommerce systems Do you have solid problem-solving abilities, metrics-driven decision making and want to solve problems with solutions that will meet the growing worldwide need for Amazon's products Then eCommerce Services is the team for you. We are part of building the world's best, most reliable, and most feature-rich eCommerce platform that provides customers the best experience possible and are looking for a top-notch Senior Business Intelligence Engineer. In eCommerce Services (eCS), we build systems that span the full range of eCommerce functio...