10148 Data Engineering Jobs - Page 27

Job Description: Python, SQL, ADF, Databricks ( PySpark, DLT, Unity Catalog, Performance Tuning, Cost Optimization ) along with leadership(self driven, responsible, accountable, Ownership and soft skill (communication, good mindset and behavioural attitude Databricks Associate or professional Certification.

Design, develop, and maintain scalable data pipelines using Python and modern data engineering tools. Understanding of banking data models, financial reporting, and regulatory compliance . Integrate data from legacy banking platforms into modern data lakes and warehouses. Collaborate with business analysts, architects, and operations teams to understand core banking workflows and data requirements. Design, build, and maintain scalable and reliable data pipelines using Azure Data Factory, Databricks, Synapse (Azure SQL DW), and Azure Data Lake. Perform data analysis and transformation using PySpark on Azure Databricks or Apache Spark. Design and implement data models optimized for storage and...

Design, develop, and maintain a robust, end-to-end data analytics pipeline for real-time data processing and visualization Take ownership of the entire project lifecycle, from architectural design and component integration to deployment, security, and presentation Work on data analytics platform (on GKE/GDC air gapped) to provide real-time hot spot prediction and ride recommendations for taxi drivers The application will use the NYC taxi dataset and showcase the capabilities of Kafka, Spark, HBase, Solr, Trino, Apache Airflow, Apache Superset and Apache Ranger Requirements Must Have: Big Data Ecosystem: Apache Kafka, Apache Spark, HBase, Apache Solr, and TrinoTools & Orchestration: Apache Ai...

Overview Machine Learning (Model Development, Deployment, MLOps), Python, Cloud Platforms (GCP/AWS) End-to-End ML Lifecycle Management, Python & Cloud Expertise, Data Engineering Scalable ML Solutions, Python, Cloud MLOps

Job Summary: We are looking for an experienced GCP Data Engineer to design, build, and optimize data pipelines and solutions on the Google Cloud Platform (GCP) . The ideal candidate will have deep expertise in BigQuery, Dataflow, Dataproc , and other GCP data services, along with strong programming and analytical skills. Key Responsibilities: Design and develop scalable data pipelines and ETL workflows using GCP services such as BigQuery, Dataflow, Dataproc, Composer, and Pub/Sub . Build and maintain data lakes, data warehouses, and analytics solutions on GCP. Collaborate with data scientists, analysts, and stakeholders to deliver clean, reliable, and structured datasets . Implement data ing...

Overview Brightspeed is building a transformative Generative AI platform to optimize circuit management across its telecom infrastructure. As Lead Engineer, youll drive Python-based development across data engineering, AI/ML, and system integration layers to enable intelligent circuit consolidation and forecasting. Key Responsibilities: Platform Development: Architect and build modular, cloud-native Python components for the optimization engine Data Engineering: Design ETL pipelines to ingest and normalize data from internal systems and vendor sources Knowledge Graphs: Implement and maintain the Canonical Circuit Intelligence Graph (CCIG) using Python and graph databases AI/ML Models: Develo...

Job Summary: We are looking for an experienced GCP Data Engineer to design, build, and optimize data pipelines and solutions on the Google Cloud Platform (GCP) . The ideal candidate will have deep expertise in BigQuery, Dataflow, Dataproc , and other GCP data services, along with strong programming and analytical skills. Key Responsibilities: Design and develop scalable data pipelines and ETL workflows using GCP services such as BigQuery, Dataflow, Dataproc, Composer, and Pub/Sub . Build and maintain data lakes, data warehouses, and analytics solutions on GCP. Collaborate with data scientists, analysts, and stakeholders to deliver clean, reliable, and structured datasets . Implement data ing...

Overview Prodapt is looking for senior data engineers having 3+ years of experience having hands on expertise on GCP, BQ, Data Pipeline creation & ETL. Responsibilities Understanding the requirements Performing in-depth analysis on data Setting up secure data pipelines from ingestion to delivery Individual contributor Requirements 3- 5 years of experience in Data Engineering with strong ability to work independently, communicate effectively, and deliver high-quality solutions. Should have strong SQL programming skills. Experience in cloud-based systems (GCP, BQ) is must. Performs data analysis required to troubleshoot data related issues and assist in the resolution of data issues. Expertise...

Overview Design and implement complex ETL/ELT pipelines using PySpark and Airflow for large-scale data processing on GCP.Lead data migration initiatives, including automating the movement of Teradata tables to BigQuery, ensuring data accuracy and consistency. Develop robust frameworks to streamline batch and streaming data ingestion workflows, leveraging Kafka, Dataflow, and NiFi. Collaborate with data scientists to build ML-ready data layers and support analytics solutions. Conduct proof of concepts (POCs) and document performance benchmarking for data throughput and velocity, ensuring optimized data workflows. Enhance CI/CD pipelines using Jenkins and GitLab for efficient deployment and mo...

Job Summary: We are looking for an experienced GCP Data Engineer to design, build, and optimize data pipelines and solutions on the Google Cloud Platform (GCP) . The ideal candidate will have deep expertise in BigQuery, Dataflow, Dataproc , and other GCP data services, along with strong programming and analytical skills. Key Responsibilities: Design and develop scalable data pipelines and ETL workflows using GCP services such as BigQuery, Dataflow, Dataproc, Composer, and Pub/Sub . Build and maintain data lakes, data warehouses, and analytics solutions on GCP. Collaborate with data scientists, analysts, and stakeholders to deliver clean, reliable, and structured datasets . Implement data ing...

We are seeking an experienced and motivated Senior Data Engineer to join our AI & Automation team. The ideal candidate will have 5-8 years of experience in data engineering, with a proven track record of designing and implementing scalable data solutions. A strong background in database technologies, data modeling, and data pipeline orchestration is essential. Additionally, hands-on experience with generative AI technologies and their applications in data workflows will set you apart. In this role, you will lead data engineering efforts to enhance automation, drive efficiency, and deliver data driven insights across the organization. Job Description: Design, build, and maintain scalable, hig...

Role Definition Leads, manages and coordinates all activities for a team developing a product, component or system. Responsibilities Provides technical support, and direction to teams and team leaders in BI Area that includes Tableau Power Bi Thoughtspot Acquires resources necessary to accomplish work on time Oversees performance of individual team members and team as a whole Identifies and addresses staff training and development needs in BI Area that includes Tableau Power Bi Thoughtspot Coordinates walkthroughs and reviews; ensures conformance to standards and adherence to design specifications Develops detailed work plans, schedules, project estimates, resource plans, and status reports ...

Job Purpose Applies existing data science techniques to new problems and datasets using specialised programming techniques. Selects from existing data sources and prepares data to be used by data science models. Evaluates the outcomes and performance of data science models. Identifies and implements opportunities to train and improve models and the data they use. Publishes and reports on model outputs to meet customer needs and conforming to agreed standards. Main Accountabilities Establishes strategic recommendations on data collections incorporating business requirements and knowledge of best practices Identifies and evaluates relevancy of data sources both internal and external, in collab...

Job Purpose Data Analyst plays a crucial lead role in managing and optimizing business intelligence solutions using Power BI. Interface with Business Owner, Project Manager, Planning Manager, Resource Managers etc. Execution of complex data analytics projects. Data Modeling and Integration: Develop complex data models, establish relationships, and ensure data integrity. Oversee data integration from various sources. Advanced Analytics: Perform advanced data analysis using DAX (Data Analysis Expressions) and other analytical tools to derive insights and support decision-making. Collaboration: Work closely with stakeholders to gather requirements, define data needs, and ensure the delivery of ...

We are seeking a Consulting Lead-Databricks with strong experience in Databricks, Python, and PySpark, along with hands-on knowledge of Unity Catalog (UC) and Lakehouse architecture. In this role, youll design and optimize large-scale data pipelines, implement robust governance and security through UC, and build scalable solutions in a modern cloud data environment. The ideal candidate must have 5+ years in data engineering, expertise with any one of the cloud platforms (Azure/AWS/GCP), and proven ability to deliver high-performing ETL/ELT pipelines that support business-critical analytics. Qualification Hands-on expertise in Databricks, Python, PySpark, and Delta Lake/Lakehouse architecture...

Role: Azure Data Engineer Experience: 9-10 Years Relevant Experience in Data Engineer: 6+ Years Notice Period: Immediate Joiners Only Job Location: Pune and Bangalore Technical and professional skills: Must have skills: Data Engineer, pyspark,Azure Data Factory and SQL. We are looking for a flexible, fast learning, technically strong Data Engineer. Expertise is required in the following fields: Bachelors Degree in Computer Science or related field 9-11 years of experience in Cloud Data Engineering. Proficient in Cloud Services Azure Build ETL and data movement solutions. Migrate data from traditional database systems to Cloud environment Strong hands-on experience in handling streaming data ...

Role & responsibilities Job title: - Data Engineer Location: Pune (Relocation is considered) Shift Timings: 4:00 PM to 1:00 AM IST Notice period -Immediate to 30 days . Experience Level: 6.5+ Years relevant experience is must. Qualifications: - Bachelor's/masters degree in computer science, Engineering, or a related field Must have skills: - Minimum of 6 years of Data Engineering Experience - - Minimum of 6 years of Data Engineering Experience - Build and manage scalable data lake solutions using Azure Data Lake Storage Gen2, integrating with services like Data Factory, Databricks, and Synapse Analytics. - Design and optimize data pipelines for efficient ingestion, transformation, and storag...

Salary: 15 to 25 LPA Exp: 4 to 7 years Location: Gurgaon/Pune/Bengalore Notice: Immediate to 30 days..!! Job Profile: Experienced Data Engineer with a strong foundation in designing, building, and maintaining scalable data pipelines and architectures. Skilled in transforming raw data into clean, structured formats for analytics and business intelligence. Proficient in modern data tools and technologies such as SQL, T-SQL, Python, Databricks, and cloud platforms (Azure). Adept at data wrangling, modeling, ETL/ELT development, and ensuring data quality, integrity, and security. Collaborative team player with a track record of enabling data-driven decision-making across business units. As a Dat...

Overview Job Summary: We are looking for a skilled GCP Data Engineer with 3 to 6 years of experience to join our dynamic team. The ideal candidate will have a strong background in data engineering, cloud technologies, and experience with Google Cloud Platform (GCP) services. You will be responsible for designing, building, and maintaining scalable data pipelines and frameworks that support our data analytics and machine learning initiatives. Key Responsibilities: Design and implement data pipelines using GCP services such as BigQuery, Dataflow, and Pub/Sub. Collaborate with data scientists and analysts to understand data needs and deliver appropriate solutions. Optimize data storage and proc...

Overview Senior PM Responsibilities Project Planning and Strategy Define project scope, objectives, and deliverables in collaboration with stakeholders. Develop comprehensive project plans including timelines, milestones, resources, and budgets. Identify project risks and create mitigation strategies. Team Leadership and Management Lead, mentor, and manage project teams, including junior project managers and other team members. Allocate resources effectively and ensure team members understand their roles and responsibilities. Foster a collaborative and productive team environment. Stakeholder Communication and Management Act as the primary point of contact between the project team and stakeh...

We are seeking a skilled Data Engineer with strong experience in SQL, PySpark, and cloud technologies to join our dynamic team. The ideal candidate will have a solid background in designing and implementing data engineering pipelines, with a focus on performance, scalability, and reliability. You will work closely with other data engineers, data scientists, and stakeholders to develop, maintain, and optimize data infrastructure. Key Responsibilities : SQL Development : - Write medium-complexity SQL queries to extract, transform, and load data efficiently. - Optimize SQL queries for performance and scalability. - Collaborate with team members to understand data requirements and translate them...

Role Description : As a Data Engineer, you will play a crucial role in designing and implementing data pipelines to support business intelligence and analytics initiatives. You will work closely with data analysts, data scientists, and business stakeholders to understand data requirements and create efficient data models. Key Responsibilities : - Design and implement data pipelines to extract, transform, and load data from various sources into a centralized data warehouse or data lake - Collaborate with data analysts and business stakeholders to understand data requirements and create efficient data models - Optimize data pipelines for performance, scalability, and reliability using tools li...

We are seeking a highly skilled and experienced SAS Demand Engineer to join our team. The ideal candidate will have extensive hands-on experience in data engineering, particularly with Base SAS and Advanced SAS. This role requires a professional with excellent problem-solving skills, deep technical knowledge, and the ability to work collaboratively in a fast-paced environment. Primary Responsibilities : - End-to-end ownership, from implementation to production deployment. - Develop, monitor, and schedule SAS jobs, resolving any issues that arise. - Contribute to the design of ETL workflows, including integration, transformation, and loading. Must Have Skills : - Overall, 5-7 years of experie...

Spark, Hive, Scala, SQL ,Python scripting ,data visualization Key Responsibilities: Leadership and Team Management: Lead and mentor a team of data engineers, fostering a culture of collaboration and continuous improvement. Oversee the planning and execution of data projects, ensuring alignment with business objectives and timelines. Provide technical guidance and expertise to the team, promoting best practices in data engineering. Data Architecture and Development: Design, develop, and maintain scalable data pipelines and ETL processes to ingest, process, and store large volumes of data. Architect and implement robust data models and schemas to support analytics and business intelligence nee...

JD: Azure Data Engineer with extensive experience on solution designing - Domain Data sets, data products & hands on experience in Azure Databricks, Python, Pyspark & ETL Pipelines, Should know & have working experience with Atlas framework. Should know and worked in medallion architecture. 1. Design, build, and maintain scalable and efficient data pipelines using Databricks. Integrate data from various sources (for eg. Azure storage account, SFTP, Fabric) using Apache Spark within Databricks. 2. Implement processes and checks to ensure the accuracy and completeness of data. 3. Work closely with data owners, analysts, and other stakeholders to understand data requirements and deliver solutio...