584 Delta Lake Jobs - Page 9

As a Data Engineer-II at TMRW House of Brands in Bengaluru, Karnataka, your role involves creating, implementing, and operating robust and scalable data pipelines for business intelligence and machine learning. You will be responsible for developing and maintaining core data framework and key infrastructures. Additionally, you will create and support the ETL pipeline to ensure correct data flow from existing and new sources to the data warehouse. Your responsibilities also include designing the Data Warehouse and data modeling for efficient and cost-effective reporting. Collaboration with data analysts, data scientists, and other data consumers is essential to manage the data warehouse table...

Role Overview: As a System Architect with 12+ years of experience, you will be responsible for designing end-to-end architecture covering React-TypeScript front-ends, FastAPI micro-services, Python business logic, Azure data layers, and OpenAI API integration. You will define interface contracts, event flows, observability standards, and non-functional requirements such as quality, security, data residency, and performance. Your role will also involve establishing prompt/response validation, abuse mitigation, and PII safeguards for GPT-4o-class models. Additionally, you will benchmark chat completion and Responses API, embed models, and tune retrieval pipelines for factual accuracy. Your res...

At BairesDev, we've been leading the way in technology projects for over 15 years. We deliver cutting-edge solutions to giants like Google and the most innovative startups in Silicon Valley. Our diverse 4,000+ team, composed of the world's Top 1% of tech talent, works on roles that drive significant impact worldwide. When you apply for this position, you're taking the first step in a process that goes beyond the ordinary. We aim to align your passions and skills with our vacancies, setting you on a path to exceptional career development and success. Principal Data Lake Engineer at BairesDev We are seeking a Principal Data Lake Engineer with deep expertise in data lake architecture, schema-on...

We are looking for an experienced Data Engineer to drive the development of scalable, secure, and high-performance data solutions. This role requires deep technical expertise in Python, Apache Spark, Delta Lake, and orchestration tools like Databricks Workflows or Azure Data Factory. The ideal candidate will also bring a strong understanding of data governance, metadata management, and regulatory compliance in the insurance and financial services domains. Proficient in developing Python Applications, Spark-based workflows, leveraging Delta Lake, and orchestrating jobs using Databricks Workflows or Azure Data Factory. Good to understand retention metadata, business rules, and data governance ...

As a Senior Technical Architect with expertise in Databricks, Apache Spark, and modern data engineering architectures, you will play a crucial role in leading the architecture, design, and implementation of scalable data platforms and AI-powered applications for global clients. Your responsibilities will include: - Leading the architecture, design, and deployment of scalable data solutions using Databricks and the medallion architecture. - Guiding technical teams in building batch and streaming data pipelines using Spark, Delta Lake, and MLflow. - Collaborating with clients and internal stakeholders to understand business needs and translate them into robust data and AI architectures. - Desi...

As a Data Engineer, you will be responsible for developing, maintaining, and optimizing robust data ingestion and ETL/ELT pipelines using Python on the Databricks platform. You will work collaboratively with the architecture team to align implementation with long-term data strategies. Leveraging Delta Lake for data storage optimization and performance tuning will be a key part of your role. Additionally, you will utilize Prometheus for monitoring the health and performance of data pipelines and services. Ensuring quality, reliability, and scalability of the data platform across multiple domains and use cases is crucial. You will also drive automation and reusability in data processing pipeli...

As a Data Engineer at Omnissa, you will play a crucial role in delivering as an individual contributor. You should have a minimum of 2 to 8 years of experience in Data Engineering on AWS. Your responsibilities will include: - Strong Proficiency in SQL, Python, Apache Spark, AWS Data Stack (S3, EMR, Glue Catalog, Athena, Lambda, Airflow), Starburst / Trino. - Solid Understanding of Lakehouse Architecture - Apache Iceberg / Delta Lake. - Experience with CI/CD practices using GitHub Actions. - Ability to collaborate in cross-functional teams and communicate technical concepts clearly. - Mindset to design solutions for scale, proactively involve in additional areas and domains. - Experience with...

As an Azure Databricks Engineer at our fast-growing cloud data and analytics consultancy, you will play a crucial role in designing high-throughput lakehouse platforms, predictive analytics, and AI services on Microsoft Azure. Your responsibilities will include: - Designing, developing, and optimizing end-to-end data pipelines on Azure Databricks using PySpark/Scala and Delta Lake. - Building scalable ETL workflows to ingest structured and semi-structured data from Azure Data Lake, SQL, and API sources. - Implementing lakehouse architectures, partitioning, and performance tuning to ensure sub-second query response. - Collaborating with Data Scientists to prepare feature stores and accelerate...

Job Description Senior Manager, Lead Data Engineer The Opportunity Based in Hyderabad, join a global healthcare biopharma company and be part of a 130-year legacy of success backed by ethical integrity, forward momentum, and an inspiring mission to achieve new milestones in global healthcare. Be part of an organisation driven by digital technology and data-backed approaches that support a diversified portfolio of prescription medicines, vaccines, and animal health products. Drive innovation and execution excellence. Join a team that is passionate about using data, analytics, and insights to drive decision-making and create custom software, allowing us to tackle some of the world's greatest h...

Role Overview: As an offshore Techlead with Databricks engineer experience, your primary responsibility will be to lead the team from offshore. You will develop and maintain a metadata-driven generic ETL framework for automating ETL code and design, build, and optimize ETL/ELT pipelines using Databricks (PySpark/SQL) on AWS. In addition, you will be responsible for ingesting data from various structured and unstructured sources (APIs, RDBMS, flat files, streaming) and developing robust data pipelines for batch and streaming data using Delta Lake and Spark Structured Streaming. Your role will also involve implementing data quality checks, validations, and logging mechanisms, optimizing pipeli...

As a Data Engineer, you will be responsible for the following: - Designing, developing, and deploying data pipelines using Databricks, which includes data ingestion, transformation, and loading (ETL) processes. - Developing and maintaining high-quality, scalable, and maintainable Databricks notebooks using Python. - Working with Delta Lake and other advanced features to ensure efficient data processing. - Leveraging Unity Catalog for data governance, access control, and data discovery. - Developing and optimizing data pipelines for improved performance and cost-effectiveness. - Integrating with various data sources such as databases, cloud storage (Azure Blob Storage, ADLS, Synapse), and API...

As a Senior Data Engineer at Milestone Technologies, you will play a crucial role in leading data ingestion, transformation, and optimization initiatives using the modern Apache and Azure data stack. Your responsibilities will include: - Building and managing high-volume data pipelines using Spark/Databricks. - Implementing ELT frameworks using Azure Data Factory/Synapse Pipelines. - Optimizing large-scale datasets in Delta/Iceberg formats. - Implementing robust data quality, monitoring, and governance layers. - Collaborating with Data Scientists, Analysts, and Business stakeholders. Your technical stack will involve: - Big Data: Apache Spark, Kafka, Hive, Airflow, Hudi/Iceberg. - Cloud: Azu...

As a Data Engineer / Analyst at our company, your main responsibilities will include: - Designing, developing, and maintaining data solutions on Azure Databricks using Python, SQL, and PySpark. - Working extensively with Delta Lake and relational databases to build scalable data pipelines. - Performing analysis, design, implementation, and automated testing of data solutions. - Enhancing the consistency check engine for sub-ledger data and other tenants. - Optimizing Spark performance for very large datasets (10s of billions of rows, 200+ columns). - Identifying, troubleshooting, and resolving data issues proactively. - Collaborating with cross-functional teams to ensure data reliability, co...

As a Data Architect with over 12 years of experience, your role will involve leading the design and implementation of modern data platforms that support AI/ML, Analytics, NLP, and GenAI workloads. Your technical expertise, architectural vision, and leadership skills will be crucial for the success of the projects. **Key Responsibilities:** - Lead the design and implementation of scalable, secure, and high-performance data architectures for AI/ML, analytics, and real-time data processing. - Architect data platforms for ML experimentation, training, and deployment, covering classical ML, deep learning, NLP, and GenAI. - Develop data lakes, feature stores, and analytical data marts. - Define an...

As a Data Engineer, your role will involve collaborating with data scientists and business stakeholders to design, develop, and maintain efficient data pipelines feeding into the organization's data lake. It will be your responsibility to maintain the integrity and quality of the data lake, enabling accurate and actionable insights for data scientists and informed decision-making for business stakeholders. Leveraging your extensive knowledge of data engineering and cloud technologies, you will enhance the organization's data infrastructure to promote a culture of data-driven decision-making. Your expertise in data engineering will be crucial in defining and optimizing data pipelines using ad...

As an Enterprise Arch. Specialist Advisor at NTT DATA in Noida, Uttar Pradesh (IN-UP), India, you will have the following responsibilities: - Hands-on understanding of designing, developing, and maintaining scalable data pipelines. - Ensure data quality, integrity, and security across all pipelines. - Implement and manage data lakes to store structured and unstructured data. - Optimize data pipelines for performance and scalability. - Collaborate with data scientists, analysts, and other stakeholders to understand data requirements. - Develop and maintain ETL (Extract, Transform, Load) processes. - Utilize Databricks for data processing and analytics. - Monitor and troubleshoot data pipeline...

As a Senior Data Engineer, you will be responsible for designing and optimizing large-scale data pipelines using Databricks and the AWS ecosystem. Your expertise in Databricks and AWS with a minimum of 7 years of experience is mandatory for this role. Key Responsibilities: - Designing and optimizing large-scale data pipelines using Databricks and AWS technologies - Working with Databricks tools such as PySpark, Delta Lake, and MLflow - Utilizing AWS services including Glue, S3, Lambda, Redshift, and EMR - Developing and maintaining Data Lake, ETL pipelines, CI/CD processes - Proficiency in Python and SQL for data processing and analysis Qualifications Required: - Minimum 7 years of experienc...

At BairesDev, we've been leading the way in technology projects for over 15 years. We deliver cutting-edge solutions to giants like Google and the most innovative startups in Silicon Valley. Our diverse 4,000+ team, composed of the world's Top 1% of tech talent, works on roles that drive significant impact worldwide. When you apply for this position, you're taking the first step in a process that goes beyond the ordinary. We aim to align your passions and skills with our vacancies, setting you on a path to exceptional career development and success. Data Lake Engineer at BairesDev We are seeking a Data Lake Engineer with expertise in data lake architecture, schema-on-read, data governance, a...

Marsh McLennan is seeking candidates for the following position based in the Pune/Mumbai office. SeniorPrincipal Engineer/Manager Data Engineer What can you expect As a Data Engineer, you will be responsible for designing and implementing scalable data pipelines using Databricks and the Medallion Architecture. You will handle end-to-end ETL/ELT processes, manage large datasets, and work with tools like Python, PySpark, and AWS S3 to ensure data is transformed and optimized for analytical use. Additionally, knowledge on Informatica, Unix, and Oracle database is also a good to have. What is in it for you Holidays (As Per the location) Medical & Insurance benefits (As Per the location) Shared T...

As a Python Developer specializing in Machine Learning and Data Analytics, you will play a crucial role in designing, developing, and deploying Python applications that utilize data analytics, statistical modeling, and machine learning techniques. Your responsibilities will include: - Designing, developing, and deploying Python applications for data analytics and machine learning. - Independently working on machine learning model development, evaluation, and optimization. - Developing ETL pipelines and processing large-scale datasets for analysis. - Implementing scalable and efficient algorithms for predictive analytics and automation. - Optimizing code for performance, scalability, and main...

As an Azure DB Architect with 8+ years of experience, your role will involve: - Designing Databricks workspaces and Unity Catalog for multi-tenant environments - Architecting Lakehouse solutions with Delta Lake - Demonstrating expertise in Spark core for real-time and batch pipeline designing - Implementing data modelling, security & governance using RBAC and ABAC - Ensuring data lineage and metadata management - Orchestrating workflows using Databricks Workflows - Applying best practices for artifact organization and naming conventions in Databricks Qualifications required for this role include: - Minimum of 10 years of experience in the field - Deep understanding of AWS services such as Re...

Role Overview: As a Senior Azure Data Engineer, you will utilize Azure technologies to facilitate data transformation, analytics, and machine learning. Your primary responsibility will involve designing scalable Databricks data pipelines using PySpark to convert raw data into actionable insights. Additionally, you will be tasked with constructing, deploying, and managing machine learning models using MLlib or TensorFlow, all while optimizing cloud data integration from various Azure sources such as Blob Storage, Data Lake, and SQL/NoSQL databases. Your role will also encompass executing large-scale data processing using Spark Pools and fine-tuning configurations for efficiency. Key Responsib...

Responsibilities: Design, develop & optimize data pipelines using Azure Data Factory, Databricks & Synapse. Collaborate with cross-functional teams on project delivery within Azure DevOps. UK Shift

Ability to perform data transformations using Python, pandas, numpy, polars. Ability to query data from SQL and NoSQL databases. API Development- RESTful API, FastAPI Apache Open source components (e.g., Spark, Kafka),Delta Lake, Apache Iceberg,

A Candidate has Hands-on exp. working with tech. like Hadoop, Hive, Pig, Oozie, Map Reduce, Spark, Sqoop, Kafka, Flume. Also Know. of selected programming languages e.g. Python, Pyspark .In-depth know. of relational database e.g. PostgreSQL, MySQL . Required Candidate profile A Candidate has Must Hands-on exp. working with tech. like Hadoop, Hive, Pig, Oozie, Map Reduce, Spark, Sqoop, Kafka, Flume. Also Know. Python, Pyspark ,PostgreSQL, MySQL .