9 Databricks Autoloader Jobs

Azure Data Engineer Location: Bengaluru, Chennai, Delhi, Mumbai, Pune, Hyderabad Exp: 10 yrs Expertise: Good Knowledge of Data Brick lakehouse and Azure DataLake concept Knowledge of Data Bricks delta concept Delta live tables (DLT) Strong hands-on experience in ELT pipeline development using Azure Data factory and Databricks Autoloader, Notebook scripting and Azure Synapse Activity Copy, Data Flow Task Strong knowledge of metadata-driven data pipeline, metadata management, dynamic logic In-depth knowledge of data storage solutions, including Azure Data Lake Storage (ADLS), and Azure Serverless SQL Pool. Experience with data transformation using Spark, and SQL technologies. Solid understandi...

As a Senior/Lead Data Engineer at our company in Mumbai, you will be responsible for the following: - Senior Data Scientist with 6-8 years of IT Experience and atleast 5+ years as a Data Engineer - Kafka integration to Databricks - Structured Streaming concepts including Processing modes, output modes, and checkpointing - Medallion Architecture knowledge (Bronze, Silver, Gold layers) - Experience with Databricks Autoloader - Proficiency in working with large volumes of data - Implementation of performance optimization techniques such as partitioning, caching, and cluster tuning - Client facing experience - Excellent communication skills Qualifications required for this role: - 6-8 years of I...

As a Data Engineer, your role will involve designing and implementing streaming data pipelines that integrate Kafka with Databricks using Structured Streaming. You will be responsible for architecting and maintaining the Medallion Architecture with well-defined Bronze, Silver, and Gold layers. Your tasks will include implementing efficient ingestion using Databricks Autoloader for high-throughput data loads and working with large volumes of structured and unstructured data to ensure high availability and performance. Additionally, you will apply performance tuning techniques such as partitioning, caching, and cluster resource optimization. Collaboration with cross-functional teams including ...

JOB DESCRIPTION Data Engineer Designation Data Engineer Experience 5+ Years Location Mumbai (onsite) Job Summary: We are seeking a highly skilled Data Engineer with deep expertise in Apache Kafka integration with Databricks, structured streaming, and large-scale data pipeline design using the Medallion Architecture. The ideal candidate will demonstrate strong hands-on experience in building and optimizing real-time and batch pipelines, and will be expected to solve real coding problems during the interview. Job Description: • Design, develop, and maintain real-time and batch data pipelines in Databricks. • Integrate Apache Kafka with Databricks using Structured Streaming. • Implement robust ...

Role Summary & Role Description: Technical Manager with specific Oracle, PL/SQL and design, develop, and optimize data workflows on the Databricks platform. The ideal candidate will have deep expertise in Apache Spark, PySpark, Python, job orchestration, and CI/CD integration to support scalable data engineering and analytics solutions. Analyzes, designs, develops and maintains software applications to support business units. Expected to spend 80% of the time on hands-on development, design and architecture and remaining 20% on guiding the team on technology and removing other impediments Capital Markets Projects experience preferred Provides advanced technical expertise in analyzing, design...

Role Summary & Role Description: Technical Manager with specific Oracle, PL/SQL and design, develop, and optimize data workflows on the Databricks platform. The ideal candidate will have deep expertise in Apache Spark, PySpark, Python, job orchestration, and CI/CD integration to support scalable data engineering and analytics solutions. Analyzes, designs, develops and maintains software applications to support business units. Expected to spend 80% of the time on hands-on development, design and architecture and remaining 20% on guiding the team on technology and removing other impediments Capital Markets Projects experience preferred Provides advanced technical expertise in analyzing, design...

The ideal candidate will be responsible for designing and implementing streaming data pipelines that integrate Kafka with Databricks using Structured Streaming. You will also be tasked with architecting and maintaining the Medallion Architecture, which consists of well-defined Bronze, Silver, and Gold layers. Additionally, you will need to implement efficient data ingestion processes using Databricks Autoloader for high-throughput data loads. You will work with large volumes of structured and unstructured data to ensure high availability and performance, applying performance tuning techniques like partitioning, caching, and cluster resource optimization. Collaboration with cross-functional t...

As a Senior/Lead Data Engineer in our team based in Mumbai, IN, you will be responsible for leveraging your 6-8 years of IT experience, with at least 5+ years specifically in Data Engineering. Your expertise will be utilized in various areas including Kafka integration to Databricks, understanding of Structured Streaming concepts such as Processing modes, output modes, and checkpointing. Familiarity with Medallion Architecture (Bronze, Silver, Gold layers) and Databricks Autoloader will be key aspects of your role. Moreover, your experience in working with large volumes of data and implementing performance optimization techniques like partitioning, caching, and cluster tuning will be crucial...

Overview: As a Data Engineer, you will work with multiple teams to deliver solutions on the AWS Cloud using core cloud data engineering tools such as Databricks on AWS, AWS Glue, Amazon Redshift, Athena, and other Big Data-related technologies. This role focuses on building the next generation of application-level data platforms and improving recent implementations. Hands-on experience with Apache Spark (PySpark, SparkSQL), Delta Lake, Iceberg, and Databricks is essential Responsibilities: Define, design, develop, and test software components/applications using AWS-native data services: Databricks on AWS, AWS Glue, Amazon S3, Amazon Redshift, Athena, AWS Lambda, Secrets Manager Build and mai...