318 Delta Lake Jobs - Page 5

As a Data Engineer, you will be responsible for designing and building efficient data pipelines using Azure Databricks (PySpark). Your role will involve implementing business logic for data transformation and enrichment at scale and managing and optimizing Delta Lake storage solutions. You will also be involved in developing REST APIs using FastAPI to expose processed data and deploying APIs on Azure Functions for scalable and serverless data access. Additionally, you will develop and manage Airflow DAGs to orchestrate ETL processes, ingest and process data from various internal and external sources on a scheduled basis. Database management will be a key part of your responsibilities, includ...

As a Lead Data Engineer at Rearc, you will play a crucial role in establishing and maintaining technical excellence within our data engineering team. Your extensive experience in data architecture, ETL processes, and data modeling will be key in optimizing data workflows for efficiency, scalability, and reliability. Collaborating closely with cross-functional teams, you will design and implement robust data solutions that align with business objectives and adhere to best practices in data management. Building strong partnerships with technical teams and stakeholders is essential as you drive data-driven initiatives and ensure their successful implementation. With over 10 years of experience ...

You will need to have at least 7-9 years of experience to apply for the role of Microsoft Fabric Architect. In this role, you will be responsible for working hands-on with Microsoft Fabric, which includes Lakehouse, Synapse, Pipelines, and Dataflows. You should be proficient in DAX, Power Query (M), and SQL, and have a strong understanding of Delta Lake, Parquet, and OneLake storage. Knowledge of Spark Notebooks, KQL, and Data Engineering within Fabric will be an added advantage. Experience with source control integration (Git) and CI/CD pipelines is essential for this role. Familiarity with Azure Data Factory, Azure Synapse Analytics, or Databricks will be considered a plus. You should poss...

As an Integration Developer at our organization, you will be an integral part of the data engineering and integration team. Your primary responsibility will be to design, develop, and maintain ETL/ELT pipelines using tools such as SSIS and Azure Data Factory (ADF). You will play a crucial role in building reliable data integration pipelines that support various business functions including business intelligence, analytics, and operational workloads. Your key responsibilities will include implementing robust data workflows to ingest, transform, and store structured and unstructured data in Azure Data Lakehouse, integrating cloud and on-premise systems for high performance and data quality, an...

Project Role : Cloud Migration Engineer Project Role Description : Provides assessment of existing solutions and infrastructure to migrate to the cloud. Plan, deliver, and implement application and data migration with scalable, high-performance solutions using private and public cloud technologies driving next-generation business outcomes. Must have skills : Microsoft Azure DevOps Good to have skills : NA Minimum 5 Year(s) Of Experience Is Required Educational Qualification : 15 years full time education We are seeking a hands-on senior professional who can lead a team of data engineers to lay the groundwork for making data accessible to AI engineers for research and/or application developme...

You are seeking a skilled Azure Databricks Architect with 12 to 17 years of experience in Python, SQL. As an Azure Databricks Architect at our company, you will be responsible for data architecture, data engineering, and analytics. You should have at least 5 years of hands-on experience with Azure Databricks, Apache Spark, and Delta Lake. Your proficiency in Azure Data Lake, Azure Synapse, Azure Data Factory, and Azure SQL is essential. Expertise in Python, Scala, and SQL for data processing, along with a deep understanding of data modeling, ETL/ELT processes, and distributed computing, are key requirements. Experience with CI/CD pipelines and DevOps practices in data engineering is also exp...

At EY, youll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we're counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. We are looking for a BI Senior Consultant to join the EY GDS Team. As part of our EY GDS TechOps team, you will be responsible for providing expert-level business intelligence support with a strong focus on Power BI and Databricks. You will work across various regions for our global clients, helping to design, develop, and maintain insightful ...

We are looking for an experienced Data Engineer with strong expertise in Databricks and Azure Data Factory (ADF) to design, build, and manage scalable data pipelines and integration solutions. The ideal candidate will have a solid background in big data technologies, cloud platforms, and data processing frameworks to support enterprise-level data transformation and analytics initiatives. Key Responsibilities: Design, develop, and maintain robust data pipelines using Azure Data Factory and Databricks . Build and optimize data flows and transformations for structured and unstructured data. Develop scalable ETL/ELT processes to extract data from various sources including SQL, APIs, and flat fil...

Roles & Responsibilities: Architect and maintain robust, scalable data pipelines using Databricks, Spark, and Delta Lake for both batch and real-time data processing. Lead technology evaluation and adoption initiatives to improve productivity, scalability, and data delivery. Optimize data processing performance through Spark tuning, job scheduling, and efficient resource utilization. Develop innovative solutions to enhance data ingestion, transformation, lineage tracking, and observability. Build metadata-driven frameworks to promote pipeline consistency and reuse. Promote a culture of engineering excellence, continuous improvement, and experimentation. Collaborate with architecture, platfor...

Key Responsibilities: Ontology Development: Design and implement ontologies based on BFO and CCO principles, ensuring alignment with business needs and industry standards. Collaborate with domain experts to capture and formalize domain knowledge into ontological structures. Develop and maintain comprehensive ontologies modeling business entities, relationships, and processes. Data Modeling: Design and implement semantic and syntactic data models adhering to ontological principles. Create scalable, flexible, and adaptable data models to meet evolving business requirements. Integrate data models with existing data infrastructure and applications. Knowledge Graph Implementation: Design and buil...

The Databricks Data Engineering Lead role requires a highly skilled individual who will architect and lead the implementation of scalable, high-performance data pipelines and platforms using the Databricks Lakehouse ecosystem. As a Data Engineering Lead, you will be responsible for managing a team of data engineers, establishing best practices, and collaborating with cross-functional stakeholders to unlock advanced analytics, AI/ML, and real-time decision-making capabilities. Your key responsibilities will include leading the design and development of modern data pipelines, data lakes, and lakehouse architectures using Databricks and Apache Spark. You will manage and mentor a team of data en...

NTT DATA is looking for a Databricks Engineer to join their team in Bangalore, Karnataka, India. As a Databricks Engineer, you will be responsible for various tasks related to data extraction, ETL pipeline modernization, job design, development, automation, metadata management, documentation, testing, collaboration, performance tuning, security, governance, and compliance. Your primary job duties will include extracting and analyzing data from SQL Server and Teradata sources, translating legacy SQL/DataStage transformations into Databricks-native code, building and orchestrating jobs within Databricks using tools like Databricks Workflows, Delta Lake, and Auto Loader, generating and maintain...

You are an experienced Senior Data Engineer with a strong background in Databricks, Apache Airflow, SQL, and Python. Your main responsibility will involve designing and optimizing data pipelines, ETL workflows, and cloud-based data solutions. Your expertise in big data processing, cloud-based analytics, and automation will be crucial in developing efficient and scalable data engineering solutions. You will be responsible for tasks such as designing, developing, and optimizing data pipelines using Databricks and Apache Airflow. Implementing PySpark-based transformations and processing in Databricks to handle large-scale data, and maintaining SQL-based data pipelines with a focus on performanc...

We have a great opportunity for the role of Senior Data Engineer - Azure, Databricks. Relevant Exp: 5+ Yrs Job Description:- Mandatory Skills -Platform Engineer, Databricks, Azure Cloud, Data Engineer, Pyspark, SQL, Databricks. o Hands-on in data structures, distributed systems, Spark, SQL, PySpark, NoSQL Databases. o Strong software development skills in at least one of: Python, Pyspark or Scala. o Develop and maintain scalable & modular data pipelines using databricks & Apache Spark. o Experience building and supporting large-scale systems in a production environment. o Experience working on either of the cloud storages - AWS S3, Azure data lake, GCS buckets. o Exposure to the ELT tools of...

The job is based in Hyderabad/Bangalore/Chennai and at Gramener, you will find a welcoming work environment with diverse colleagues, a clear career path, and ample opportunities for growth and innovation. The company aims to develop a range of easily configurable data applications focused on storytelling for both public and private use. As part of your role, you will be involved in various impactful customer technical data platform projects, taking the lead on strategic initiatives that cover the design, development, and deployment of cutting-edge solutions. Collaboration with platform engineering teams will be key in implementing Data Brick services effectively within the company's infrastr...

About GyanSys Inc. GyanSys Inc. is a leading global systems integrator headquartered in the U.S., specializing in SAP, Salesforce, Microsoft, and ServiceNow solutions. With over 2,000 consultants worldwide and 300+ enterprise clients, GyanSys is at the forefront of delivering digital transformation and business process automation across industries. Our India delivery centers in Bengaluru, Hyderabad, and Pune play a critical role in driving global implementations for Fortune 500 companies. At GyanSys, youll find a collaborative culture, cutting-edge projects, continuous learning opportunities , and the ability to make a real impact . Job Title: Databricks Consultant Experience: 3-5 Years Loca...

We are looking for a seasoned Senior Data Architect with extensive knowledge in Databricks and Microsoft Fabric to join our team. In this role, you will be responsible for leading the design and implementation of scalable data solutions for BFSI and HLS clients. As a Senior Data Architect specializing in Databricks and Microsoft Fabric, you will play a crucial role in architecting and implementing secure, high-performance data solutions on the Databricks and Azure Fabric platforms. Your responsibilities will include leading discovery workshops, designing end-to-end data pipelines, optimizing workloads for performance and cost efficiency, and ensuring compliance with data governance, security...

You will be responsible for designing, developing, and maintaining scalable data pipelines using Azure Databricks. Your role will involve building and optimizing ETL/ELT processes for structured and unstructured data, collaborating with data scientists, analysts, and business stakeholders, integrating Databricks with Azure Data Lake, Synapse, Data Factory, and Blob Storage, developing real-time data streaming pipelines, and managing data models/data warehouses. Additionally, you will optimize performance, manage resources, ensure cost efficiency, implement best practices for data governance, security, and quality, troubleshoot and improve existing data workflows, contribute to architecture a...

Responsible for developing, optimize, and maintaining business intelligence and data warehouse systems, ensuring secure, efficient data storage and retrieval, enabling self-service data exploration, and supporting stakeholders with insightful reporting and analysis. Grade - T5 Please note that the Job will close at 12am on Posting Close date, so please submit your application prior to the Close Date Accountabilities What your main responsibilities are: Data Pipeline - Develop and maintain scalable data pipelines and builds out new API integrations to support continuing increases in data volume and complexity Data Integration - Connect offline and online data to continuously improve overall u...

At EY, you'll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture, and technology to become the best version of you. And we're counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. Our Technology team builds innovative digital solutions rapidly and at scale to deliver the next generation of Financial and Non-Financial services across the globe. The Position is a senior technical, hands-on delivery role, requiring knowledge of data engineering, cloud infrastructure, platform engineering, platform operations, and product...

Req ID: 335675 NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Databricks Engineers to join our team in Bangalore, Karn?taka (IN-KA), India (IN). "Job Duties: 1. Data Extraction & Source Analysis Extract and analyze data from SQL Server and Teradata sources. Review and interpret existing ETL logic from DataStage, BTEQ scripts, and custom SQL. Identify filtering conditions, incremental load logic, and audit/control columns. Work with SMEs to understand data models, including workflow, reference, and source tables. ETL...

Req ID: 335672 NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Databricks Engineers to join our team in Bangalore, Karn?taka (IN-KA), India (IN). "Job Duties: 1. Data Extraction & Source Analysis Extract and analyze data from SQL Server and Teradata sources. Review and interpret existing ETL logic from DataStage, BTEQ scripts, and custom SQL. Identify filtering conditions, incremental load logic, and audit/control columns. Work with SMEs to understand data models, including workflow, reference, and source tables. ETL...

As a dedicated professional at Leighton Asia, you will be responsible for spearheading the development of end-to-end Azure integration solutions in alignment with CIMIC's OneIT Target Reference Architecture. Your key responsibilities will include ensuring compliance with CIMIC's Security Standards, constructing data and integration solutions that adhere to CIMIC's Target Reference Architecture, as well as hands-on development and unit testing of the solutions. You will play a crucial role in following CIMIC's path to production environments, providing technical support during test phases, and conducting handover sessions to transition solutions to BAU Support. Additionally, you will be invol...

As a Data Modeler, your primary responsibility will be to design and develop conceptual, logical, and physical data models supporting enterprise data initiatives. You will work with modern storage formats like Parquet and ORC, and build and optimize data models within Databricks Unity Catalog. Collaborating with data engineers, architects, analysts, and stakeholders, you will ensure alignment with ingestion pipelines and business goals. Translating business and reporting requirements into robust data architecture, you will follow best practices in data warehousing and Lakehouse design. Your role will involve maintaining metadata artifacts, enforcing data governance, quality, and security pro...

As a Big Data Architect specializing in Databricks at Codvo, a global empathy-led technology services company, your role is critical in designing sophisticated data solutions that drive business value for enterprise clients and power internal AI products. Your expertise will be instrumental in architecting scalable, high-performance data lakehouse platforms and end-to-end data pipelines, making you the go-to expert for modern data architecture in a cloud-first world. Your key responsibilities will include designing and documenting robust, end-to-end big data solutions on cloud platforms (AWS, Azure, GCP) with a focus on the Databricks Lakehouse Platform. You will provide technical guidance a...