601 Delta Lake Jobs - Page 16

You have 5+ years of experience in designing and building data pipelines using Apache Spark, Databricks, or equivalent big data frameworks. You possess hands-on expertise with streaming and messaging systems like Apache Kafka, Confluent Cloud, RabbitMQ, or Azure Event Hub. Your experience includes creating producers, consumers, and topics, and integrating them into downstream processing. You have a deep understanding of relational databases and Change Data Capture (CDC). You are proficient in SQL Server, Oracle, or other RDBMSs, and have experience capturing change events using tools like Debezium or native CDC tools, transforming them for downstream consumption. Your proficiency extends to ...

You will be responsible for building and maintaining secure, scalable data pipelines using Databricks and Azure. This includes handling ingestion from various sources such as files, APIs, and streaming, performing data transformation, and ensuring quality validation. Collaboration with subsystem data science and product teams will be essential to prepare for machine learning readiness. Your technical expertise should cover working with Notebooks (SQL, Python), Delta Lake, Unity Catalog, ADLS/S3, job orchestration, APIs, structured logging, and Infrastructure as Code (IaC) tools like Terraform. Delivery practices like Trunk-based development, Test-Driven Development (TDD), Git, and Continuous...

You should have at least 5 years of experience working as a Data Engineer. Your expertise should include a strong background in Azure Cloud services and proficiency in tools such as Azure Databricks, PySpark, and Delta Lake. It is essential to have solid experience in Python and FastAPI for API development, as well as familiarity with Azure Functions for serverless API deployments. Experience in managing ETL pipelines using Apache Airflow is also required. Hands-on experience with databases like PostgreSQL and MongoDB is necessary. Strong SQL skills and the ability to work with large datasets are key for this role.,

About the Company At Tide, we are dedicated to creating a business management platform that aims to streamline operations for small businesses, enabling them to save valuable time and resources. Our services include offering business accounts, banking solutions, as well as a range of integrated administrative tools spanning from invoicing to accounting. Established in 2017, Tide has garnered a user base of over 1 million small businesses globally, catering to SMEs in the UK, India, and Germany. Headquartered in central London, we also have offices in Sofia, Hyderabad, Delhi, Berlin, and Belgrade, with a team of more than 2,000 employees. Tide is on a trajectory of rapid growth, continuously ...

The Infor Data Services team at Infor is looking for a Software Engineer to develop customer-focused data solutions. As a Software Engineer, you will primarily focus on building backend systems for Infor Datamesh, a platform that provides scalable and robust data solutions for enterprise customers. Your responsibilities will include designing, developing, and maintaining backend services and microservices using Python, Docker, and Kafka. You will also be integrating these services with various databases and external storage solutions to ensure high performance and scalability in alignment with Infor's data management strategy. Your essential duties will involve developing and maintaining dat...

You are an experienced Data Engineer proficient in Databricks, PySpark, and Python, seeking to join a healthcare data-focused team. Your responsibilities will include designing and maintaining scalable data pipelines and solutions for complex healthcare datasets, utilizing Azure Cloud and advanced distributed data processing frameworks. Strong client-facing communication skills are crucial as you will engage directly with US-based stakeholders. Your key responsibilities will involve designing, developing, and maintaining large-scale data processing systems using Databricks and PySpark. You will build and optimize robust, scalable data pipelines for data ingestion, cleaning, transformation, a...

We have an exciting and rewarding opportunity for you to take your software engineering career to the next level. As a Software Engineer, you will be responsible for executing software solutions, design, development, and technical troubleshooting with the ability to think beyond routine or conventional approaches. Your primary focus will be on creating secure and high-quality production code, maintaining algorithms that run synchronously with appropriate systems, and producing architecture and design artifacts for complex applications. You will also be accountable for ensuring design constraints are met by software code development. In this role, you will gather, analyze, synthesize, and dev...

At EY, you'll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we're counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. Our Technology team builds innovative digital solutions rapidly and at scale to deliver the next generation of Financial and Non-Financial services across the globe. The Position is a senior technical, hands-on delivery role, requiring the knowledge of data engineering, cloud infrastructure and platform engineering, platform operations, and p...

As a Data Engineer, you will be responsible for designing and building efficient data pipelines using Azure Databricks (PySpark). Your role will involve implementing business logic for data transformation and enrichment at scale and managing and optimizing Delta Lake storage solutions. You will also be involved in developing REST APIs using FastAPI to expose processed data and deploying APIs on Azure Functions for scalable and serverless data access. Additionally, you will develop and manage Airflow DAGs to orchestrate ETL processes, ingest and process data from various internal and external sources on a scheduled basis. Database management will be a key part of your responsibilities, includ...

As a Lead Data Engineer at Rearc, you will play a crucial role in establishing and maintaining technical excellence within our data engineering team. Your extensive experience in data architecture, ETL processes, and data modeling will be key in optimizing data workflows for efficiency, scalability, and reliability. Collaborating closely with cross-functional teams, you will design and implement robust data solutions that align with business objectives and adhere to best practices in data management. Building strong partnerships with technical teams and stakeholders is essential as you drive data-driven initiatives and ensure their successful implementation. With over 10 years of experience ...

You will need to have at least 7-9 years of experience to apply for the role of Microsoft Fabric Architect. In this role, you will be responsible for working hands-on with Microsoft Fabric, which includes Lakehouse, Synapse, Pipelines, and Dataflows. You should be proficient in DAX, Power Query (M), and SQL, and have a strong understanding of Delta Lake, Parquet, and OneLake storage. Knowledge of Spark Notebooks, KQL, and Data Engineering within Fabric will be an added advantage. Experience with source control integration (Git) and CI/CD pipelines is essential for this role. Familiarity with Azure Data Factory, Azure Synapse Analytics, or Databricks will be considered a plus. You should poss...

As an Integration Developer at our organization, you will be an integral part of the data engineering and integration team. Your primary responsibility will be to design, develop, and maintain ETL/ELT pipelines using tools such as SSIS and Azure Data Factory (ADF). You will play a crucial role in building reliable data integration pipelines that support various business functions including business intelligence, analytics, and operational workloads. Your key responsibilities will include implementing robust data workflows to ingest, transform, and store structured and unstructured data in Azure Data Lakehouse, integrating cloud and on-premise systems for high performance and data quality, an...

Project Role : Cloud Migration Engineer Project Role Description : Provides assessment of existing solutions and infrastructure to migrate to the cloud. Plan, deliver, and implement application and data migration with scalable, high-performance solutions using private and public cloud technologies driving next-generation business outcomes. Must have skills : Microsoft Azure DevOps Good to have skills : NA Minimum 5 Year(s) Of Experience Is Required Educational Qualification : 15 years full time education We are seeking a hands-on senior professional who can lead a team of data engineers to lay the groundwork for making data accessible to AI engineers for research and/or application developme...

You are seeking a skilled Azure Databricks Architect with 12 to 17 years of experience in Python, SQL. As an Azure Databricks Architect at our company, you will be responsible for data architecture, data engineering, and analytics. You should have at least 5 years of hands-on experience with Azure Databricks, Apache Spark, and Delta Lake. Your proficiency in Azure Data Lake, Azure Synapse, Azure Data Factory, and Azure SQL is essential. Expertise in Python, Scala, and SQL for data processing, along with a deep understanding of data modeling, ETL/ELT processes, and distributed computing, are key requirements. Experience with CI/CD pipelines and DevOps practices in data engineering is also exp...

At EY, youll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we're counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. We are looking for a BI Senior Consultant to join the EY GDS Team. As part of our EY GDS TechOps team, you will be responsible for providing expert-level business intelligence support with a strong focus on Power BI and Databricks. You will work across various regions for our global clients, helping to design, develop, and maintain insightful ...

We are looking for an experienced Data Engineer with strong expertise in Databricks and Azure Data Factory (ADF) to design, build, and manage scalable data pipelines and integration solutions. The ideal candidate will have a solid background in big data technologies, cloud platforms, and data processing frameworks to support enterprise-level data transformation and analytics initiatives. Key Responsibilities: Design, develop, and maintain robust data pipelines using Azure Data Factory and Databricks . Build and optimize data flows and transformations for structured and unstructured data. Develop scalable ETL/ELT processes to extract data from various sources including SQL, APIs, and flat fil...

Roles & Responsibilities: Architect and maintain robust, scalable data pipelines using Databricks, Spark, and Delta Lake for both batch and real-time data processing. Lead technology evaluation and adoption initiatives to improve productivity, scalability, and data delivery. Optimize data processing performance through Spark tuning, job scheduling, and efficient resource utilization. Develop innovative solutions to enhance data ingestion, transformation, lineage tracking, and observability. Build metadata-driven frameworks to promote pipeline consistency and reuse. Promote a culture of engineering excellence, continuous improvement, and experimentation. Collaborate with architecture, platfor...

Key Responsibilities: Ontology Development: Design and implement ontologies based on BFO and CCO principles, ensuring alignment with business needs and industry standards. Collaborate with domain experts to capture and formalize domain knowledge into ontological structures. Develop and maintain comprehensive ontologies modeling business entities, relationships, and processes. Data Modeling: Design and implement semantic and syntactic data models adhering to ontological principles. Create scalable, flexible, and adaptable data models to meet evolving business requirements. Integrate data models with existing data infrastructure and applications. Knowledge Graph Implementation: Design and buil...

The Databricks Data Engineering Lead role requires a highly skilled individual who will architect and lead the implementation of scalable, high-performance data pipelines and platforms using the Databricks Lakehouse ecosystem. As a Data Engineering Lead, you will be responsible for managing a team of data engineers, establishing best practices, and collaborating with cross-functional stakeholders to unlock advanced analytics, AI/ML, and real-time decision-making capabilities. Your key responsibilities will include leading the design and development of modern data pipelines, data lakes, and lakehouse architectures using Databricks and Apache Spark. You will manage and mentor a team of data en...

NTT DATA is looking for a Databricks Engineer to join their team in Bangalore, Karnataka, India. As a Databricks Engineer, you will be responsible for various tasks related to data extraction, ETL pipeline modernization, job design, development, automation, metadata management, documentation, testing, collaboration, performance tuning, security, governance, and compliance. Your primary job duties will include extracting and analyzing data from SQL Server and Teradata sources, translating legacy SQL/DataStage transformations into Databricks-native code, building and orchestrating jobs within Databricks using tools like Databricks Workflows, Delta Lake, and Auto Loader, generating and maintain...

You are an experienced Senior Data Engineer with a strong background in Databricks, Apache Airflow, SQL, and Python. Your main responsibility will involve designing and optimizing data pipelines, ETL workflows, and cloud-based data solutions. Your expertise in big data processing, cloud-based analytics, and automation will be crucial in developing efficient and scalable data engineering solutions. You will be responsible for tasks such as designing, developing, and optimizing data pipelines using Databricks and Apache Airflow. Implementing PySpark-based transformations and processing in Databricks to handle large-scale data, and maintaining SQL-based data pipelines with a focus on performanc...

We have a great opportunity for the role of Senior Data Engineer - Azure, Databricks. Relevant Exp: 5+ Yrs Job Description:- Mandatory Skills -Platform Engineer, Databricks, Azure Cloud, Data Engineer, Pyspark, SQL, Databricks. o Hands-on in data structures, distributed systems, Spark, SQL, PySpark, NoSQL Databases. o Strong software development skills in at least one of: Python, Pyspark or Scala. o Develop and maintain scalable & modular data pipelines using databricks & Apache Spark. o Experience building and supporting large-scale systems in a production environment. o Experience working on either of the cloud storages - AWS S3, Azure data lake, GCS buckets. o Exposure to the ELT tools of...

The job is based in Hyderabad/Bangalore/Chennai and at Gramener, you will find a welcoming work environment with diverse colleagues, a clear career path, and ample opportunities for growth and innovation. The company aims to develop a range of easily configurable data applications focused on storytelling for both public and private use. As part of your role, you will be involved in various impactful customer technical data platform projects, taking the lead on strategic initiatives that cover the design, development, and deployment of cutting-edge solutions. Collaboration with platform engineering teams will be key in implementing Data Brick services effectively within the company's infrastr...

About GyanSys Inc. GyanSys Inc. is a leading global systems integrator headquartered in the U.S., specializing in SAP, Salesforce, Microsoft, and ServiceNow solutions. With over 2,000 consultants worldwide and 300+ enterprise clients, GyanSys is at the forefront of delivering digital transformation and business process automation across industries. Our India delivery centers in Bengaluru, Hyderabad, and Pune play a critical role in driving global implementations for Fortune 500 companies. At GyanSys, youll find a collaborative culture, cutting-edge projects, continuous learning opportunities , and the ability to make a real impact . Job Title: Databricks Consultant Experience: 3-5 Years Loca...

We are looking for a seasoned Senior Data Architect with extensive knowledge in Databricks and Microsoft Fabric to join our team. In this role, you will be responsible for leading the design and implementation of scalable data solutions for BFSI and HLS clients. As a Senior Data Architect specializing in Databricks and Microsoft Fabric, you will play a crucial role in architecting and implementing secure, high-performance data solutions on the Databricks and Azure Fabric platforms. Your responsibilities will include leading discovery workshops, designing end-to-end data pipelines, optimizing workloads for performance and cost efficiency, and ensuring compliance with data governance, security...