601 Delta Lake Jobs - Page 14

Job Title: Senior Data Engineer Basic Function The Senior Data Engineer should be an expert familiar with all areas of data warehousing technical components (e.g., ETL, Reporting, Data Model), connected infrastructure, and their integrations. The ideal candidate will be responsible for developing the overall architecture and high-level design of the data schema environment. The candidate must have extensive experience with Star Schemas, Dimensional Models, and Data Marts. The individual is expected to build efficient, flexible, extensible, and scalable ETL design and mappings. Excellent written and verbal communication skills are required as the candidate will work very closely with diverse ...

Your journey at Crowe starts here with an opportunity to build a meaningful and rewarding career. At Crowe, you will enjoy real flexibility to balance work with life moments while being trusted to deliver results and make a significant impact. You will be embraced for who you are, your well-being will be cared for, and your career will be nurtured. Equitable access to opportunities for career growth and leadership is available to everyone. With a history spanning over 80 years, delivering excellent service through innovation is ingrained in our DNA across audit, tax, and consulting groups. We continuously invest in innovative ideas like AI-enabled insights and technology-powered solutions to...

You should have 5+ years of experience with skills in DataBricks, Delta Lake, pyspark or scala spark, unity catalog. Good to have skills in Azure/AWS Cloud. You will be responsible for ingesting and transforming batch and streaming data on the Databricks Lakehouse Platform. Excellent communication skills are required. Your responsibilities will include overseeing and supporting processes by reviewing daily transactions, reviewing performance dashboards, supporting the team in improving performance parameters, recording and tracking all queries received, ensuring standard processes are followed, resolving client queries within defined SLAs, developing understanding of processes/products for t...

You should have at least 5 years of hands-on experience in data engineering/ETL using Databricks on AWS/Azure cloud infrastructure and functions. Additionally, you must possess a minimum of 3 years of experience in Power BI and Data Warehousing, with the ability to perform root cause analysis on internal and external data and processes to address specific business questions and identify areas for improvement. It is preferred that you have experience with AWS services such as S3, Athena, Glue, and Lambda. A deep understanding of data warehousing concepts including Dimensional (star-schema), SCD2, Data Vault, Denormalized, OBT is essential for implementing highly performant data ingestion pipe...

The ideal candidate will be responsible for designing and implementing streaming data pipelines that integrate Kafka with Databricks using Structured Streaming. You will also be tasked with architecting and maintaining the Medallion Architecture, which consists of well-defined Bronze, Silver, and Gold layers. Additionally, you will need to implement efficient data ingestion processes using Databricks Autoloader for high-throughput data loads. You will work with large volumes of structured and unstructured data to ensure high availability and performance, applying performance tuning techniques like partitioning, caching, and cluster resource optimization. Collaboration with cross-functional t...

As a skilled Data Engineer with 3 to 5 years of experience, you will be responsible for designing, building, and maintaining robust ETL/ELT pipelines using Databricks (PySpark/SQL). You will develop and optimize data workflows and pipelines on Delta Lake and Databricks Lakehouse architecture. Your role will involve integrating data from multiple sources to ensure data quality, reliability, and performance. Collaboration with data scientists, analysts, and business stakeholders will be essential to translate requirements into scalable data solutions. Monitoring and troubleshooting production data pipelines for performance and cost optimization will also be part of your responsibilities. Worki...

You are a Spark & Delta Lake Specialist with 8+ years of experience and proficiency in Python, Spark (PySpark), SQL, and Delta Lake. Your role involves understanding Spark core concepts such as RDDs, DataFrames, DataSets, SparkSQL, and Spark Streaming. You have experience with Spark optimization techniques and deep knowledge of Delta Lake features like time travel, schema evolution, and data partitioning. Your responsibilities include designing and implementing data pipelines using Spark and Delta Lake as the data storage layer. You are expected to have proficiency in Python, Scala, Java for Spark development, and integration with ETL processes. Additionally, you have knowledge of data inges...

The ML Platform Specialist role involves designing, implementing, and maintaining machine learning infrastructure and workflows on the Databricks Lakehouse Platform. Your primary responsibility will be to ensure the successful deployment, monitoring, and scaling of machine learning models across the organization. You will design and implement scalable ML infrastructure on the Databricks Lakehouse Platform, develop CI/CD pipelines for machine learning models, and create automated testing and validation processes using Databricks MLflow. Additionally, you will be responsible for managing model monitoring systems, collaborating with various teams to optimize machine learning workflows, and main...

Line of Service Advisory Industry/Sector Not Applicable Specialism Data, Analytics & AI Management Level Senior Associate Job Description & Summary At PwC, our people in data and analytics focus on leveraging data to drive insights and make informed business decisions. They utilise advanced analytics techniques to help clients optimise their operations and achieve their strategic goals. In business intelligence at PwC, you will focus on leveraging data and analytics to provide strategic insights and drive informed decision-making for clients. You will develop and implement innovative solutions to optimise business performance and enhance competitive advantage. *Why PWC At PwC, you will be pa...

Key Responsibilities: Design and develop the migration strategies and processes Collaborate with stakeholders to understand business requirements and technical challenges. Analyze current data and scope for optimization during the migration process. Define the architecture and roadmap for cloud-based data and analytics transformation on Databricks. Design, implement, and optimize scalable, high-performance data architectures using Databricks. Build and manage data pipelines and workflows within Databricks. Ensure that best practices for security, scalability, and performance are followed. Implement Databricks solutions that enable machine learning, business intelligence, and data science wor...

As an Advisory Consultant at Dell Technologies, you will play a crucial role in delivering consultative business and technical services for complex customer-facing consulting engagements related to digital transformation. Your responsibilities will involve collaborating with Global Pre-Sales, Account Management, and Solutioning Teams to deploy, administer, and configure digital transformation software stacks. Being one of the senior technical members in the Digital Transformation Practice, you will earn customer trust through competence, technical acumen, consulting expertise, and partnership. You will guide and oversee other team members, providing technical grooming activities for their sk...

Key Skills: Spark, Azure Databricks, Apache Spark, Python, Delta Lake, Kafka, Snowflake, Qlik, CI/CD, Azure Roles & Responsibilities: Design, develop, and maintain scalable data pipelines using Azure Databricks and Spark. Implement data processing solutions with Delta Lakes and manage multiple file formats. Collaborate with cross-functional teams to ensure data quality and integrity. Utilize CI/CD practices for efficient deployment and management of data solutions. Work with Snowflake and Qlik for data visualization and reporting. Leverage Kafka or similar streaming technologies for real-time data processing. Ensure compliance with data governance and security standards in Azure. Experience ...

Why Join Us Are you inspired to grow your career at one of India's Top 25 Best Workplaces in IT industry Do you want to do the best work of your life at one of the fastest growing IT services companies Do you aspire to thrive in an award-winning work culture that values your talent and career aspirations It's happening right here at Iris Software. About Iris Software At Iris Software, our vision is to be our client's most trusted technology partner, and the first choice for the industry's top professionals to realize their full potential. With over 4,300 associates across India, U.S.A, and Canada, we help our enterprise clients thrive with technology-enabled transformation across financial s...

Why Join Us Are you inspired to grow your career at one of India's Top 25 Best Workplaces in IT industry Do you want to do the best work of your life at one of the fastest growing IT services companies Do you aspire to thrive in an award-winning work culture that values your talent and career aspirations It's happening right here at Iris Software. About Iris Software At Iris Software, our vision is to be our client's most trusted technology partner, and the first choice for the industry's top professionals to realize their full potential. With over 4,300 associates across India, U.S.A, and Canada, we help our enterprise clients thrive with technology-enabled transformation across financial s...

Work Location: ~CHENNAI, TN. Skill Required: Digital : Cloud Application Migration~Data Migration Experience Range in Required Skills: 5-7 Job Description: • Data Pipeline Design: Architect and implement scalable, reliable data pipelines using Databricks, Spark, and Delta Lake • Medallion Architecture Implementation : Ingest raw data into the Bronze layer, process and clean it in the Silver layer, and aggregate/transform it for analytics in the Gold layer • Data Modeling : Design and optimize data models for efficient storage, retrieval, and analytics • Data Quality & Security: Implement data validation, quality checks, and security controls throughout the pipeline • Automation & Monitoring:...

About the Role We are seeking a highly skilled Senior Data Engineer to design, build, and optimize scalable data platforms that empower data-driven decision-making. This role requires deep technical expertise in modern data engineering frameworks, architectural patterns, and cloud-native solutions on AWS. You will be a key contributor to our data strategy, ensuring data quality, governance, and reliability while mentoring other engineers in the team. Key Responsibilities Design, develop, and own robust, scalable, and maintainable data pipelines (batch & real-time). Architect and implement Data Lake, Data Warehouse, and Lakehouse solutions using modern frameworks and architectural patterns. E...

You specialize in modeling ServiceNow data for analytics, transforming raw extracts into curated lakehouse formats. This includes creating partitioned layers, business domains, and semantic enrichment for consumption by GenAI and BI tools. You ensure schema enforcement, lineage capture, and data classification are integrated into the transformation logic. Additionally, you have experience in building Silver and Gold zones for structured ITSM and CMDB use cases. Your expertise includes ServiceNow schema knowledge, data modeling, Delta Lake, and governance tagging. You are proficient in tools such as Databricks and Delta Lake, as well as holding the Azure Data Engineer Associate certification....

You will be working as a full-time on-site Databricks Developer in Hyderabad. Your responsibilities will include designing, developing, and maintaining highly scalable efficient data pipelines using Databricks, PySpark, and other related technologies to process large-scale datasets. Collaboration with cross-functional teams to design and implement data engineering and analytics solutions will be a key part of your role. To excel in this role, you should have expertise in using Unity Catalog and Metastore, optimizing Databricks notebooks, delta lake, and DLT pipelines. You should also be experienced in using Databricks SQL, implementing highly configured data processing solutions, and solutio...

Key Result Areas and Activities: Design and implement Lakehouse architectures using Databricks, Delta Lake, and Apache Spark. Lead the development of data pipelines, ETL/ELT processes, and data integration strategies. Collaborate with business and technical teams to define data architecture standards, governance, and security models. Optimize performance and cost-efficiency of Databricks clusters and jobs. Provide technical leadership and mentorship to data engineers and developers. Integrate Databricks with cloud platforms (Azure, AWS, or GCP) and enterprise systems. Evaluate and recommend tools and technologies to enhance the data ecosystem. Ensure compliance with data privacy and regulato...

Data Architect - Databricks (Azure/AWS) Role Overview: We are seeking an experienced Data Architect specializing in Databricks to lead the architecture, design, and migration of enterprise data workloads from on-premises systems (e.g., Oracle, Exadata, Hadoop) to Databricks on Azure or AWS . The role involves designing scalable, secure, and high-performing data platforms based on the medallion architecture (bronze, silver, gold layers), supporting large-scale ingestion, transformation, and publishing of data. Required Skills and Experience: 8+ years of experience in data architecture or engineering roles, with at least 3+ years specializing in cloud-based big data solutions. Hands-on experti...

As a Senior Azure Data Engineer, your responsibilities will include: Building scalable data pipelines using Databricks and PySpark Transforming raw data into usable business insights Integrating Azure services like Blob Storage, Data Lake, and Synapse Analytics Deploying and maintaining machine learning models using MLlib or TensorFlow Executing large-scale Spark jobs with performance tuning on Spark Pools Leveraging Databricks Notebooks and managing workflows with MLflow Qualifications: Bachelors/Masters in Computer Science, Data Science, or equivalent 7+ years in Data Engineering, with 3+ years in Azure Databricks Strong hands-on in: PySpark, Spark SQL, RDDs, Pandas, NumPy, Delta Lake Azur...

Mode: Contract As an Azure Data Architect, you will: Lead architectural design and migration strategies, especially from Oracle to Azure Data Lake Architect and build end-to-end data pipelines leveraging Databricks, Spark, and Delta Lake Design secure, scalable data solutions integrating ADF, SQL Data Warehouse, and on-prem/cloud systems Optimize cloud resource usage and pipeline performance Set up CI/CD pipelines with Azure DevOps Mentor team members and align architecture with business needs Qualifications: 10-15 years in Data Engineering/Architecture roles Extensive hands-on with: Databricks, Azure Data Factory, Azure SQL Data Warehouse Data integration, migration, cluster configuration, ...

Role Overview: We are seeking an experienced and visionary AI & Distributed Systems Architect to lead the design and implementation of large-scale, AI-driven distributed systems. The successful candidate will pioneer transformative AI solutions, including leveraging Generative AI, Machine Learning (ML), sophisticated agentic workflows, and context-aware data platforms. You will play a crucial role in defining architecture strategies, driving innovation, and leading technical teams to deliver cutting-edge capabilities at scale. Key Responsibilities: Lead the architecture design, development, and deployment of AI-powered distributed systems. Develop scalable, reliable, and high-performance sol...

We are seeking an experienced PySpark Developer / Data Engineer to design, develop, and optimize big data processing pipelines using Apache Spark and Python (PySpark). The ideal candidate should have expertise in distributed computing, ETL workflows, data lake architectures, and cloud-based big data solutions. Key Responsibilities: Develop and optimize ETL/ELT data pipelines using PySpark on distributed computing platforms (Hadoop, Databricks, EMR, HDInsight). Work with structured and unstructured data to perform data transformation, cleansing, and aggregation. Implement data lake and data warehouse solutions on AWS (S3, Glue, Redshift), Azure (ADLS, Synapse), or GCP (BigQuery, Dataflow). Op...

Work Mode: Hybrid (3 days WFO) Locations: Bangalore, Noida, Pune, Mumbai, Hyderabad (Candidates must be in Accion cities to collect assets and attend in-person meetings as required). Key Requirements: Technical Skills: Databricks Expertise: 5+ years of hands-on experience in data engineering/ETL using Databricks on AWS/Azure cloud infrastructure. Proficiency in Delta Lake, Unity Catalog, Delta Sharing, Delta Live Tables (DLT), MLflow, and Databricks SQL. Experience with Databricks CI/CD tools (e.g., BitBucket, GitHub Actions, Databricks CLI). Data Warehousing & Engineering: Strong understanding of data warehousing concepts (Dimensional, SCD2, Data Vault, OBT, etc.). Proven ability to impleme...