2404 Azure Databricks Jobs - Page 46

As a Data Engineer, you will be responsible for designing and building efficient data pipelines using Azure Databricks (PySpark). You will implement business logic for data transformation and enrichment at scale, as well as manage and optimize Delta Lake storage solutions. Additionally, you will develop REST APIs using FastAPI to expose processed data and deploy them on Azure Functions for scalable and serverless data access. You will play a key role in data orchestration by developing and managing Airflow DAGs to orchestrate ETL processes. This includes ingesting and processing data from various internal and external sources on a scheduled basis. Database management will also be part of you...

As a Senior Software Engineer (SSE) specializing in Kafka, Python, and Azure Databricks, you will play a crucial role in leading and contributing to healthcare data engineering projects. Your primary responsibility will involve building scalable real-time data pipelines and processing large healthcare datasets within a secure and compliant cloud environment. Your key responsibilities will include designing and developing scalable real-time data streaming solutions using Apache Kafka and Python. You will also be tasked with architecting and implementing ETL/ELT pipelines utilizing Azure Databricks for both structured and unstructured healthcare data. Additionally, you will optimize and mainta...

Lead : 6-8 years Focus on production cost for the techniques and features • Mentoring the team on benchmarking costs, performance KPI’s • Guarding the focus on the team towards objectives •Advanced proficiency in Python and/or Scala for data engineering tasks. •Proficiency in PySpark and Scala Spark for distributed data processing, with hands-on experience in Azure Databricks. •Expertise in Azure Databricks for data engineering, including Delta Lake, MLflow, and cluster management. •Familiarity with cloud platforms (e.g., AWS, Azure, GCP) and their big data and data warehousing services (e.g., Azure Data Factory, AWS Redshift). •Expertise in data warehousing platforms such as Snowflake, Azur...

Company Overview Incedo is a US-based consulting, data science and technology services firm with over 3000 people helping clients from our six offices across US, Mexico and India. We help our clients achieve competitive advantage through end-to-end digital transformation. Our uniqueness lies in bringing together strong engineering, data science, and design capabilities coupled with deep domain understanding. We combine services and products to maximize business impact for our clients in telecom, Banking, Wealth Management, product engineering and life science & healthcare industries. Working at Incedo will provide you an opportunity to work with industry leading client organizations, deep te...

Role- Azure Devops EXP- 2+yrs Location- Pan INDIA Job description Hiring for Azure developer with experience range 2 to 9 years Mandatory Skills: Azure, ADF, ADB, Azure synapse Education: BE/B.Tech/BCA/B.SC/MCA/M.Tech/MSc./MS Location: Pan India Responsibilities A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to actively aid the consulting team in different phases of the project including problem definition, effort estimation, diagnosis, solution generation and design and deployment You will explore the alternatives to the recommended solutions based on research that includes literature surveys, information available in public domains, ven...

Dear Candidate, Hope you are doing well. Greeting from NAM Info INC. NAM Info Inc. is a technology-forward talent management organization dedicated to bridging the gap between industry leaders and exceptional human resources. They pride themselves on delivering quality candidates, deep industry coverage, and knowledge-based training for consultants. Their commitment to long-term partnerships, rooted in ethical practices and trust, positions them as a preferred partner for many industries. Learn more about their vision, achievements, and services on their website at www.nam-it.com. We have an open position for Data Engineer role with our company for Bangalore, Pune and Mumbai location. Job De...

Develop and execute test plans and cases to ensure software quality, identifying and reporting defects. Collaborate with developers to resolve issues, participate in code reviews, and maintain test documentation. Contribute to improving the QA process by applying testing best practices and utilizing bug tracking systems within the SDLC. Key Responsibilities Develop and execute test cases and test plans. Identify and report software defects. Perform functional, regression, and performance testing. Collaborate with developers to resolve issues. Participate in code reviews and provide feedback on testability. Document test results and maintain test documentation. Learn and apply software testin...

Key Responsibilities : Oversee the entire data infrastructure to ensure scalability, operation efficiency and resiliency. - Mentor junior data engineers within the organization. - Design, develop, and maintain data pipelines and ETL processes using Microsoft Azure services (e.g., Azure Data Factory, Azure Synapse, Azure Databricks, Azure Fabric). - Utilize Azure data storage accounts for organizing and maintaining data pipeline outputs. (e.g., Azure Data Lake Storage Gen 2 & Azure Blob storage). - Collaborate with data scientists, data analysts, data architects and other stakeholders to understand data requirements and deliver high-quality data solutions. - Optimize data pipelines in the Azu...

JOB DESCRIPTION: We at HCL Tech are looking for candidates for the job role of "Test Lead". Overview of the role- As a Senior Test Lead, you will spearhead cloud and data warehouse automation testing initiatives, ensuring the robustness and reliability of software solutions. This role is pivotal in maintaining the quality standards of HCL Tech cloud-based applications, directly impacting the company objectives to deliver top-notch technology solutions. Job Responsibilities: Develop and execute comprehensive test strategies for cloud applications. Implement and manage automated testing frameworks to enhance testing efficiency. Lead and mentor the testing team, ensuring timely and high-quality...

Job Title: Senior Engineer Work Location: PUNE, MH/ HYDERABAD, TA/ KOLKATA, WB/ BANGALORE, KA/ THANE, MH/ NOIDA, UP Skills Required: Databricks, Azure Data Factory Experience Range in Required Skills: 4-6 Years Job Description: Very strong hands-on experience in using Microsoft Azure services like Azure Data Factory, Data brick services (like processing streaming data using Spark clusters), usage of Blob containers, ESB, Event Grid, Azure SQL server, Cosmos DB, Azure functions, Analytics (like Power BI) is a mandatory requirement

SUMMARY We are looking for a highly motivated Azure Data engineer to join our dynamic development team. This is an exciting opportunity for an individual who is passionate about working on trending technology and wants to contribute to the growth of our company. KEY RESPONSIBILITIES Design, develop, and maintain scalable data pipelines and use Azure Data Factory and Azure Stream Analytics • Collaborate with data scientists and analysts to understand data requirements and implement solutions that support analytics and machine learning initiatives. • Optimize data storage and retrieval mechanisms to ensure performance, reliability, and cost-effectiveness. • Implement data governance and securi...

Design and build data pipelines using Spark-SQL and PySpark in Azure Databricks Design and build ETL pipelines using ADF Build and maintain a Lakehouse architecture in ADLS / Databricks. Perform data preparation tasks including data cleaning, normalization, deduplication, type conversion etc. Work with DevOps team to deploy solutions in production environments. Control data processes and take corrective action when errors are identified. Corrective action may include executing a work around process and then identifying the cause and solution for data errors. Participate as a full member of the global Analytics team, providing solutions for and insights into data related items. Collaborate wi...

Role Purpose The purpose of the role is to support process delivery by ensuring daily performance of the Production Specialists, resolve technical escalations and develop technical capability within the Production Specialists. Do Oversee and support process by reviewing daily transactions on performance parameters Review performance dashboard and the scores for the team Support the team in improving performance parameters by providing technical support and process guidance Record, track, and document all queries received, problem-solving steps taken and total successful and unsuccessful resolutions Ensure standard processes and procedures are followed to resolve all client queries Resolve cl...

Role: Senior Associate Tower: Data, Analytics & Specialist Managed Service Experience: 6 - 10 years Key Skills: Azure Educational Qualification: BE / B Tech / ME / M Tech / MBA Work Location: Bangalore Job Description As a Senior Associate, you will work as part of a team of problem solvers, helping to solve complex business issues from strategy to execution. PwC Professional skills and responsibilities for this management level include but are not limited to: Use feedback and reflection to develop self-awareness, personal strengths, and address development areas. Flexible to work in stretch opportunities/assignments. Demonstrate critical thinking and the ability to bring order to unstructur...

Application of Architectural Principles and Methods | Is responsible for the overall architecture of the data & analytics platform. Develops architectural guidelines for data management and implementation concepts of the data Required Candidate profile Azure platform, including Azure Data Factory, Azure SQL Database, Azure Synapse, Azure Databricks, and Power BI. Extensive knowledge of data modeling, ETL processes, and data warehouse

As a Mid Databricks Engineer, you will play a pivotal role in designing, implementing, and optimizing data processing pipelines and analytics solutions on the Databricks platform. You will collaborate closely with cross-functional teams to understand business requirements, architect scalable solutions, and ensure the reliability and performance of our data infrastructure. This role requires deep expertise in Databricks, strong programming skills, and a passion for solving complex engineering challenges. What you'll do : - Design and develop data processing pipelines and analytics solutions using Databricks. - Architect scalable and efficient data models and storage solutions on the Databrick...

Min 3+ yrs in Data engineer(GenAI platform) ETL/ELT workflows using AWS,Azure Databricks,Airflow,Azure Data Factory Exp in Azure Databricks,Snowflake,Airflow,Python,SQL,Spark,Spark Streaming,AWS EKS, CI/CD(Jenkins) Elasticsearch,SOLR,OpenSearch,Vespa

Experience : 6+ years as Azure Data Engineer including at least 1 E2E Implementation in Microsoft Fabric. Responsibilities : - Lead the design and implementation of Microsoft Fabric-centric data platforms and data warehouses. - Develop and optimize ETL/ELT processes within the Microsoft Azure ecosystem, effectively utilizing relevant Fabric solutions. - Ensure data integrity, quality, and governance throughout Microsoft Fabric environment. - Collaborate with stakeholders to translate business needs into actionable data solutions. - Troubleshoot and optimize existing Fabric implementations for enhanced performance. Skills : - Solid foundational knowledge in data warehousing, ETL/ELT processes...

- Building and operationalizing large scale enterprise data solutions and applications using one or more of AZURE data and analytics services in combination with custom solutions - Azure Synapse/Azure SQL DWH, Azure Data Lake, Azure Blob Storage, Spark, HDInsights, Databricks, CosmosDB, EventHub/IOTHub. - Experience in migrating on-premise data warehouses to data platforms on AZURE cloud. - Designing and implementing data engineering, ingestion, and transformation functions - Azure Synapse or Azure SQL data warehouse - Spark on Azure is available in HD insights and data bricks - Good customer communication. - Good Analytical skill

- Develop Azure Data Factory Objects - ADF pipeline, configuration, parameters, variables, Integration services runtime - Hands-on knowledge of ADF activities(such as Copy, SP, lkp etc) and DataFlows - ADF data Ingestion and Integration with other services Azure Databricks: - Experience in Big Data components such as Kafka, Spark SQL, Dataframes, HIVE DB etc implemented using Azure Data Bricks would be preferred. - Azure Databricks integration with other services - Read and write data in Azure Databricks - Best practices in Azure Databricks Synapse Analytics: - Import data into Azure Synapse Analytics with and without using PolyBase - Implement a Data Warehouse with Azure Synapse Analytics -...

Overview Seeking an Associate Manager, Data Operations, to support our growing data organization. In this role, you will assist in maintaining data pipelines and corresponding platforms (on-prem and cloud) while working closely with global teams on DataOps initiatives. Support the day-to-day operations of data pipelines, ensuring data governance, reliability, and performance optimization on Microsoft Azure. Hands-on experience with Azure Data Factory (ADF), Azure Synapse Analytics, Azure Databricks, and real-time streaming architectures is preferred. Assist in ensuring the availability, scalability, automation, and governance of enterprise data pipelines supporting analytics, AI/ML, and busi...

Responsibilities A day in the life of an Infoscion • As part of the Infosys delivery team, your primary role would be to ensure effective Design, Development, Validation and Support activities, to assure that our clients are satisfied with the high levels of service in the technology domain. • You will gather the requirements and specifications to understand the client requirements in a detailed manner and translate the same into system requirements. • You will play a key role in the overall estimation of work requirements to provide the right information on project estimations to Technology Leads and Project Managers. • You would be a key contributor to building efficient programs/ systems ...

Azure Data Factory: - Develop Azure Data Factory Objects - ADF pipeline, configuration, parameters, variables, Integration services runtime - Hands-on knowledge of ADF activities(such as Copy, SP, lkp etc) and DataFlows - ADF data Ingestion and Integration with other services Azure Databricks: - Experience in Big Data components such as Kafka, Spark SQL, Dataframes, HIVE DB etc implemented using Azure Data Bricks would be preferred. - Azure Databricks integration with other services - Read and write data in Azure Databricks - Best practices in Azure Databricks Synapse Analytics: - Import data into Azure Synapse Analytics with and without using PolyBase - Implement a Data Warehouse with Azure...

We are looking for a Strategic thinker, have ability grasp new technologies, innovate, develop, and nurture new solutions. a self-starter to work in a diverse and fast-paced environment to support, maintain and advance the capabilities of the unified data platform. This a global role that requires partnering with the broader JLLT team at the country, regional and global level by utilizing in-depth knowledge of cloud infrastructure technologies and platform engineering experience. Responsibilities Developing ETL/ELT pipelines using Synapse pipelines and data flows Integrating Synapse with other Azure data services (Data Lake Storage, Data Factory, etc.) Building and maintaining data warehousi...

- Building and operationalizing large scale enterprise data solutions and applications using one or more of AZURE data and analytics services in combination with custom solutions - Azure Synapse/Azure SQL DWH, Azure Data Lake, Azure Blob Storage, Spark, HDInsights, Databricks, CosmosDB, EventHub/IOTHub. - Experience in migrating on-premise data warehouses to data platforms on AZURE cloud. - Designing and implementing data engineering, ingestion, and transformation functions - Azure Synapse or Azure SQL data warehouse - Spark on Azure is available in HD insights and data bricks - Good customer communication. - Good Analytical skill