1262 Azure Databricks Jobs - Page 7

Data Engineer Job Summary: The Data Engineer responsible for implementing and managing the operational aspects of cloud-native and hybrid data platform solutions built with Azure Databricks. They ensure the efficient and effective functioning of the Azure Databricks environment, including monitoring and troubleshooting data pipelines, managing data storage and access, and optimizing performance. They work closely with data engineers, data scientists, and other stakeholders to understand data requirements, design solutions, and implement data integration and transformation processes. Key Responsibilities: Provide expertise and ownership of Azure Databricks development tasks within the scrum team. Interact effectively with clients and leadership and can adapt communication for the appropriate audience. Read and comprehend software requirements, assisting with development of agile user stores and tasks. Assist with troubleshooting configuration and performance issues. Assist with Azure Databricks deployments, testing, configuration, and installation. Ensure security is a priority and understand the various areas where security vulnerabilities arise with database technologies. Ensure database resiliency, and disaster recovery capabilities. Required Skills & Qualifications: 5+ years proven experience working with Azure Databricks Analytics database capabilities, specifically Azure Databricks and other relational database technologies supported in Azure. 5+ years proven experience with Azure Data Lake Storage Gen 2, Azure Databricks, Azure Data Explorer, Azure Event Hubs, Spark Pools, Python, PySpark, SQL, Azure Landing Zone, Azure Networking Services, Microsoft EntraID. 5+ years proven experience with Azure geo-redundancy, HA/failover technologies. 5+ years proven experience designing and implementing data pipelines using Azure Databricks for data cleaning, transformation, and loading into Data Lakehouse. 5+ years proven experience with Infrastructure as Code (IaC) tools such as Terraform. 5+ years proven experience with programming languages such as Python, PySpark and data constructs such as JSON or XML.

Job Summary We are seeking a highly skilled Sr. Developer with 6 to 9 years of experience in Azure Databricks Azure Data Factory Azure ML Studio ML Ops Docker AI & ML Concepts Machine Learning Artificial Intelligence Kubernetes Terraform and Azure Machine Learning. The ideal candidate will work from our office during day shifts and will not be required to travel. This role is crucial in driving our AI and ML initiatives to enhance our business solutions and impact society positively. Responsibilities Develop and implement advanced machine learning models using Azure ML Studio to solve complex business problems. Design and manage data pipelines with Azure Data Factory to ensure efficient data flow and processing. Utilize Azure Databricks for big data analytics and machine learning model training. Implement ML Ops practices to streamline the deployment and monitoring of machine learning models. Use Docker to containerize applications and ensure consistent environments for development and production. Apply AI and ML concepts to create innovative solutions that improve business processes. Collaborate with cross-functional teams to integrate machine learning models into existing systems. Manage Kubernetes clusters to orchestrate containerized applications. Use Terraform for infrastructure as code to automate the provisioning of cloud resources. Ensure version control and collaboration using GIT for all development activities. Provide technical guidance and mentorship to junior developers on the team. Conduct code reviews to maintain high-quality standards and best practices. Stay updated with the latest advancements in AI and ML to continuously improve our solutions. Qualifications Must have extensive experience with Azure Databricks Azure Data Factory and Azure ML Studio. Should have a strong understanding of ML Ops practices and Docker. Must be proficient in AI & ML Concepts Machine Learning and Artificial Intelligence. Should have experience with Kubernetes and Terraform for managing cloud infrastructure. Must be skilled in using GIT for version control and collaboration. Nice to have experience in other cloud platforms and related technologies. Should possess excellent problem-solving skills and the ability to work in a team environment. Must have strong communication skills to effectively collaborate with stakeholders. Should be detail-oriented and able to manage multiple tasks efficiently. Must be committed to continuous learning and staying updated with industry trends. Should have a proactive approach to identifying and addressing potential issues. Certifications Required Azure Data Engineer Associate Azure AI Engineer Associate Certified Kubernetes Administrator

We are seeking a skilled and proactive Data Engineer with 26 years of hands-on experience in building robust data solutions on the Azure platform. The ideal candidate will have expertise in Azure Data Factory, Azure Data Lake, Azure Databricks, SQL, and Python, with additional experience in Power BI or Tableau considered a plus. You will be instrumental in developing and maintaining scalable data pipelines and ensuring efficient data integration and transformation to drive business intelligence and analytics initiatives. Key Responsibilities Design, build, and manage data pipelines and data integration workflows using tools like Azure Data Factory and Azure Databricks. •Develop scalable ETL/ELT processes to ingest and transform structured and unstructured data from various internal and external sources. •Implement data warehousing solutions using Azure Synapse Analytics, Delta Lake, or other suitable Azure storage layers. •Collaborate with Data Scientists, Analysts, and Business Stakeholders to understand data requirements and deliver data models that support advanced analytics and reporting. •Optimize data workflows for performance and cost efficiency in a cloud-first environment. •Manage data in Azure Data Lake Gen2, including partitioning, schema management, and access control. •Ensure data quality, security, and compliance with organizational and industry standards. •Monitor and troubleshoot data pipeline issues; implement logging, alerting, and recovery mechanisms. •Maintain comprehensive documentation of pipelines, data flow diagrams, and design specifications. •Contribute to the development and enforcement of data governance and metadata management practices. Required Skills and Qualifications • •Strong proficiency in: •Python for data manipulation and pipeline orchestration •SQL for querying and transforming large datasets •Azure Data Factory, Azure Data Lake Storage Gen2, Azure Databricks, and Synapse Analytics •Experience in building Delta Lake tables, working with Apache Spark, and optimizing performance within Databricks notebooks or jobs. •Understanding of DevOps for Data using tools like Azure DevOps, Git, and CI/CD pipelines for deploying data workflows. •Familiarity with Power BI or Tableau for data validation or collaboration with reporting teams. •Experience with handling semi-structured data (JSON, Parquet, CSV, XML). •Knowledge of data governance, security (RBAC, ACLs), and compliance best practices. Preferred/Bonus Skills •Working knowledge of Microsoft Purview for data cataloging and governance. •Exposure to event-driven architecture using Event Hubs, Service Bus, or Azure Functions. •Experience with streaming data pipelines using Azure Stream Analytics or Structured Streaming in Databricks. •Familiarity with cost monitoring and performance tuning in cloud environments.

Responsibilities Manage a team of data engineers and data analysts by delegating project responsibilities and managing their flow of work as well as empowering them to realize their full potential. Design, structure and store data into unified data models and link them together to make the data reusable for downstream products. Manage and scale data pipelines from internal and external data sources to support new product launches and drive data quality across data products. Create reusable accelerators and solutions to migrate data from legacy data warehouse platforms such as Teradata to Azure Databricks and Azure SQL. Enable and accelerate standards-based development prioritizing reuse of code, adopt test-driven development, unit testing and test automation with end-to-end observability of data Build and own the automation and monitoring frameworks that captures metrics and operational KPIs for data pipeline quality, performance and cost. Collaborate with internal clients (product teams, sector leads, data science teams) and external partners (SI partners/data providers) to drive solutioning and clarify solution requirements. Evolve the architectural capabilities and maturity of the data platform by engaging with enterprise architects to build and support the right domain architecture for each application following well-architected design standards. Define and manage SLAs for data products and processes running in production. Create documentation for learnings and knowledge transfer to internal associates. Qualifications 12+ years of overall technology experience that includes at least 5+ years of hands-on software development, data engineering, and systems architecture. 8+ years of experience with Data Lakehouse, Data Warehousing, and Data Analytics tools. 6+ years of experience in SQL optimization and performance tuning on MS SQL Server, Azure SQL or any other popular RDBMS 6+ years of experience in Python/Pyspark/Scala programming on big data platforms like Databricks 4+ years in cloud data engineering experience in Azure or AWS. Fluent with Azure cloud services. Azure Data Engineering certification is a plus. Experience with integration of multi cloud services with on-premises technologies. Experience with data modelling, data warehousing, and building high-volume ETL/ELT pipelines. Experience with data profiling and data quality tools like Great Expectations. Experience building/operating highly available, distributed systems of data extraction, ingestion, and processing of large data sets. Experience with at least one business intelligence tool such as Power BI or Tableau Experience with running and scaling applications on the cloud infrastructure and containerized services like Kubernetes. Experience with version control systems like ADO, Github and CI/CD tools for DevOps automation and deployments. Experience with Azure Data Factory, Azure Databricks and Azure Machine learning tools. Experience with Statistical/ML techniques is a plus. Experience with building solutions in the retail or in the supply chain space is a plus. Understanding of metadata management, data lineage, and data glossaries is a plus. BA/BS in Computer Science, Math, Physics, or other technical fields. Candidate must be flexible to work an alternative work schedule either on tradition work week from Monday to Friday; or Tuesday to Saturday or Sunday to Thursday depending upon product and project coverage requirements of the job. Candidates are expected to be in the office at the assigned location at least 3 days a week and the days at work needs to be coordinated with immediate supervisor Skills, Abilities, Knowledge: Excellent communication skills, both verbal and written, along with the ability to influence and demonstrate confidence in communications with senior level management. Proven track record of leading, mentoring data teams. Strong change manager. Comfortable with change, especially that which arises through company growth. Ability to understand and translate business requirements into data and technical requirements. High degree of organization and ability to manage multiple, competing projects and priorities simultaneously. Positive and flexible attitude to enable adjusting to different needs in an ever-changing environment. Strong leadership, organizational and interpersonal skills; comfortable managing trade-offs. Foster a team culture of accountability, communication, and self-management. Proactively drives impact and engagement while bringing others along. Consistently attain/exceed individual and team goals. Ability to lead others without direct authority in a matrixed environment. Comfortable working in a hybrid environment with teams consisting of contractors as well as FTEs spread across multiple PepsiCo locations. Domain Knowledge in CPG industry with Supply chain/GTM background is preferred.

Design, build, and optimize data pipelines and warehouses. Ensure data quality, collaborate with teams, and document processes. Proficient in Azure, Python, SQL, Spark. Mentor juniors, support pre-sales, and drive scalable data solutions.

ETL (IBM Datastage), Shell Python Scripting Database Expertise DevOps CI/CD Experience Mandatory Competencies Beh - Communication and collaboration ETL - ETL - Data Stage DevOps/Configuration Mgmt - DevOps/Configuration Mgmt - Basic Bash/Shell script writing Data Science and Machine Learning - Data Science and Machine Learning - Python Cloud - Azure - Azure Data Factory (ADF), Azure Databricks, Azure Data Lake Storage, Event Hubs, HDInsight Development Tools and Management - Development Tools and Management - CI/CD Database - Oracle - PL/SQL Packages Database - Sql Server - SQL Packages

Collaborate with business stakeholders and other technical team members to acquire data sources that are most relevant to business needs and goals. Demonstrate deep technical and domain knowledge of relational and non-relation databases, Data Warehouses, Data lakes among other structured and unstructured storage options. Determine solutions that are best suited to develop a pipeline for a particular data source. Develop data flow pipelines to extract, transform, and load data from various data sources in various forms, including custom ETL pipelines that enable model and product development. Write custom scripts to extract data from unstructured/semi-structured sources. Provide clear documentation for delivered solutions and processes, integrating documentation with the appropriate corporate stakeholders. Identify and implement internal process improvements for data management (automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability). Stay current with and adopt new tools and applications to ensure high quality and efficient solutions. Build cross-platform data strategy to aggregate multiple sources and process development datasets. Technical Skills Nice-to-have skills 2+ years of experience with Big Data Management (BDM) for relational and non-relational data (formats like json, xml, Avro, parquet, copybook, etc.) Knowledge of Dev-Ops processes (CI/CD) and infrastructure as code. Knowledge of Master Data Management (MDM) and Data Quality tools. Experience developing REST APIs. Knowledge of key machine learning concepts & MLOPS Qualifications Bachelors degree in computer engineering 3+ years of ETL design, development, and performance tuning using ETL tools such as SSIS/ADF in a multi-dimensional Data Warehousing environment. 3+ years of experience with setting up and operating data pipelines using Python or SQL 3+ years of advanced SQL Programming: PL/SQL, T-SQL 3+ years of strong and extensive hands-on experience in Azure, preferably data heavy / analytics applications leveraging relational and NoSQL databases, Data Warehouse and Big Data 3+ years of experience with Azure Data Factory, Azure Synapse Analytics, Azure Analysis Services, Azure Databricks, Blob Storage, Databricks/Spark, Azure SQL DW/Synapse, and Azure functions. 2+ years of experience in defining and enabling data quality standards for auditing, and monitoring. Strong analytical abilities and a strong intellectual curiosity In-depth knowledge of relational database design, data warehousing and dimensional data modeling concepts Deep understanding of REST and good API design. Strong collaboration and teamwork skills & excellent written and verbal communications skills. Self-starter and motivated with ability to work in a fast-paced development environment. Agile experience highly desirable. Proficiency in the development environment, including IDE, database server, GIT, Continuous Integration, unit-testing tool, and defect management tools.

Interested candidates can share your resume to aweaz.pasha@wisseninfotech.com Power BI developer Total BI experience should be 5+ years Should have knowledge on data bricks Should be able to create the new reports on databricks Should be able to work on reports enhancement and cosmotic implementations Should be strong in troubleshooting of issues Should be strong in writing the queries (SQL) Should have knowledge on Python ADF experience is add-on Regards, Aweaz Pasha Trainee-Talent Acquisition M: 9738144394 E: aweaz.pasha@wisseninfotech.com W: www.wisseninfotech.com

Healthcare experience is Mandatory Position Overview : We are seeking an experienced Data Modeler/Lead with deep expertise in health plan data models and enterprise data warehousing to drive our healthcare analytics and reporting initiatives. The candidate should have hands-on experience with modern data platforms and a strong understanding of healthcare industry data standards. Key Responsibilities : Data Architecture & Modeling : - Design and implement comprehensive data models for health plan operations, including member enrollment, claims processing, provider networks, and medical management - Develop logical and physical data models that support analytical and regulatory reporting requirements (HEDIS, Stars, MLR, risk adjustment) - Create and maintain data lineage documentation and data dictionaries for healthcare datasets - Establish data modeling standards and best practices across the organization Technical Leadership : - Lead data warehousing initiatives using modern platforms like Databricks or traditional ETL tools like Informatica - Architect scalable data solutions that handle large volumes of healthcare transactional data - Collaborate with data engineers to optimize data pipelines and ensure data quality Healthcare Domain Expertise : - Apply deep knowledge of health plan operations, medical coding (ICD-10, CPT, HCPCS), and healthcare data standards (HL7, FHIR, X12 EDI) - Design data models that support analytical, reporting and AI/ML needs - Ensure compliance with healthcare regulations including HIPAA/PHI, and state insurance regulations - Partner with business stakeholders to translate healthcare business requirements into technical data solutions Data Governance & Quality : - Implement data governance frameworks specific to healthcare data privacy and security requirements - Establish data quality monitoring and validation processes for critical health plan metrics - Lead eAorts to standardize healthcare data definitions across multiple systems and data sources Required Qualifications : Technical Skills : - 10+ years of experience in data modeling with at least 4 years focused on healthcare/health plan data - Expert-level proficiency in dimensional modeling, data vault methodology, or other enterprise data modeling approaches - Hands-on experience with Informatica PowerCenter/IICS or Databricks platform for large-scale data processing - Strong SQL skills and experience with Oracle Exadata and cloud data warehouses (Databricks) - Proficiency with data modeling tools (Hackolade, ERwin, or similar) Healthcare Industry Knowledge : - Deep understanding of health plan data structures including claims, eligibility, provider data, and pharmacy data - Experience with healthcare data standards and medical coding systems - Knowledge of regulatory reporting requirements (HEDIS, Medicare Stars, MLR reporting, risk adjustment) - Familiarity with healthcare interoperability standards (HL7 FHIR, X12 EDI) Leadership & Communication : - Proven track record of leading data modeling projects in complex healthcare environments - Strong analytical and problem-solving skills with ability to work with ambiguous requirements - Excellent communication skills with ability to explain technical concepts to business stakeholders - Experience mentoring team members and establishing technical standards Preferred Qualifications : - Experience with Medicare Advantage, Medicaid, or Commercial health plan operations - Cloud platform certifications (AWS, Azure, or GCP) - Experience with real-time data streaming and modern data lake architectures - Knowledge of machine learning applications in healthcare analytics - Previous experience in a lead or architect role within healthcare organization.

Job Location: Chennai, Bangalore, Pune, Mumbai Job Location WFO or Hybrid Hybrid Detailed Job Description:Azure Data Bricks, Azure Data Factory, SQL, ETL testing.They should have Testing / QA experience in these skills- Rate 1150/hr Must Required Skills in KEYWORDS: Strong experience in SQL along with performing Azure Data bricks Testing Shift Timings: 3:30 PM IST to 12:30 AM ISTAzure Data Bricks, Azure Data Factory, SQL, ETL testing.They should have Testing / QA experience in these skills- Rate 1150/hr

We are currently hiring for Senior Data Engineers, please find the below gist of the role. Interested candidates can share their profile to archana@skoruz.com Location: Bangalore Min 6+years of experience in Data Engineer Expertise in Microsoft Fabric and its components (e.g., Synapse, Data Factory, Azure Data Lake, Power BI). Strong proficiency in SQL, Python, and Spark, DataBricks or Snowflake. Experience with cloud platforms, particularly Microsoft Azure. Solid understanding of data modeling, data warehousing, and ETL/ELT best practices. Excellent problem-solving, communication, and project management skills. Preferred: Familiarity with other cloud platforms (e.g., AWS, GCP). Experience with machine learning pipelines or integrating AI into data workflows. Certifications in Microsoft Azure or related technologies.

As an experienced Data Engineer with 8+ years of industry experience, you will be responsible for developing robust, scalable Python-based applications that meet the company's requirements. Your key responsibilities will include integrating and implementing Generative AI models into business applications, designing, building, and maintaining data pipelines and data engineering solutions on Azure, and collaborating closely with cross-functional teams to define, design, and deploy innovative AI and data solutions. You will be required to build, test, and optimize AI pipelines, ensuring seamless integration with Azure-based data systems. Continuous research and evaluation of new AI and Azure data technologies to enhance system capabilities will be essential. Additionally, you will participate actively in code reviews, troubleshooting, debugging, and documentation to maintain high standards of code quality, performance, security, and reliability. To excel in this role, you must possess advanced proficiency in Python programming, including knowledge of libraries and frameworks like Django, Flask, and FastAPI. Experience in Generative AI technologies such as GPT models, LangChain, and Hugging Face will be beneficial. Solid expertise in Azure Data Engineering tools like Azure Data Factory, Azure Databricks, Azure Synapse Analytics, and Azure Data Lake Storage is required. Familiarity with AI/ML libraries like TensorFlow, PyTorch, or OpenAI API, RESTful APIs, microservices architecture, and web application development is essential. You should also have a strong understanding of databases (SQL, NoSQL), ETL processes, containerization, and orchestration technologies like Docker and Kubernetes. Strong problem-solving, analytical, and debugging skills are a must-have for this role. Preferred qualifications include a Bachelor's or Master's degree in computer science, engineering, or related fields, prior experience developing AI-enabled products or implementing AI into applications, and Azure certifications (AZ-204, DP-203, AI-102) or equivalent. Exposure to DevOps practices and CI/CD pipelines, especially in Azure DevOps, will be an added advantage. In addition to technical skills, soft skills such as strong communication, teamwork, ability to work independently, and a passion for continuous learning and professional growth are valued. This full-time position is located in Gurgaon, Noida, Pune, Bengaluru, or Kochi. Join us at Infogain, a human-centered digital platform and software engineering company based in Silicon Valley, where we engineer business outcomes for Fortune 500 companies and digital natives across various industries. We accelerate experience-led transformation in the delivery of digital platforms using cutting-edge technologies such as cloud, microservices, automation, IoT, and artificial intelligence. Infogain is a Microsoft Gold Partner and Azure Expert Managed Services Provider, with global offices and delivery centers in multiple locations worldwide.,

Dear Candidate, We have an urgent opening with one of Multinational Company for Bengaluru location. Interested candidate can share the resume on Deepaksharma@thehrsolutions.in OR WhatsApp on 8882505093 Experience : 5.5+ Years Profile : Mlops Notice period : Only immediate joiners OR Already serving. Please find the below job description : MLOPS Senior Engineer - Azure ML + Azure Databricks at BLR Location 5.5 years of experience in Al Domain & 3+ years in MLOPS (preferably in a large-scale enterprise) Mandatory Skill Experience in developing MLOps framework cutting ML lifecycle: model development, training, evaluation, deployment, monitoring including Model Governance Expert in Azure Databricks, Azure ML, Unity Catalog Hands-on experience with Azure DevOps, MLOPS CI/CD Pipelines, Python, Git, Docker Experience in developing standards and practices for MLOPS life cycle Nice to Have Skill Strong understanding of data privacy, compliance, and responsible Al Azure Data Factory (ADF)

Key Responsibilities: Requirement gathering and analysis Experience with different databases like Synapse, SQL DB, Snowflake etc. Design and implement data pipelines using Azure Data Factory, Databricks, Synapse Create and manage Azure SQL Data Warehouses and Azure Cosmos DB databases Extract, transform, and load (ETL) data from various sources into Azure Data Lake Storage Implement data security and governance measures Monitor and optimize data pipelines for performance and efficiency Troubleshoot and resolve data engineering issues Provide optimized solution for any problem related to data engineering Ability to work with a variety of sources like Relational DB, API, File System, Realtime streams, CDC etc. Strong knowledge on Databricks, Delta tables Required Skills: 48 years of experience in Data Engineering or related roles. Hands-on experience in Azure Databricks , ADF , or Synapse Analytics Proficiency in Python for data processing and scripting. Strong command over SQL writing complex queries, performance tuning, etc. Experience working with Azure Data Lake Storage and Data Warehouse concepts (e.g., dimensional modeling, star/snowflake schemas). Understanding CI/CD practices in a data engineering context. Excellent problem-solving and communication skills. Good to Have: Hands on experience in Microsoft Fabric, Logic Apps, Azure OpenAI basics Experienced in Delta Lake , Power BI , or Azure DevOps . Knowledge of Spark , Scala , or other distributed processing frameworks. Exposure to BI tools like Power BI , Tableau , or Looker . Familiarity with data security and compliance in the cloud. Experience in leading a development team.

You are a highly skilled and strategic Data Architect with deep expertise in the Azure Data ecosystem. Your role will involve defining and driving the overall Azure-based data architecture strategy aligned with enterprise goals. You will architect and implement scalable data pipelines, data lakes, and data warehouses using Azure Data Lake, ADF, and Azure SQL/Synapse. Providing technical leadership on Azure Databricks for large-scale data processing and advanced analytics use cases is a crucial aspect of your responsibilities. Integrating AI/ML models into data pipelines and supporting the end-to-end ML lifecycle including training, deployment, and monitoring will be part of your day-to-day tasks. Collaboration with cross-functional teams such as data scientists, DevOps engineers, and business analysts is essential. You will evaluate and recommend tools, platforms, and design patterns for data and ML infrastructure while mentoring data engineers and junior architects on best practices and architectural standards. Your role will require a strong background in data modeling, ETL/ELT frameworks, and data warehousing concepts. Proficiency in SQL, Python, PySpark, and a solid understanding of AI/ML workflows and tools are necessary. Exposure to Azure DevOps and excellent communication and stakeholder management skills are also key requirements. As a Data Architect at Lexmark, you will play a vital role in designing and overseeing robust, scalable, and secure data architectures to support advanced analytics and machine learning workloads. If you are an innovator looking to make your mark with a global technology leader, apply now to join our team in Kolkata, India.,

As a Data Engineer, you will be responsible for designing and building efficient data pipelines using Azure Databricks (PySpark). You will implement business logic for data transformation and enrichment at scale, as well as manage and optimize Delta Lake storage solutions. Additionally, you will develop REST APIs using FastAPI to expose processed data and deploy them on Azure Functions for scalable and serverless data access. You will play a key role in data orchestration by developing and managing Airflow DAGs to orchestrate ETL processes. This includes ingesting and processing data from various internal and external sources on a scheduled basis. Database management will also be part of your responsibilities, involving handling data storage and access using PostgreSQL and MongoDB, and writing optimized SQL queries to support downstream applications and analytics. Collaboration is essential in this role, as you will work cross-functionally with teams to deliver reliable, high-performance data solutions. It is important to follow best practices in code quality, version control, and documentation to ensure the success of data projects. To excel in this position, you must have at least 5 years of hands-on experience as a Data Engineer and strong expertise in Azure Cloud services. Proficiency in Azure Databricks, PySpark, and Delta Lake is required, along with solid experience in Python and FastAPI for API development. Experience with Azure Functions for serverless API deployments, managing ETL pipelines using Apache Airflow, and working with PostgreSQL and MongoDB is also necessary. Strong SQL skills and experience handling large datasets are essential. Preferred qualifications include familiarity with data ingestion from APIs or third-party data providers, experience optimizing data pipelines for performance and scalability, and a working knowledge of Azure services like Azure Data Lake and Azure Storage.,

As a Senior Software Engineer (SSE) specializing in Kafka, Python, and Azure Databricks, you will play a crucial role in leading and contributing to healthcare data engineering projects. Your primary responsibility will involve building scalable real-time data pipelines and processing large healthcare datasets within a secure and compliant cloud environment. Your key responsibilities will include designing and developing scalable real-time data streaming solutions using Apache Kafka and Python. You will also be tasked with architecting and implementing ETL/ELT pipelines utilizing Azure Databricks for both structured and unstructured healthcare data. Additionally, you will optimize and maintain Kafka applications, Python scripts, and Databricks workflows to ensure high performance and reliability. Ensuring data integrity, security, and compliance with healthcare standards such as HIPAA and HITRUST will be a critical aspect of your role. Collaboration with data scientists, analysts, and business stakeholders to gather requirements and translate them into robust data solutions is also expected. You will be responsible for mentoring junior engineers, conducting code reviews, and advocating for engineering best practices. Remaining updated with the latest technologies in cloud computing, big data, and healthcare data standards will be essential. Contribution to the development of CI/CD pipelines and containerized environments, including Docker and Kubernetes, is part of the role's expectations. To excel in this position, you should possess at least 4 years of hands-on experience in data engineering roles. Proficiency in Kafka, Python, Azure Databricks, SQL, NoSQL databases, and data modeling for large-scale systems is required. Exposure to containerization tools like Docker, orchestration using Kubernetes, and CI/CD pipelines for data applications is advantageous. Prior experience with healthcare datasets such as EHR, HL7, FHIR, and claims data is highly desirable. In addition to technical skills, you should have excellent problem-solving abilities, a proactive mindset, and strong communication and interpersonal skills to collaborate effectively in cross-functional teams.,

Lead : 6-8 years Focus on production cost for the techniques and features • Mentoring the team on benchmarking costs, performance KPI’s • Guarding the focus on the team towards objectives •Advanced proficiency in Python and/or Scala for data engineering tasks. •Proficiency in PySpark and Scala Spark for distributed data processing, with hands-on experience in Azure Databricks. •Expertise in Azure Databricks for data engineering, including Delta Lake, MLflow, and cluster management. •Familiarity with cloud platforms (e.g., AWS, Azure, GCP) and their big data and data warehousing services (e.g., Azure Data Factory, AWS Redshift). •Expertise in data warehousing platforms such as Snowflake, Azure Synapse Analytics, or Redshift, including schema design, ETL/ELT processes, and query optimization. •Experience with Hadoop ecosystem (HDFS, Hive, HBase, etc.), Apache Airflow for workflow orchestration and scheduling. •Advanced knowledge of SQL for data warehousing and analytics, with experience in NoSQL databases (e.g., MongoDB) as a plus. •Experience with version control systems (e.g., Git) and CI/CD pipelines. •Familiarity with Java or other programming languages is a plus.

Company Overview Incedo is a US-based consulting, data science and technology services firm with over 3000 people helping clients from our six offices across US, Mexico and India. We help our clients achieve competitive advantage through end-to-end digital transformation. Our uniqueness lies in bringing together strong engineering, data science, and design capabilities coupled with deep domain understanding. We combine services and products to maximize business impact for our clients in telecom, Banking, Wealth Management, product engineering and life science & healthcare industries. Working at Incedo will provide you an opportunity to work with industry leading client organizations, deep technology and domain experts, and global teams. Incedo University, our learning platform, provides ample learning opportunities starting with a structured onboarding program and carrying throughout various stages of your career. A variety of fun activities is also an integral part of our friendly work environment. Our flexible career paths allow you to grow into a program manager, a technical architect or a domain expert based on your skills and interests. Our Mission is to enable our clients to maximize business impact from technology by Harnessing the transformational impact of emerging technologies Bridging the gap between business and technology Role Description Collaborate with business stakeholders and other technical team members to acquire data sources that are most relevant to business needs and goals. Demonstrate deep technical and domain knowledge of relational and non-relation databases, Data Warehouses, Data lakes among other structured and unstructured storage options. Determine solutions that are best suited to develop a pipeline for a particular data source. Develop data flow pipelines to extract, transform, and load data from various data sources in various forms, including custom ETL pipelines that enable model and product development. Write custom scripts to extract data from unstructured/semi-structured sources. Provide clear documentation for delivered solutions and processes, integrating documentation with the appropriate corporate stakeholders. Identify and implement internal process improvements for data management (automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability). Stay current with and adopt new tools and applications to ensure high quality and efficient solutions. Build cross-platform data strategy to aggregate multiple sources and process development datasets. Technical Skills Nice-to-have skills 2+ years of experience with Big Data Management (BDM) for relational and non-relational data (formats like json, xml, Avro, parquet, copybook, etc.) Knowledge of Dev-Ops processes (CI/CD) and infrastructure as code. Knowledge of Master Data Management (MDM) and Data Quality tools. Experience developing REST APIs. Knowledge of key machine learning concepts & MLOPS Qualifications Bachelors degree in computer engineering 3+ years of ETL design, development, and performance tuning using ETL tools such as SSIS/ADF in a multi-dimensional Data Warehousing environment. 3+ years of experience with setting up and operating data pipelines using Python or SQL 3+ years of advanced SQL Programming: PL/SQL, T-SQL 3+ years of strong and extensive hands-on experience in Azure, preferably data heavy / analytics applications leveraging relational and NoSQL databases, Data Warehouse and Big Data 3+ years of experience with Azure Data Factory, Azure Synapse Analytics, Azure Analysis Services, Azure Databricks, Blob Storage, Databricks/Spark, Azure SQL DW/Synapse, and Azure functions. 2+ years of experience in defining and enabling data quality standards for auditing, and monitoring. Strong analytical abilities and a strong intellectual curiosity In-depth knowledge of relational database design, data warehousing and dimensional data modeling concepts Deep understanding of REST and good API design. Strong collaboration and teamwork skills & excellent written and verbal communications skills. Self-starter and motivated with ability to work in a fast-paced development environment. Agile experience highly desirable. Proficiency in the development environment, including IDE, database server, GIT, Continuous Integration, unit-testing tool, and defect management tools.

Role- Azure Devops EXP- 2+yrs Location- Pan INDIA Job description Hiring for Azure developer with experience range 2 to 9 years Mandatory Skills: Azure, ADF, ADB, Azure synapse Education: BE/B.Tech/BCA/B.SC/MCA/M.Tech/MSc./MS Location: Pan India Responsibilities A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to actively aid the consulting team in different phases of the project including problem definition, effort estimation, diagnosis, solution generation and design and deployment You will explore the alternatives to the recommended solutions based on research that includes literature surveys, information available in public domains, vendor evaluation information, etc. and build POCs You will create requirement specifications from the business needs, define the to-be-processes and detailed functional designs based on requirements. You will support configuring solution requirements on the products; understand if any issues, diagnose the root-cause of such issues, seek clarifications, and then identify and shortlist solution alternatives You will also contribute to unit-level and organizational initiatives with an objective of providing high quality value adding solutions to customers. If you think you fit right in to help our clients navigate their next in their digital transformation journey, this is the place for you!

Dear Candidate, Hope you are doing well. Greeting from NAM Info INC. NAM Info Inc. is a technology-forward talent management organization dedicated to bridging the gap between industry leaders and exceptional human resources. They pride themselves on delivering quality candidates, deep industry coverage, and knowledge-based training for consultants. Their commitment to long-term partnerships, rooted in ethical practices and trust, positions them as a preferred partner for many industries. Learn more about their vision, achievements, and services on their website at www.nam-it.com. We have an open position for Data Engineer role with our company for Bangalore, Pune and Mumbai location. Job Description Position: Sr / Lead Data Engineer Location: Bangalore, Pune and Mumbai Experience: 5 + years Required Skills: Azure, Data warehouse, Python, Spark, PySpark, Snowflake / Databricks, Any RDBMS, Any ETL Tool, SQL, Unix Scripting, GitHub Strong experience in Azure / AWS / GCP Permanent with NAM Info Pvt Ltd Work Location: Bangalore, Pune and Mumbai Working time: 12 PM to 9 PM or 2 PM to 11 PM 5 Days work from office, Monday to Friday L1 interview virtual, L2 face to face at Banashankari office (for Bangalore candidate) Notice period immediate to 15 days If you are fine with the above job details then please share your resume to ananya.das@nam-it.com Regards, Recruitment Team NAM Info INC

Develop and execute test plans and cases to ensure software quality, identifying and reporting defects. Collaborate with developers to resolve issues, participate in code reviews, and maintain test documentation. Contribute to improving the QA process by applying testing best practices and utilizing bug tracking systems within the SDLC. Key Responsibilities Develop and execute test cases and test plans. Identify and report software defects. Perform functional, regression, and performance testing. Collaborate with developers to resolve issues. Participate in code reviews and provide feedback on testability. Document test results and maintain test documentation. Learn and apply software testing best practices. Work with bug tracking systems. Understand software development lifecycle (SDLC). Assist in creating and maintaining automated test scripts. Familiarity with testing tools and frameworks. Ability to analyze and interpret test results. Basic understanding of different testing methodologies. Contribute to improving the QA process. Follow project testing standards. Qualifications Extensive experience in ETL, data warehousing, and BI reporting testing. Proficiency in SQL, Python for automation, and Azure Data Bricks. Strong understanding of relational databases and XML. Experience with test automation, Agile/Waterfall methodologies, and Atlassian tools. Excellent communication and problem-solving skills

Key Responsibilities : Oversee the entire data infrastructure to ensure scalability, operation efficiency and resiliency. - Mentor junior data engineers within the organization. - Design, develop, and maintain data pipelines and ETL processes using Microsoft Azure services (e.g., Azure Data Factory, Azure Synapse, Azure Databricks, Azure Fabric). - Utilize Azure data storage accounts for organizing and maintaining data pipeline outputs. (e.g., Azure Data Lake Storage Gen 2 & Azure Blob storage). - Collaborate with data scientists, data analysts, data architects and other stakeholders to understand data requirements and deliver high-quality data solutions. - Optimize data pipelines in the Azure environment for performance, scalability, and reliability. - Ensure data quality and integrity through data validation techniques and frameworks. - Develop and maintain documentation for data processes, configurations, and best practices. - Monitor and troubleshoot data pipeline issues to ensure timely resolution. - Stay current with industry trends and emerging technologies to ensure our data solutions remain cutting-edge. - Manage the CI/CD process for deploying and maintaining data solutions.

JOB DESCRIPTION: We at HCL Tech are looking for candidates for the job role of "Test Lead". Overview of the role- As a Senior Test Lead, you will spearhead cloud and data warehouse automation testing initiatives, ensuring the robustness and reliability of software solutions. This role is pivotal in maintaining the quality standards of HCL Tech cloud-based applications, directly impacting the company objectives to deliver top-notch technology solutions. Job Responsibilities: Develop and execute comprehensive test strategies for cloud applications. Implement and manage automated testing frameworks to enhance testing efficiency. Lead and mentor the testing team, ensuring timely and high-quality deliverables. Collaborate with cross-functional teams to understand testing requirements and set priorities. Conduct ETL testing to ensure data accuracy and integrity in automation projects. Create test cases, scenarios, and scripts for validating data transformation and loading processes. Monitor test outcomes, identify defects, and coordinate with development teams for prompt resolution. Stay abreast of industry best practices and emerging technologies in cloud and data automation testing. Skill Requirements: Extensive experience in cloud automation testing, ETL, and data warehouse automation. In-depth knowledge of testing methodologies and tools applicable to cloud environments. Proficiency in automation tools such as Selenium, JMeter, or similar. Strong understanding of ETL processes and data validation techniques. Excellent analytical skills to effectively identify and address testing gaps. Proven leadership abilities with strong communication skills for stakeholder engagement. Certifications in Cloud Testing, Automation Testing, or related fields are advantageous. Other requirements: Familiarity with data reconciliation techniques. Experience with agile methodologies is a plus. Job Type : Full-time and permanent Job Location : PAN India (Noida, Bangalore, Hyderabad, Chennai and Pune) Notice period : immediate to 30 days Work mode: On-site role with 5 days of working. Interested candidates can directly share their updated resumes with us at aditi-agarwal@hcltech.com OR can apply here.

Job Title: Senior Engineer Work Location: PUNE, MH/ HYDERABAD, TA/ KOLKATA, WB/ BANGALORE, KA/ THANE, MH/ NOIDA, UP Skills Required: Databricks, Azure Data Factory Experience Range in Required Skills: 4-6 Years Job Description: Very strong hands-on experience in using Microsoft Azure services like Azure Data Factory, Data brick services (like processing streaming data using Spark clusters), usage of Blob containers, ESB, Event Grid, Azure SQL server, Cosmos DB, Azure functions, Analytics (like Power BI) is a mandatory requirement