196 Delta Lake Jobs - Page 6

Work Mode: Hybrid (3 days WFO) Locations: Bangalore, Noida, Pune, Mumbai, Hyderabad (Candidates must be in Accion cities to collect assets and attend in-person meetings as required). Key Requirements: Technical Skills: Databricks Expertise: 5+ years of hands-on experience in data engineering/ETL using Databricks on AWS/Azure cloud infrastructure. Proficiency in Delta Lake, Unity Catalog, Delta Sharing, Delta Live Tables (DLT), MLflow, and Databricks SQL. Experience with Databricks CI/CD tools (e.g., BitBucket, GitHub Actions, Databricks CLI). Data Warehousing & Engineering: Strong understanding of data warehousing concepts (Dimensional, SCD2, Data Vault, OBT, etc.). Proven ability to implement highly performant data ingestion pipelines from multiple sources. Experience integrating end-to-end Databricks pipelines to ensure data quality and consistency. Programming: Strong proficiency in Python and SQL. Basic working knowledge of API or stream-based data extraction processes (e.g., Salesforce API, Bulk API). Cloud Technologies: Preferred experience with AWS services (e.g., S3, Athena, Glue, Lambda). Power BI: 3+ years of experience in Power BI and data warehousing for root cause analysis and business improvement opportunities. Additional Skills: Working knowledge of Data Management principles (quality, governance, security, privacy, lifecycle management, cataloging). Nice to have: Databricks certifications and AWS Solution Architect certification. Nice to have: Experience with building data pipelines from business applications like Salesforce, Marketo, NetSuite, Workday, etc. Responsibilities: Develop, implement, and maintain highly efficient ETL pipelines on Databricks. Perform root cause analysis and identify opportunities for data-driven business improvements. Ensure quality, consistency, and governance of all data pipelines and repositories. Work in an Agile/DevOps environment to deliver iterative solutions. Collaborate with cross-functional teams to meet business requirements. Stay updated on the latest Databricks and AWS features, tools, and best practices. Work Schedule: Regular: 11:00 AM to 8:00 PM. Flexibility is required for project-based overlap. Interested candidates should share their resumes with the following details: Current CTC Expected CTC Preferred Location: Bangalore, Noida, Pune, Mumbai, Hyderabad Notice Period Contact Information:

Job Purpose: Assist with the development and maintenance of software solutions for new and existing projects. Deep technical skills are required, along with an ability to understand how all pieces fit together and are validated in a complex, distributed system. Duties and Responsibilities : 1. Develop software solutions by studying information needs; conferring with users; studying systems flow, data usage, and work processes; investigating problem areas; following the software development lifecycle 2. Determine operational feasibility by evaluating analysis, problem definition, requirements, solution development, and proposed solutions 3. Improve operations by conducting systems analysis; recommending changes in guidelines and procedures 4. Update job knowledge by studying state-of-the-art development tools, programming techniques, and computing equipment; participating in educational opportunities; reading professional publications; maintaining personal networks; participating in professional organizations 5. Actively participates hand on in product development and roadmap definition 6. Develop prototypes to prove the solutions business value to the Product Requirements 7. Represents technical viewpoint for various technologies during strategic planning 8. Respond to stakeholder requirements from a requirements & technology standpoint and discuss concepts, solutions, technical feasibility & risks with them 9. Support Developers by providing advice, coaching and educational opportunities 10. Participate in knowledge-sharing code reviews 11. Adhere to the Code of Conduct and be familiar with all compliance policies and procedures. Experience Required : 1. Seven years plus experience in software development 2. Healthcare experience / RCM applications knowledge / project experience preferred 3. Experience working with global team and working in a team-oriented, collaborative environment 4 . Experience with agile development 5. Immediate Joiners / Short Notice Period candidates Preferred. Required skills and knowledge: 1. Advanced coding skills in C#, .NET 2. Strong working knowledge in SQL, REST, Angular / React / Node 3. Experience in Cloud native development using Azure / AWS / GCP, Containerization, GenAI, Agentic AI ML Algorithm 4. Expert in git, CI/CD, Terraform, Containerization, ADFS, MS Entra ID, Agile methodologies 5. Knowledge in application logging, security, authentications, authorizations 6. Object-Oriented Programming and design principles. Preferred skills and knowledge : 1. Knowledge in Modern data technologies e.g. Delta Lake, Azure Data Lake, Blob storage, NoSQL DB, Databricks, PySpark / Scala/ Spark SQL 2. Ability to solve problems quickly and completely 3. Ability to multi-task and stay organized in a dynamic work environment 4. Possesses a positive attitude and ability to think outside the box 5. Understands and anticipates the possible failures in a growing system and knows how to prevent them 6. Utilizes source control with multiple concurrent branches 7. Must possess hands on technical skills, along with an ability to work independently or under limited supervision & guidance 8. Ability to write routine reports and correspondence 9. Ability to communicate effectively verbally and in writing

Job Title : Data Scientist Location State : Tamil Nadu,Telangana Location City : Hyderabad, Chennai Experience Required : 4 to 6 Year(s) CTC Range : 7 to 11 LPA Shift: Day Shift Work Mode: Onsite Position Type: C2H Openings: 2 Company Name: VARITE INDIA PRIVATE LIMITED About The Client: Client is an Indian multinational technology company specializing in information technology services and consulting. Headquartered in Mumbai, it is a part of the Tata Group and operates in 150 locations across 46 countries. About The Job: Requirements: 5+ years in predictive analytics, with expertise in regression, classification, time-series modeling. Hands-on experience with Databricks Runtime for ML, Spark SQL, and PySpark. Familiarity with MLflow, Feature Store, and Unity Catalog for governance. Industry experience in Life Insurance or P&C. Skills: Python, PySpark , MLflow, Databricks AutoML. Predictive MoClienting (Classification , Clustering , Regression, timeseries and NLP). Cloud platform (Azure/AWS) , Delta Lake, Unity Catalog. Certifications; Databricks Certified ML Practitioner (Optional) Essential Job Functions: Design and deploy predictive models (e.g., forecasting, churn analysis, fraud detection) using Python/SQL, Spark MLlib, and Databricks ML. Build end-to-end ML pipelines (data ingestion, feature engineering, model training, deployment) on Databricks Lakehouse. Optimize model performance via hyperparameter tuning, AutoML, and MLflow tracking. Collaborate with engineering teams to operationalize models (batch/real-time) using Databricks Jobs or REST APIs. Implement Delta Lake for scalable, ACID-compliant data workflows. Enable CI/CD for ML pipelines using Databricks Repos and GitHub Actions. Troubleshoot issues in Spark Jobs and Databricks Environment. Qualifications: Skill Required: Data Science, Python for Data Science Experience Range in Required Skills: 4-6 Years How to Apply: Interested candidates are invited to submit their resume using the apply online button on this job post. About VARITE: VARITE is a global staffing and IT consulting company providing technical consulting and team augmentation services to Fortune 500 Companies in USA, UK, CANADA and INDIA. VARITE is currently a primary and direct vendor to the leading corporations in the verticals of Networking, Cloud Infrastructure, Hardware and Software, Digital Marketing and Media Solutions, Clinical Diagnostics, Utilities, Gaming and Entertainment, and Financial Services. Equal Opportunity Employer: VARITE is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. We do not discriminate on the basis of race, color, religion, sex, sexual orientation, gender identity or expression, national origin, age, marital status, veteran status, or disability status. Unlock Rewards: Refer Candidates and Earn. If you're not available or interested in this opportunity, please pass this along to anyone in your network who might be a good fit and interested in our open positions. VARITE offers a Candidate Referral program, where you'll receive a one-time referral bonus based on the following scale if the referred candidate completes a three-month assignment with VARITE. Exp Req - Referral Bonus 0 - 2 Yrs. - INR 5,000 2 - 6 Yrs. - INR 7,500 6 + Yrs. - INR 10,000

Technical Experience 5-7 years of hands-on experience in data pipeline development and ETL processes 3+ years of deep AWS experience , specifically with Kinesis, Glue, Lambda, S3, and Step Functions Strong proficiency in NodeJS/JavaScript and Java for serverless and containerized applications Production experience with Apache Spark, Apache Flink, or similar big data processing frameworks Data Engineering Expertise Proven experience with real-time streaming architectures and event-driven systems Hands-on experience with Parquet, Avro, Delta Lake, and columnar storage optimization Experience implementing data quality frameworks such as Great Expectations or similar tools Knowledge of star schema modeling, slowly changing dimensions, and data warehouse design patterns Experience with medallion architecture or similar progressive data refinement strategies AWS Skills Experience with Amazon EMR, Amazon MSK (Kafka), or Amazon Kinesis Analytics Knowledge of Apache Airflow for workflow orchestration Experience with DynamoDB, ElastiCache, and Neptune for specialized data storage Familiarity with machine learning pipelines and Amazon SageMaker integration

Role: Azure Databricks Data Engineer Location: Hyderabad, Bangalore, Chennai, Mumbai, Pune, Kolkata, Gurgaon Experience: 5-8 years Work Mode: Hybrid Job Summary: We are seeking an experienced Azure Databricks Data Engineer who will play a key role in designing, building, and maintaining scalable data solutions using Databricks on Azure . This role demands expertise in cloud services, big data engineering, and data architecture to drive business insights through advanced analytics solutions. Key Responsibilities: Design and implement scalable, high-performance data solutions using Databricks on Azure to support business analytics and reporting needs. Collaborate with cross-functional teams (data science, BI, product, infrastructure) to integrate big data solutions with the existing IT ecosystem. Develop, optimize, and manage ETL/ELT pipelines , data lakes, and data warehouses ensuring efficiency and robustness. Perform data modeling , validation, and ensure data accuracy, integrity, and reliability across platforms. Provide subject matter expertise on data storage solutions and manage large-scale data ingestion and transformation workflows. Implement CI/CD practices using tools such as Azure DevOps, Jenkins, TFS, PowerShell , and automate deployment pipelines. Ensure adherence to data security, privacy policies , and compliance requirements. Mentor and guide junior engineers, lead project segments, and contribute to architectural reviews and standards. Must-Have Skills: Strong hands-on experience with Azure Databricks and associated Azure cloud services (e.g., Data Lake, Synapse , Blob Storage ) Proficiency in Spark, PySpark , and SQL for data transformation and analytics Solid understanding of big data architecture and distributed computing principles Experience in CI/CD automation using Azure DevOps, Jenkins, TFS, or equivalent tools Ability to design and manage large-scale data pipelines and complex data workflows Strong communication, collaboration, and problem-solving skills Good to Have: Exposure to machine learning workflows on Databricks Familiarity with Delta Lake, Azure Machine Learning, or Power BI integration Certifications in Azure or Databricks

Hiring Data Engineers with 3+ yrs in Databricks, PySpark, Delta Lake, and AWS (S3, Glue, Redshift, Lambda, EMR). Must have strong SQL/Python, CI/CD, and data pipeline experience. Only Tier-1 company backgrounds are considered.

Job Summary We are looking for a highly skilled Cloud Engineer with a strong background in real-time and batch data ingestion and data processing, azure products-devops, azure cloud. The ideal candidate should have a deep understanding of streaming architectures and performance optimization techniques in cloud environments, preferably in subsurface domain. Key Responsibilities Automation experience essential : Scripting, using PowerShell. ARM Templates, using JSON (PowerShell also acceptable) Azure DevOps with CI/CD, Site Reliability Engineering Must be able to understand the concept of how the applications function. The ability to priorities workload and operate across several initiatives simultaneously Update and maintain the Kappa-Automate database and connectivity with the pi historian and data lake Participate in troubleshooting, performance tuning, and continuous improvement of the Kappa Automate platform Designing and implementing highly configurable Deployment pipelines in Azure Configuring Delta Lake on Azure Databricks Apply performance tuning techniques such as partitioning, caching, and cluster Working on various Azure storage types Work with large volumes of structured and unstructured data, ensuring high availability and performance. Collaborate with cross-functional teams (data scientists, analysts, business users) Qualifications • Bachelors or Master’s degree in Computer Science, Information Technology, or a related field. • 8+ years of experience in data engineering or a related role. • Proven experience with Azure technologies.

Roles and Responsibilities Design, develop, and maintain automated testing frameworks using AWS services such as Glue, Lambda, Step Functions, etc. Develop data pipelines using Delta Lake and ETL processes to extract insights from large datasets. Collaborate with cross-functional teams to identify requirements for test cases and create comprehensive test plans. Ensure high-quality deliverables by executing thorough testing procedures and reporting defects. Desired Candidate Profile 3-7 years of experience in QA Automation with a focus on AWS native stack (Glue, Lambda). Strong understanding of SQL concepts and ability to write complex queries. Experience working with big data technologies like Hadoop/Hive/Pyspark is an added advantage.

Job Description: We are looking for energetic, self-motivated and exceptional Data engineer to work on extraordinary enterprise products based on AI and Big Data engineering leveraging AWS/Databricks tech stack. He/she will work with star team of Architects, Data Scientists/AI Specialists, Data Engineers and Integration. Skills and Qualifications: 5+ years of experience in DWH/ETL Domain; Databricks/AWS tech stack 2+ years of experience in building data pipelines with Databricks/ PySpark /SQL Experience in writing and interpreting SQL queries, designing data models and data standards. Experience in SQL Server databases, Oracle and/or cloud databases. Experience in data warehousing and data mart, Star and Snowflake model. Experience in loading data into database from databases and files. Experience in analyzing and drawing design conclusions from data profiling results. Understanding business process and relationship of systems and applications. Must be comfortable conversing with the end-users. Must have ability to manage multiple projects/clients simultaneously. Excellent analytical, verbal and communication skills. Role and Responsibilities: Work with business stakeholders and build data solutions to address analytical & reporting requirements. Work with application developers and business analysts to implement and optimise Databricks/AWS-based implementations meeting data requirements. Design, develop, and optimize data pipelines using Databricks (Delta Lake, Spark SQL, PySpark), AWS Glue, and Apache Airflow Implement and manage ETL workflows using Databricks notebooks, PySpark and AWS Glue for efficient data transformation Develop/ optimize SQL scripts, queries, views, and stored procedures to enhance data models and improve query performance on managed databases. Conduct root cause analysis and resolve production problems and data issues. Create and maintain up to date documentation of the data model, data flow and field level mappings. Provide support for production problems and daily batch processing. Provide ongoing maintenance and optimization of database schemas, data lake structures (Delta Tables, Parquet), and views to ensure data integrity and performance

Job Summary We are looking for a highly skilled Cloud Engineer with a strong background in real-time and batch data ingestion and data processing, azure products-devops, azure cloud. The ideal candidate should have a deep understanding of streaming architectures and performance optimization techniques in cloud environments, preferably in subsurface domain. Key Responsibilities Automation experience essential : Scripting, using PowerShell. ARM Templates, using JSON (PowerShell also acceptable) Azure DevOps with CI/CD, Site Reliability Engineering Must be able to understand the concept of how the applications function. The ability to priorities workload and operate across several initiatives simultaneously Update and maintain the Kappa-Automate database and connectivity with the pi historian and data lake Participate in troubleshooting, performance tuning, and continuous improvement of the Kappa Automate platform Designing and implementing highly configurable Deployment pipelines in Azure Configuring Delta Lake on Azure Databricks Apply performance tuning techniques such as partitioning, caching, and cluster Working on various Azure storage types Work with large volumes of structured and unstructured data, ensuring high availability and performance. Collaborate with cross-functional teams (data scientists, analysts, business users) Qualifications • Bachelors or Masters degree in Computer Science, Information Technology, or a related field. • 8+ years of experience in data engineering or a related role. • Proven experience with Azure technologies.

ABOUT THE ROLE Role Description: We are seeking a seasoned Principal Architect - Solutions to drive the architecture, development and implementation of data solutions to Amgen functional groups. The ideal candidate able to work in large scale Data Analytic initiatives, engage and work along with Business, Program Management, Data Engineering and Analytic Engineering teams. Be champions of enterprise data analytic strategy, data architecture blueprints and architectural guidelines. As a Principal Architect, you will play a crucial role in designing, building, and optimizing data solutions to Amgen functional groups such as RD, Operations and GCO. Roles Responsibilities: Implement and manage large scale data analytic solutions to Amgen functional groups that align with the Amgen Data strategy Collaborate with Business, Program Management, Data Engineering and Analytic Engineering teams to deliver data solutions Responsible for design, develop, optimize, delivery and support of Data solutions on AWS and Databricks architecture Leverage cloud platforms (AWS preferred) to build scalable and efficient data solutions. Provide expert guidance and mentorship to the team members, fostering a culture of innovation and best practices. Be passionate and hands-on to quickly experiment with new data related technologies Define guidelines, standards, strategies, security policies and change management policies to support the Enterprise Data platform. Collaborate and align with EARB, Cloud Infrastructure, Security and other technology leaders on Enterprise Data Architecture changes Work with different project and application groups to drive growth of the Enterprise Data Platform using effective written/verbal communication skills, and lead demos at different roadmap sessions Overall management of the Enterprise Data Platform on AWS environment to ensure that the service delivery is cost effective and business SLAs around uptime, performance and capacity are met Ensure scalability, reliability, and performance of data platforms by implementing best practices for architecture, cloud resource optimization, and system tuning. Collaboration with RunOps engineers to continuously increase our ability to push changes into production with as little manual overhead and as much speed as possible. Maintain knowledge of market trends and developments in data integration, data management and analytics software/tools Work as part of team in a SAFe Agile/Scrum model Basic Qualifications and Experience: Master s degree with 12 - 15 years of experience in Computer Science, IT or related field OR Bachelor s degree with 14 - 17 years of experience in Computer Science, IT or related field Functional Skills: Must-Have Skills: 8+ years of hands-on experience in Data integrations, Data Management and BI technology stack. Strong experience with one or more Data Management tools such as AWS data lake, Snowflake or Azure Data Fabric Expert-level proficiency with Databricks and experience in optimizing data pipelines and workflows in Databricks environments. Strong experience with Python, PySpark, and SQL for building scalable data workflows and pipelines. Experience with Apache Spark, Delta Lake, and other relevant technologies for large-scale data processing. Familiarity with BI tools including Tableau and PowerBI Demonstrated ability to enhance cost-efficiency, scalability, and performance for data solutions Strong analytical and problem-solving skills to address complex data solutions Good-to-Have Skills: Preferred to have experience in life science or tech or consultative solution architecture roles Experience working with agile development methodologies such as Scaled Agile. Professional Certifications AWS Certified Data Engineer preferred Databricks Certificate preferred Soft Skills: Excellent analytical and troubleshooting skills. Strong verbal and written communication skills Ability to work effectively with global, virtual teams High degree of initiative and self-motivation. Ability to manage multiple priorities successfully. Team-oriented, with a focus on achieving team goals Strong presentation and public speaking skills. EQUAL OPPORTUNITY STATEMENT Amgen is an Equal Opportunity employer and will consider you without regard to your race, color, religion, sex, sexual orientation, gender identity, national origin, protected veteran status, or disability status. We will ensure that individuals with disabilities are provided with reasonable accommodation to participate in the job application or interview process, to perform essential job functions, and to receive other benefits and privileges of employment. Please contact us to request an accommodation. .

Job Title: Data/ML Platform Engineer Location: Gurgaon, Pune, Bangalore, Chennai, Bhopal, Jaipur, Hyderabad (Work from office) Notice Period: ImmediateiSource Services is hiring for one of their client for the position of Data/ML Platform Engineer. As a Data Engineer you will be relied on to independently develop and deliver high-quality features for our new ML Platform, refactor and translate our data products and finish various tasks to a high standard. Youll be part of the Data Foundation Team, which focuses on creating and maintaining the Data Platform for Marktplaats. 5 years of hands-on experience in using Python, Spark,Sql. Experienced in AWS Cloud usage and management. Experience with Databricks (Lakehouse, ML, Unity Catalog, MLflow). Experience using various ML models and frameworks such as XGBoost, Lightgbm, Torch. Experience with orchestrators such as Airflow and Kubeflow. Familiarity with containerization and orchestration technologies (e.g., Docker, Kubernetes). Fundamental understanding of Parquet, Delta Lake and other data file formats. Proficiency on an IaC tool such as Terraform, CDK or CloudFormation. Strong written and verbal English communication skill and proficient in communication with non-technical stakeholderst Location - Gurgaon, Pune, Bangalore, Chennai, Bhopal, Jaipur, Hyderabad (Work from office)

We are looking for energetic, self-motivated and exceptional Data engineer to work on extraordinary enterprise products based on AI and Big Data engineering leveraging AWS/Databricks tech stack. He/she will work with star team of Architects, Data Scientists/AI Specialists, Data Engineers and Integration. Skills and Qualifications: 5+ years of experience in DWH/ETL Domain; Databricks/AWS tech stack 2+ years of experience in building data pipelines with Databricks/ PySpark /SQL Experience in writing and interpreting SQL queries, designing data models and data standards. Experience in SQL Server databases, Oracle and/or cloud databases. Experience in data warehousing and data mart, Star and Snowflake model. Experience in loading data into database from databases and files. Experience in analyzing and drawing design conclusions from data profiling results. Understanding business process and relationship of systems and applications. Must be comfortable conversing with the end-users. Must have ability to manage multiple projects/clients simultaneously. Excellent analytical, verbal and communication skills. Role and Responsibilities: Work with business stakeholders and build data solutions to address analytical & reporting requirements. Work with application developers and business analysts to implement and optimise Databricks/AWS-based implementations meeting data requirements. Design, develop, and optimize data pipelines using Databricks (Delta Lake, Spark SQL, PySpark), AWS Glue, and Apache Airflow Implement and manage ETL workflows using Databricks notebooks, PySpark and AWS Glue for efficient data transformation Develop/ optimize SQL scripts, queries, views, and stored procedures to enhance data models and improve query performance on managed databases. Conduct root cause analysis and resolve production problems and data issues. Create and maintain up to date documentation of the data model, data flow and field level mappings. Provide support for production problems and daily batch processing. Provide ongoing maintenance and optimization of database schemas, data lake structures (Delta Tables, Parquet), and views to ensure data integrity and performance

5 to 7 years of experience in data engineering Architect and maintain scalable, secure, and reliable data platforms and pipelines Design and implement data lake/data warehouse solutions such as Redshift, BigQuery, Snowflake, or Delta Lake Build real-time and batch data pipelines using tools like Apache Airflow, Kafka, Spark, and DBT Ensure data governance, lineage, quality, and observability

Role : Were looking for a skilled Databricks Solution Architect who will lead the design and implementation of data migration strategies and cloud-based data and analytics transformation on the Databricks platform. This role involves collaborating with stakeholders, analyzing data, defining architecture, building data pipelines, ensuring security and performance, and implementing Databricks solutions for machine learning and business intelligence. Key Responsibilities: Define the architecture and roadmap for cloud-based data and analytics transformation on Databricks. Design, implement, and optimize scalable, high-performance data architectures using Databricks. Build and manage data pipelines and workflows within Databricks. Ensure that best practices for security, scalability, and performance are followed. Implement Databricks solutions that enable machine learning, business intelligence, and data science workloads. Oversee the technical aspects of the migration process, from planning through to execution. Create documentation of the architecture, migration processes, and solutions. Provide training and support to teams post-migration to ensure they can leverage Databricks. Preferred candidate profile: Experience: 7+ years of experience in data engineering, cloud architecture, or related fields. 3+ years of hands-on experience with Databricks, including the implementation of data engineering solutions, migration projects, and optimizing workloads. Strong experience with cloud platforms (e.g., AWS, Azure, GCP) and their integration with Databricks. Experience in end-to-end data migration projects involving large-scale data infrastructure. Familiarity with ETL tools, data lakes, and data warehousing solutions. Skills: Expertise in Databricks architecture and best practices for data processing. Strong knowledge of Spark, Delta Lake, DLT, Lakehouse architecture, and other latest Databricks components. Proficiency in Databricks Asset Bundles Expertise in design and development of migration frameworks using Databricks Proficiency in Python, Scala, SQL, or similar languages for data engineering tasks. Familiarity with data governance, security, and compliance in cloud environments. Solid understanding of cloud-native data solutions and services.

Key Responsibilities: Design and develop the migration strategies and processes Collaborate with stakeholders to understand business requirements and technical challenges. Analyze current data and scope for optimization during the migration process. Define the architecture and roadmap for cloud-based data and analytics transformation on Databricks. Design, implement, and optimize scalable, high-performance data architectures using Databricks. Build and manage data pipelines and workflows within Databricks. Ensure that best practices for security, scalability, and performance are followed. Implement Databricks solutions that enable machine learning, business intelligence, and data science workloads. Oversee the technical aspects of the migration process, from planning through to execution. Work closely with engineering and data teams to ensure proper migration of ETL processes, data models, and analytics workloads. Troubleshoot and resolve issues related to migration, data quality, and performance. Create documentation of the architecture, migration processes, and solutions. Provide training and support to teams post-migration to ensure they can leverage Databricks. Experience: 7+ years of experience in data engineering, cloud architecture, or related fields. 3+ years of hands-on experience with Databricks, including the implementation of data engineering solutions, migration projects, and optimizing workloads. Strong experience with cloud platforms (e.g., AWS, Azure, GCP) and their integration with Databricks. Experience in end-to-end data migration projects involving large-scale data infrastructure. Familiarity with ETL tools, data lakes, and data warehousing solutions. Skills: Expertise in Databricks architecture and best practices for data processing. Strong knowledge of Spark, Delta Lake, DLT, Lakehouse architecture, and other latest Databricks components. Proficiency in Databricks Asset Bundles Expertise in design and development of migration frameworks using Databricks Proficiency in Python, Scala, SQL, or similar languages for data engineering tasks. Familiarity with data governance, security, and compliance in cloud environments. Solid understanding of cloud-native data solutions and services.

1. Strategic Leadership: Define and lead the vision for the Databricks Center of Excellence (CoE), positioning Databricks as a strategic data and AI platform for Apexon and its clients. Drive the adoption of Databricks for large-scale data processing, analytics, and AI/ML workloads. 2. Architecture and Implementation: Design and implement data architectures leveraging Databricks, Delta Lake, and Spark for big data processing and analytics. Develop frameworks for ETL pipelines, real-time streaming, and batch processing using Databricks. Optimize the use of Databricks features, such as MLFlow, AutoML, and Delta Lake for enterprise solutions. 3. Performance and Optimization: Lead efforts to optimize Spark jobs, cluster configurations, and data storage for performance and cost efficiency. Ensure scalability of Databricks solutions to handle increasing data volumes and compute requirements. 4. Integration with Cloud and Ecosystem Tools: Integrate Databricks with cloud platforms (AWS, Azure, GCP) and tools such as Snowflake, Tableau, or Power BI. Implement CI/CD pipelines for Databricks workflows and model deployment. 5. Governance and Security: Implement robust data governance, access control, and security policies in Databricks environments. Ensure compliance with industry regulations and best practices for data privacy. 6. Thought Leadership and Team Development: Build and manage a team of Databricks professionals, fostering innovation and technical excellence. Stay updated on Databricks advancements and advocate for their adoption through client presentations, workshops, and internal knowledge-sharing. Technical Competencies: Core Expertise: Proficiency in Spark, Delta Lake, and Databricks Workflows. Programming: Advanced skills in Python, Scala, and SQL for data engineering tasks. Real-Time Processing: Experience with real-time data integration using Kafka, Event Hub, or Databricks Structured Streaming. Cloud Ecosystem: Expertise in deploying Databricks on AWS, Azure, or GCP. Qualifications: Must Have: Bachelor's or Master's degree in Data Engineering, Computer Science, or related field. 10+ years of experience, with 3+ years focused on Databricks-based data solutions. Nice to Have/Preferred: Databricks certifications such as Databricks Certified Data Engineer Professional. Experience with multi-cloud and hybrid Databricks deployments.

Job Title: Senior Data Engineer (4-6 Years Experience) Location: Kotak Life HO Department: Data Science & Analytics Employment Type: Full-Time About the Role: We are seeking a highly skilled Data Engineer with 4-6 years of hands-on experience in designing and developing scalable, reliable, and efficient data solutions. The ideal candidate will have a strong background in cloud platforms (AWS or Azure), experience in building both batch and streaming data pipelines, and familiarity with modern data architectures including event-driven and medallion architectures. Key Responsibilities: .Design, build, and maintain scalable data pipelines (batch and streaming) to process structured and unstructured data from various sources. .Develop and implement solutions based on event-driven architectures using technologies like Kafka, Event Hubs, or Kinesis. .Architect and manage data workflows based on the Medallion architecture (Bronze, Silver, Gold layers). .Work with cloud platforms (AWS or Azure) to manage data infrastructure and storage, compute, and orchestration services. .Leverage cloud-native or open-source tools for data transformation, orchestration, monitoring, and quality checks. .Collaborate with data scientists, analysts, and product manager to deliver high-quality data solutions. .Ensure best practices in data governance, security, lineage, and observability. Required Skills & Qualifications: .4-6 years of professional experience in data engineering or related roles. .Strong experience in cloud platforms: AWS (e.g., S3, Glue, Lambda, Redshift) or Azure (e.g., Data Lake, Synapse, Data Factory, Functions). .Proven expertise in building batch and streaming pipelines using tools like Spark, Flink, Kafka, Kinesis, or similar. .Practical knowledge of event-driven architectures and experience with message/event brokers. .Hands-on experience implementing Medallion architecture or similar layered data architectures. .Familiarity with data orchestration tools (e.g., Airflow, Azure Data Factory, AWS Step Functions). .Proficiency in SQL, Python, or Scala for data processing and pipeline development. .Exposure to open-source tools in the modern data stack (e.g., dbt, Delta Lake, Apache Hudi, Great Expectations). Preferred Qualifications: .Experience with containerization and CI/CD for data workflows (Docker, GitHub Actions, etc.). .Knowledge of data quality frameworks and observability tooling. .Experience with Delta Lake or Lakehouse implementations. .Strong problem-solving skills and ability to work in fast-paced environments. What We Offer:

We are looking for a skilled Data Engineer with strong expertise in Python, PySpark, SQL, and AWS to join our data engineering team. The ideal candidate will be responsible for building scalable data pipelines, transforming large datasets, and enabling data-driven decision-making across the organization. Role & responsibilities Data Pipeline Development: Design, build, and maintain scalable data pipelines for ingesting, processing, and transforming large datasets from diverse sources into usable formats. Performance Optimization: Optimize data processing and storage systems for cost efficiency and high performance, including managing compute resources and cluster configurations. Automation and Workflow Management: Automate data workflows using tools like Airflow, Databricks APIs, and other orchestration technologies to streamline data ingestion, processing, and reporting tasks. Data Quality and Validation: Implement data quality checks, validation rules, and transformation logic to ensure the accuracy, consistency, and reliability of data. Cloud Platform Management: Manage and optimize cloud infrastructure (AWS, Databricks) for data storage, processing, and compute resources, ensuring seamless data operations. Preferred candidate profile Strong proficiency in Python for scripting and data manipulation. Hands-on experience with PySpark for distributed data processing. Proficient in writing complex SQL queries for large-scale data extraction and transformation. Solid understanding and experience with AWS cloud ecosystem (especially S3, Glue, EMR, Lambda). Knowledge of data warehousing, data lakes, and ETL/ELT processes. Familiarity with version control tools like Git and workflow orchestration tools (e.g., Airflow) is a plus.

Responsibilities: Responsible for pipeline maintenance, transformations, and platform reliability. Skills: Scheduling Monitoring: Azure Data Factory (ADF) triggers, Azure Monitor, Log Analytics (KQL) Ingestion Validation: ADF, Event Hub, APIs, PySpark, schema drift handling CI/CD: Azure DevOps YAML pipelines, deployment for ADF/Synapse/Databricks Transformation Performance: PySpark, Delta Lake optimization (partitioning, caching) Security: Azure RBAC, Key Vault, NSGs, IR config Stability RCA: RCA documentation, disaster recovery playbooks, proactive alerting

Job Summary: We are seeking a detail-oriented and results-driven Project Manager to lead projects within our Data Platform organization, with a strong focus on Databricks implementations . The ideal candidate will have experience managing end-to-end delivery of cloud-based data platforms and collaborating cross-functionally across data engineering, analytics, DevOps, and business stakeholders. Key Responsibilities: Lead and manage full lifecycle projects related to data platform initiatives, especially Databricks-based solutions across AWS or Azure. Develop and maintain project plans, schedules, budgets, and resource forecasts using tools like Jira, MS Project, or similar. Coordinate across technical teams (data engineering, ML, DevOps) and business units to define scope, deliverables, and success metrics. Facilitate sprint planning, daily stand-ups, retrospectives, and status reporting following Agile/Scrum or hybrid methodologies. Identify risks, dependencies, and blockers early; drive resolution through mitigation plans and stakeholder communication. Manage vendor relationships (where applicable), ensuring delivery quality, alignment with architecture standards, and on-time execution. Ensure compliance with data governance, security, and documentation standards. Communicate regularly with senior leadership on project status, KPIs, and key decisions. Required Qualifications: 5+ years of experience managing technical or data-related projects, with at least 2+ years in cloud data platforms . Proven experience leading projects involving Databricks , Delta Lake , DBT , and distributed data pipelines. Strong knowledge of data lakehouse architecture , data ingestion/ETL, and modern data platforms (AWS, Azure). Solid understanding of Agile delivery practices, change management, and cross-functional coordination. Proficiency in project tracking tools (Jira, Confluence, Smartsheet, or Microsoft Project). Exceptional written and verbal communication skills; able to translate technical concepts to business audiences. Preferred Qualifications: PMP, PMI-ACP, or Certified Scrum Master (CSM) certification. Prior experience in enterprise data modernization or AI/ML-enabling platforms. Prior experience on multi-cloud platforms is preferred Familiarity with tools such as Airflow, Unity Catalog, Power BI/Tableau, and Git-based CI/CD processes. Soft Skills: Strong leadership and stakeholder management Proactive problem solver with a bias for execution Excellent time management and multitasking ability Comfortable working in a fast-paced, evolving environment

Primary Job Responsibilities: Collaborate with team members to maintain, monitor, and improve data ingestion pipelines on the Data & AI platform. Attend the office 3 times a week for collaborative sessions and team alignment. Drive innovation in ingestion and analytics domains to enhance performance and scalability. Work closely with the domain architect to implement and evolve data engineering strategies. Required Skills: Minimum 5 years of experience in Python development focused on Data Engineering. Hands-on experience with Databricks and Delta Lake format. Strong proficiency in SQL, data structures, and robust coding practices. Solid understanding of scalable data pipelines and performance optimization. Preferred / Nice to Have: Familiarity with monitoring tools like Prometheus and Grafana. Experience using Copilot or AI-based tools for code enhancement and efficiency.

The Development Lead will oversee the design, development, and delivery of advanced data solutions using Azure Databricks, SQL, and data visualization tools like Power BI. The role involves leading a team of developers, managing data pipelines, and creating insightful dashboards and reports to drive data-driven decision-decision-making across the organization. The individual will ensure best practices are followed in data architecture, development, and reporting while maintaining alignment with business objectives. Key Responsibilities: Data Integration & ETL Processes: Design, build, and optimize ETL pipelines to manage the flow of data from various sources into data lakes, data warehouses, and reporting platforms. Data Visualization & Reporting: Lead the development of interactive dashboards and reports using Power BI, ensuring that business users have access to actionable insights and performance metrics. SQL Development & Optimization: Write, optimize, and review complex SQL queries for data extraction, transformation, and reporting, ensuring high performance and scalability across large datasets. Azure Cloud Solutions: Implement and manage cloud-based solutions using Azure services (Azure Databricks, Azure SQL Database, Data Lake) to support business intelligence and reporting initiatives. Collaboration with Stakeholders: Work closely with business leaders and cross-functional teams to understand reporting and analytics needs, translating them into technical requirements and actionable data solutions. Quality Assurance & Best Practices: Implement and maintain best practices in development, ensuring code quality, version control, and adherence to data governance standards. Performance Monitoring & Tuning: Continuously monitor the performance of data systems, reporting tools, and dashboards to ensure they meet SLAs and business requirements. Documentation & Training: Create and maintain comprehensive documentation for all data solutions, including architecture diagrams, ETL workflows, and data models. Provide training and support to end-users on Power BI reports and dashboards. Required Qualifications: Bachelor's or Master's degree in Computer Science, Information Systems, or a related field. Proven experience as a Development Lead or Senior Data Engineer with expertise in Azure Databricks, SQL, Power BI, and data reporting/visualization. Hands-on experience in Azure Databricks for large-scale data processing and analytics, including Delta Lake, Spark SQL, and integration with Azure Data Lake. Strong expertise in SQL for querying, data transformation, and database management. Proficiency in Power BI for developing advanced dashboards, data models, and reporting solutions. Experience in ETL design and data integration across multiple systems, with a focus on performance optimization. Knowledge of Azure cloud architecture, including Azure SQL Database, Data Lake, and other relevant services. Experience leading agile development teams, with a strong focus on delivering high-quality, scalable solutions. Strong problem-solving skills, with the ability to troubleshoot and resolve complex data and reporting issues. Excellent communication skills, with the ability to interact with both technical and non-technical stakeholders. Preferred Qualifications: Knowledge of additional Azure services (e.g., Azure Synapse, Data Factory, Logic Apps) is a plus. Experience in Power BI for data visualization and custom calculations.

Greetings from TCS Recruitment Team! Role: DATABRICKS LEAD/ DATABRICKS SOLUTION ARCHITECT/ DATABRICKS ML ENGINEER Years of experience: 7 to 18 Years Walk-In-Drive Location: Kochi Walk-in-Location Details: Tata Consultancy Services TCS Centre SEZ Unit, Infopark Kochi Phase 1, Infopark Kochi P.O, Kakkanad, Kochi - 682042, Kerala India Drive Time: 9 am to 1:00 PM Date: 21-Jun-25 Must have 5+ years of experience in data engineering or related fields At least 2-3 years of hands-on experience with Databricks (using Apache Spark, Delta Lake, etc.) Solid experience in working with big data technologies such as Hadoop, Spark, Kafka, or similar Experience with cloud platforms (AWS, Azure, or GCP) and cloud-native data tools Experience with machine learning frameworks and pipelines, particularly in Databricks. Experience with AI/ML model deployment, MLOps, and ML lifecycle management using Databricks and related tools.

Primary Job Responsibilities: Collaborate with team members to maintain, monitor, and improve data ingestion pipelines on the Data & AI platform. Attend the office 3 times a week for collaborative sessions and team alignment. Drive innovation in ingestion and analytics domains to enhance performance and scalability. Work closely with the domain architect to implement and evolve data engineering strategies Required Skills: Minimum 5 years of experience in Python development focused on Data Engineering. Hands-on experience with Databricks and Delta Lake format. Strong proficiency in SQL, data structures, and robust coding practices. Solid understanding of scalable data pipelines and performance optimization. Preferred / Nice to Have: Familiarity with monitoring tools like Prometheus and Grafana. Experience using Copilot or AI-based tools for code enhancement and efficiency.