24 Spark Sql Jobs

Databricks developer with deep SQL expertise to support the development of scalable data pipelines and analytics workflows, will work closely with data engineers BIanalysts to prepare clean, query-optimized datasets for reporting and modeling.

About the Role: We are seeking a skilled and experienced Data Engineer to join our remote team. The ideal candidate will have 5-7 years of professional experience working with Python, PySpark, SQL, and Spark SQL, and will play a key role in building scalable data pipelines, optimizing data workflows, and supporting data-driven decision-making across the organization. Key Responsibilities: Design, build, and maintain scalable and efficient data pipelines using PySpark and SQL. Develop and optimize Spark jobs for large-scale data processing. Collaborate with data scientists, analysts, and other engineers to ensure data quality and accessibility. Implement data integration from multiple sources into a unified data warehouse or lake. Monitor and troubleshoot data pipelines and ETL jobs for performance and reliability. Ensure best practices in data governance, security, and compliance. Create and maintain technical documentation related to data pipelines and infrastructure. Location-Delhi NCR,Bangalore,Chennai,Pune,Kolkata,Ahmedabad,Mumbai,Hyderabad,Remote

Role Purpose The role incumbent is focused on implementation of roadmaps for business process analysis, data analysis, diagnosis of gaps, business requirements & functional definitions, best practices application, meeting facilitation, and contributes to project planning. Consultants are expected to contribute to solution building for the client & practice. The role holder can handle higher scale and complexity compared to a Consultant profile and is more proactive in client interactions. Do Assumes responsibilities as the main client contact leading engagement w/ 10-20% support from Consulting & Client Partners. Develops, assesses, and validates a clients business strategy, including industry and competitive positioning and strategic direction Develops solutions and services to suit clients business strategy Estimates scope and liability for delivery of the end product/solution Seeks opportunities to develop revenue in existing and new areas Leads an engagement and oversees others contributions at a customer end, such that customer expectations are met or exceeded. Drives Proposal creation and presales activities for the engagement; new accounts Contributes towards the development of practice policies, procedures, frameworks etc. Guides less experienced team members in delivering solutions. Leads efforts towards building go-to-market/ off the shelf / point solutions and process smethodologies for reuse Creates reusable IP from managed projects B?usiness System Analyst Skills required: Data warehousing, data analysis, ETL, SQL, spark SQL, data mapping, Azure, Databricks Pyspark and data bricks Experience in AML/ Banking/ Capital Market Experience with Agile methodology Plan, develop, test and deploy Experience with SQL server management tools and writing queries, with large SQL data marts, relational database experience. Profile data and prepare source to target mappings, map source data to the target tables for the multi hop architecture. Closely working with data modelers, data engineers and architect and product owners to identify any discrepancies and get it resolved. Strong knowledge of data extraction, design, load, and reporting solutions, strong ability to read SQL server SSIS packages, SQL stored procedures/functions and back trace views Work in the team of data designer/ developer to translate user and/or systems requirements into functional technical specifications. Work with business stakeholders and other SMEs to assess current capabilities, understand high-level business requirements and apply technical background/understanding in the development of System Requirements Specification (SRS) documents. Collaborate closely with Application Owners, Application Managers, and Solution Designers as the business/functional counterpart in solution identification and maintenance. Support testing teams in translating requirements and use cases into test conditions and expected results for product, performance, user acceptance, and operational acceptance testing; participate in the testing of developed systems/solutions. Act as a technical resource for business partners to ensure deliverables meet business and end-user requirements. Identifying the source data/source tables as per project specific use case and get that ingested into source raw zone. Raise Data access control requests and make source data available in analytical zone for data profiling and data analysis. Understanding the existing codes and data bricks workflows and leverage that for our use case. Mandatory Skills: Institutional Compliance. Experience: 8-10 Years.

Designation : Azure Data Engineer Experience : 5+ Years Location: Chennai, Bangalore, Pune, Mumbai Notice Period: Immediate Joiners/ Serving Notice Period Shift Timing: 3:30 PM IST to 12:30 AM IST Job Description : Azure Data Engineer: Must Have Azure Data Bricks, Azure Data Factory, Spark SQL with analytical knowledge Years 6-7 years of development experience in data engineering skills Strong experience in Spark. Understand complex data system by working closely with engineering and product teams Develop scalable and maintainable applications to extract, transform, and load data in various formats to SQL Server, Hadoop Data Lake or other data storage locations. Sincerely, Sonia HR Recruiter Talent Sketchers

Seeking an experienced Data Engineer who can play a crucial role in the company's fintech data lake project. Technical/Functional Skills: Must have 5+ years of experience working in data warehousing systems Strong experience in Oracle Fusion ecosystem, with strong data-extracting experience using Oracle BICC/BIP. Must have good functional understanding of Fusion data structures. Must have strong and proven data engineering experience in big data Databricks environment Must have hands-on experience building data ingestion pipelines from Oracle Fusion Cloud to a Databricks environment Strong data transformation/ETL skills using Spark SQL, Pyspark, Unity Catalog working in Databricks Medallion architecture Capable of independently delivering work items and leading data discussions with Tech Leads & Business Analysts Nice to have: Experience with Fivetran or any equivalent data extraction tools is nice to have. Experience in supporting Splash report development activities is a plus. Prefer experience with Git, CI/CD tools, and code management processes The candidate is expected to: Follow engineering best practices, coding standards, and deployment processes. Troubleshoot any performance, system or data related issues, and work to ensure data consistency and integrity. Effectively communicate with users at all levels of the organization, both in written and verbal presentations. Effectively communicate with other data engineers, help other team members with design and implementation activities. Location: Delhi NCR,Bangalore,Chennai,Pune,Kolkata,Ahmedabad,Mumbai,Hyderabad

What this job involves: JLL, an international real estate management company, is seeking an Data Engineer to join our JLL Technologies Team. We are seeking candidates that are self-starters to work in a diverse and fast-paced environment that can join our Enterprise Data team. We are looking for a candidate that is responsible for designing and developing of data solutions that are strategic for the business using the latest technologies Azure Databricks, Python, PySpark, SparkSQL, Azure functions, Delta Lake, Azure DevOps CI/CD. Responsibilities Design, Architect, and Develop solutions leveraging cloud big data technology to ingest, process and analyze large, disparate data sets to exceed business requirements. Design & develop data management and data persistence solutions for application use cases leveraging relational, non-relational databases and enhancing our data processing capabilities. Develop POCs to influence platform architects, product managers and software engineers to validate solution proposals and migrate. Develop data lake solution to store structured and unstructured data from internal and external sources and provide technical guidance to help migrate colleagues to modern technology platform. Contribute and adhere to CI/CD processes, development best practices and strengthen the discipline in Data Engineering Org. Develop systems that ingest, cleanse and normalize diverse datasets, develop data pipelines from various internal and external sources and build structure for previously unstructured data. Using PySpark and Spark SQL, extract, manipulate, and transform data from various sources, such as databases, data lakes, APIs, and files, to prepare it for analysis and modeling. Build and optimize ETL workflows using Azure Databricks and PySpark. This includes developing efficient data processing pipelines, data validation, error handling, and performance tuning. Perform the unit testing, system integration testing, regression testing and assist with user acceptance testing. Articulates business requirements in a technical solution that can be designed and engineered. Consults with the business to develop documentation and communication materials to ensure accurate usage and interpretation of JLL data. Implement data security best practices, including data encryption, access controls, and compliance with data protection regulations. Ensure data privacy, confidentiality, and integrity throughout the data engineering processes. Performs data analysis required to troubleshoot data related issues and assist in the resolution of data issues. Experience & Education Minimum of 4 years of experience as a data developer using Python, PySpark, Spark Sql, ETL knowledge, SQL Server, ETL Concepts. Bachelors degree in Information Science, Computer Science, Mathematics, Statistics or a quantitative discipline in science, business, or social science. Experience in Azure Cloud Platform, Databricks, Azure storage. Effective written and verbal communication skills, including technical writing. Excellent technical, analytical and organizational skills. Technical Skills & Competencies Experience handling un-structured, semi-structured data, working in a data lake environment, leveraging data streaming and developing data pipelines driven by events/queues Hands on Experience and knowledge on real time/near real time processing and ready to code Hands on Experience in PySpark, Databricks, and Spark Sql. Knowledge on json, Parquet and Other file format and work effectively with them No Sql Databases Knowledge like Hbase, Mongo, Cosmos etc. Preferred Cloud Experience on Azure or AWS Python-spark, Spark Streaming, Azure SQL Server, Cosmos DB/Mongo DB, Azure Event Hubs, Azure Data Lake Storage, Azure Search etc. Team player, Reliable, self-motivated, and self-disciplined individual capable of executing on multiple projects simultaneously within a fast-paced environment working with cross functional teams.

Role & responsibilities Urgent Hiring for one of the reputed MNC Exp - 5+ Years Location - Pan India Immediate Joiners only Snowflake developer , Pyspark , Python , API, CI/CD , Cloud services ,Azure , Azure Devops Subject: Fw : TMNA SNOWFLAKE POSITION Please share profiles for Snowflake developers having strong Pyspark experience Job Description: Strong hands-on experience in Snowflake development including Streams, Tasks, and Time Travel Deep understanding of Snowpark for Python and its application for data engineering workflows Proficient in PySpark , Spark SQL, and distributed data processing Experience with API development . Proficiency in cloud services (preferably Azure, but AWS/GCP also acceptable) Solid understanding of CI/CD practices and tools like Azure DevOps, GitHub Actions, GitLab, or Jenkins for snowflake. Knowledge of Delta Lake, Data Lakehouse principles, and schema evolution is a plus Preferred candidate profile

Role & responsibilities Urgent hiring for one of the reputed MNC Data Analyst Exp - 5 - 10 Years Only immediate joiners Location - Bangalore JD: Data Analyst Mandatory SKILLS 1. SQL : Proficient in database object creation including tables, views, indexes etc. Strong expertise in SQL queries ,Stored procedure & Function etc. Experienced in performance tuning & optimization techniques. 2.PowerBI : Proficiency in Power BI development, including report and dashboard creation Design, develop, and maintain complex Power BI data models, ensuring data integrity and consistency. Comprehensive understanding of data modeling and data visualization concepts Identify and resolve performance bottlenecks in Power BI reports and data models. Experience with Power Query & DAX 3. Problem-Solving Skills: Strong analytical and problem-solving skills to identify and resolve data-related issues. 4.Python : Strong proficiency in Python programming. 5.PySpark: Extensive experience with PySpark, including DataFrames & SparkSQL. Preferred candidate profile

Responsibilities: Solve problems, analyze and isolate issues. Provide technical guidance and mentoring to the team and help them adopt change as new processes are introduced. Champion best practices and serve as a subject matter authority. Develop solutions to develop/support key business needs. Engineer components and common services based on standard development models, languages and tools Produce system design documents and lead technical walkthroughs Produce high quality code Collaborate effectively with technical and non-technical partners As a team-member should continuously improve the architecture Basic Qualifications: 6-8 years of experience in application development using Java or Dot Net (.NET) Technologies Bachelor's /Masters degree in computer science, Information Systems or equivalent. Knowledge of object-oriented design, .NET framework and design patterns. Command of essential technologies: Java and/or C#, ASP.NET Experience with developing solutions involving relational database technologies: SQL, stored procedures Proficient with software development lifecycle (SDLC) methodologies like Agile, Test-Driven Development. Good communication and collaboration skills Preferred Qualifications: Search Technologies: Query and indexing content for Apache Solr, Elastic Search Big Data Technologies: Apache Spark, Spark SQL, Hadoop, Hive, Airflow Data Science Search Technologies: Personalization and Recommendation models, Learn to Rank (LTR) Preferred Languages: Python Database Technologies: MS SQL Server platform, stored procedure programming experience using Transact SQL. Ability to lead, train and mentor.

As the Full Stack Developer within the Data and Analytics team, you will be responsible for delivery of innovative data and analytics solutions, ensuring Al Futtaim Business stays at the forefront of technical development.

Role :- Resident Solution ArchitectLocation: RemoteThe Solution Architect at Koantek builds secure, highly scalable big data solutions to achieve tangible, data-driven outcomes all the while keeping simplicity and operational effectiveness in mind This role collaborates with teammates, product teams, and cross-functional project teams to lead the adoption and integration of the Databricks Lakehouse Platform into the enterprise ecosystem and AWS/Azure/GCP architecture This role is responsible for implementing securely architected big data solutions that are operationally reliable, performant, and deliver on strategic initiatives Specific requirements for the role include: Expert-level knowledge of data frameworks, data lakes and open-source projects such as Apache Spark, MLflow, and Delta Lake Expert-level hands-on coding experience in Python, SQL ,Spark/Scala,Python or Pyspark In depth understanding of Spark Architecture including Spark Core, Spark SQL, Data Frames, Spark Streaming, RDD caching, Spark MLib IoT/event-driven/microservices in the cloud- Experience with private and public cloud architectures, pros/cons, and migration considerations Extensive hands-on experience implementing data migration and data processing using AWS/Azure/GCP services Extensive hands-on experience with the Technology stack available in the industry for data management, data ingestion, capture, processing, and curation: Kafka, StreamSets, Attunity, GoldenGate, Map Reduce, Hadoop, Hive, Hbase, Cassandra, Spark, Flume, Hive, Impala, etc Experience using Azure DevOps and CI/CD as well as Agile tools and processes including Git, Jenkins, Jira, and Confluence Experience in creating tables, partitioning, bucketing, loading and aggregating data using Spark SQL/Scala Able to build ingestion to ADLS and enable BI layer for Analytics with strong understanding of Data Modeling and defining conceptual logical and physical data models Proficient level experience with architecture design, build and optimization of big data collection, ingestion, storage, processing, and visualization Responsibilities : Work closely with team members to lead and drive enterprise solutions, advising on key decision points on trade-offs, best practices, and risk mitigationGuide customers in transforming big data projects,including development and deployment of big data and AI applications Promote, emphasize, and leverage big data solutions to deploy performant systems that appropriately auto-scale, are highly available, fault-tolerant, self-monitoring, and serviceable Use a defense-in-depth approach in designing data solutions and AWS/Azure/GCP infrastructure Assist and advise data engineers in the preparation and delivery of raw data for prescriptive and predictive modeling Aid developers to identify, design, and implement process improvements with automation tools to optimizing data delivery Implement processes and systems to monitor data quality and security, ensuring production data is accurate and available for key stakeholders and the business processes that depend on it Employ change management best practices to ensure that data remains readily accessible to the business Implement reusable design templates and solutions to integrate, automate, and orchestrate cloud operational needs and experience with MDM using data governance solutions Qualifications : Overall experience of 12+ years in the IT field Hands-on experience designing and implementing multi-tenant solutions using Azure Databricks for data governance, data pipelines for near real-time data warehouse, and machine learning solutions Design and development experience with scalable and cost-effective Microsoft Azure/AWS/GCP data architecture and related solutions Experience in a software development, data engineering, or data analytics field using Python, Scala, Spark, Java, or equivalent technologies Bachelors or Masters degree in Big Data, Computer Science, Engineering, Mathematics, or similar area of study or equivalent work experience Good to have- - Advanced technical certifications: Azure Solutions Architect Expert, - AWS Certified Data Analytics, DASCA Big Data Engineering and Analytics - AWS Certified Cloud Practitioner, Solutions Architect - Professional Google Cloud Certified Location : - Mumbai, Delhi / NCR, Bengaluru , Kolkata, Chennai, Hyderabad, Ahmedabad, Pune, Remote

Warm Greetings from SP Staffing!! Role: Pyspark Developer Experience Required :4 to 6 yrs Work Location: Hyderabad/Bangalore/Pune/Chennai/Kochi Required Skills, pyspark/python/spark sql/ETL Interested candidates can send resumes to nandhini.spstaffing@gmail.com

Position Summary The Senior BI Data Analyst will collaborate with business stakeholders and team members to define BI solutions, data requirements, and project scope. They will recommend innovative BI solution approaches that meet or exceed business needs and are responsible for managing BI development projects from inception to completion. The role also includes providing ongoing support and enhancements, ensuring that solutions remain effective and aligned with organizational goals. Additionally, the analyst will contribute to the BI Center of Excellence by sharing knowledge, training users, and staying abreast of emerging BI trends to maintain a competitive edge. Key Responsibilities BI Report, Dashboard Data Development Partner with internal stakeholders to understand their BI and data needs, aligning priorities with business objectives. Define development efforts, and design state-of-the-art dashboards, routine reports, automated reporting tools, ad-hoc reports, and critical business analytics essential for data-driven decision-making and insights. Strategic Insights Work closely with business leaders and subject matter experts to identify opportunities where BI solutions can enhance decision-making capabilities. Recommend and implement BI/Analytic solutions tailored to these needs. Innovation and Research Stay ahead of emerging trends and technologies in BI, data analytics, and data integration. Recommend new strategies, systems, and tools to build the organizations capabilities, enhance knowledge bases, and improve BI platforms, giving the company a competitive edge. Qualifications and Education Educational Background Masters/Bachelors degree in Computer Science, Information Systems, Analytics, or a related field is preferred. Alternatively, a BA/BS with relevant work experience will be considered. Experience, Knowledge Skills 7-10+ years in progressively responsible roles in BI/Data dashboard and report development teams. 10+ years of experience with BI systems, data integration, and the development of BI/Data/Analytics Solutions. Advanced knowledge proficiency in BI platforms and related tools such as Power BI, Power BI Report Server, Power Platform, MS Fabric. Proficiency working with data integration orchestration platforms such as Databricks, Azure Data Factory, SSIS, Informatica, and related technologies. Strong understanding of SQL development best practices using programming languages such as Oracle PL/SQL, Transact SQL, Spark SQL, and/or Python. Knowledge of BI systems architecture, and database platforms, both on-premises and cloud-based (SaaS) platforms. Demonstrated ability to create BI strategies that align with organizational/departmental goals, ensuring that BI efforts directly support the companys and departments strategic objectives. Ability to work well with non-technical business partners. Excellent written and oral communication skills. Ability to deliver results in a timely manner with consistency and precision. Strong project and time management skills to successfully navigate multifaceted responsibilities. Ability to work independently or as part of a collaborative team.

Experience in Spark 2.0 and above. Java 8 and above required.Design, code, test, document and implement application release projects as part of development team. Experience in programming/debugging used in business applications.

Key Responsibilities: Develop and optimize data pipelines in Databricks for transforming and processing data from various sources. Integrate data using Unity Catalog and external data sources (data lakes, APIs, etc.). Write Spark SQL and PySpark scripts for data transformations, optimizations, and creating views/procedures. Perform data analysis to identify quality issues, optimize pipelines, and enhance data processing for analytics. Collaborate on report generation and dashboard creation with front-end teams. Use GitLab for version control, CI/CD automation, and task management (Jira). Required Skills and Qualifications: Masters or Bachelor’s degree in Data Engineering or related field. 3+ years experience with Databricks and advanced SQL. Experience with ETL processes, views, and procedures. Strong experience with Databricks, Spark SQL, PySpark, and SQL. Expertise in creating and optimizing views and stored procedures in Databricks. Experience building ETL workflows and data models. Knowledge of cloud platforms (AWS, Azure) and version control tools (Git, GitLab). Experience with healthcare or clinical trial data. Familiarity with DevOps practices.

Position Summary: Data engineer on the Data integration team Job Description & Responsibilities: Work with business and technical leadership to understand requirements. Design to the requirements and document the designs Ability to write product-grade performant code for data extraction, transformations and loading using Spark, Py-Spark Do data modeling as needed for the requirements. Write performant queries using Teradata SQL, Hive SQL and Spark SQL against Teradata and Hive Implementing dev-ops pipelines to deploy code artifacts on to the designated platform / servers like AWS / Azure / GCP. Troubleshooting the issues, providing effective solutions and jobs monitoring in the production environment Participate in sprint planning sessions, refinement/story-grooming sessions, daily scrums, demos and retrospectives. Experience Required: Overall 8-10 years of experience Experience Desired: Strong development experience in Spark, Py-Spark, Shell scripting, Teradata. Strong experience in writing complex and effective SQLs (using Teradata SQL, Hive SQL and Spark SQL) and Stored Procedures Health care domain knowledge is a plus Education and Training Required: Primary Skills: Excellent work experience on Databricks as Data Lake implementations Experience in Agile and working knowledge on DevOps tools (Git, Jenkins, Artifactory) Experience in AWS (S3, EC2, SNS, SQS, Lambda, ECS, Glue, IAM, and CloudWatch) / GCP / Azure Databricks (Delta lake, Notebooks, Pipelines, cluster management, Azure / AWS integration Additional Skills: Experience in Jira and Confluence Exercises considerable creativity, foresight, and judgment in conceiving, planning, and delivering initiatives.

Position Summary: Data engineer on the Data integration team Job Description & Responsibilities: Work with business and technical leadership to understand requirements. Design to the requirements and document the designs Ability to write product-grade performant code for data extraction, transformations and loading using Spark, Py-Spark Do data modeling as needed for the requirements. Write performant queries using Teradata SQL, Hive SQL and Spark SQL against Teradata and Hive Implementing dev-ops pipelines to deploy code artifacts on to the designated platform/servers like AWS. Troubleshooting the issues, providing effective solutions and jobs monitoring in the production environment Participate in sprint planning sessions, refinement/story-grooming sessions, daily scrums, demos and retrospectives. Experience Required: Overall 8-10 years of experience Experience Desired: Strong development experience in Spark, Py-Spark, Shell scripting, Teradata. Strong experience in writing complex and effective SQLs (using Teradata SQL, Hive SQL and Spark SQL) and Stored Procedures Health care domain knowledge is a plus Primary Skills: Excellent work experience on Databricks as Data Lake implementations Experience in Agile and working knowledge on DevOps tools (Git, Jenkins, Artifactory) AWS (S3, EC2, SNS, SQS, Lambda, ECS, Glue, IAM, and CloudWatch) Databricks (Delta lake, Notebooks, Pipelines, cluster management, Azure/AWS integration Additional Skills: Experience in Jira and Confluence Exercises considerable creativity, foresight, and judgment in conceiving, planning, and delivering initiatives.

JD -7+ years of hands on experience in Python especially dealing with Pandas and Numpy Good hands-on experience in Spark PySpark and Spark SQL Hands on experience in Databricks Unity Catalog Delta Lake Lake house Platform Medallion Architecture Azure Data Factory ADLS Experience in dealing with Parquet and JSON file format Knowledge in Snowflake.

Job Title: Azure Data Engineer - Manager (5-9 Years of Experience) Location: [Bangalore, Coimbatore, Pune] Employment Type: Full-Time Job Description: We are looking for an experienced Azure Data Engineer with 9-13 years of experience to take on a strong Technical and Managerial role within our organization. The ideal candidate will have a strong technical background in Databricks , Azure Data Services , and data engineering, coupled with proven leadership and team management skills. This role requires a balance of hands-on technical expertise and the ability to lead, mentor, and manage a team of data engineers to deliver high-quality data solutions. Key Responsibilities: Technical Responsibilities: Data Pipeline Development: Design, develop, and optimize scalable data pipelines using Databricks , Azure Data Factory , and other Azure data services. Implement advanced ETL/ELT processes to handle large volumes of data from diverse sources. Ensure data pipelines are efficient, reliable, and scalable to meet business needs. Data Processing and Analytics: Write and optimize complex SQL queries for data extraction, transformation, and analysis. Use PySpark for large-scale data processing, transformation, and analytics. Implement data partitioning, indexing, and caching strategies for optimal performance. Data Integration and Governance: Integrate data from multiple sources, including structured, semi-structured, and unstructured data. Implement data governance practices to ensure data quality, consistency, and security. Monitor and troubleshoot data pipelines to ensure data accuracy and availability. Architecture and Design: Define and implement data architecture best practices, including data lake and data warehouse design. Collaborate with cross-functional teams to design and deliver end-to-end data solutions. Evaluate and recommend new tools and technologies to enhance data engineering capabilities. Managerial Responsibilities: Team Leadership: Lead, mentor, and manage a team of data engineers, ensuring high performance and professional growth. Assign tasks, set priorities, and ensure timely delivery of projects. Conduct regular team meetings, performance reviews, and one-on-one sessions. Project Management: Oversee the end-to-end delivery of data engineering projects, ensuring alignment with business goals. Collaborate with stakeholders to define project scope, timelines, and deliverables. Manage project risks, issues, and dependencies to ensure successful project execution. Stakeholder Collaboration: Work closely with data scientists, analysts, and business stakeholders to understand requirements and deliver solutions. Act as a bridge between technical teams and business stakeholders, ensuring clear communication and alignment. Process Improvement: Identify opportunities for process improvement and implement best practices in data engineering. Drive innovation and continuous improvement within the team. Must-Have Skills: Databricks: Extensive hands-on experience with Databricks for data processing, analytics, and machine learning. Azure Data Services: Proficiency in Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, and Azure SQL Database. SQL: Expertise in writing and optimizing complex SQL queries. PySpark: Advanced knowledge of PySpark for large-scale data processing and transformation. ETL/ELT: Strong understanding of ETL/ELT processes and tools. Data Modeling: Deep knowledge of data modeling techniques and best practices. Leadership: Proven experience in leading and managing teams of data engineers. Project Management: Strong project management skills with the ability to manage multiple projects simultaneously. Good-to-Have Skills: Experience with Azure DevOps for CI/CD pipelines. Knowledge of Delta Lake for building reliable data lakes. Familiarity with streaming data technologies like Azure Event Hubs or Kafka. Qualifications: Bachelors or Masters degree in Computer Science, Information Technology, or a related field. 9-13 years of experience in data engineering, with a focus on Azure and Databricks. Proven experience in a leadership or managerial role, leading teams of 5+ members. Relevant certifications such as Microsoft Certified: Azure Data Engineer Associate or Databricks Certified Associate Developer are a plus. Soft Skills: Strong leadership and team management skills. Excellent communication and interpersonal skills. Ability to work in a fast-paced, dynamic environment. Strong problem-solving and analytical skills. Self-motivated with a strong sense of ownership and accountability.

Hands-on experience in Scala programming and Apache Spark. Strong expertise in Spark architecture+ including RDDs+ DataFrames+ and Spark SQL. Proven experience in performance tuning and optimization of Spark applications. Must have hands-on experience with Spark Streaming for real-time data processing. Solid understanding of distributed computing and big data processing concepts. Proficient in Linux with the ability to work in a Linux environment. Strong knowledge of data structures and algorithms+ with a focus on space and time complexity analysis. Ability to work independently and deliver results in a fast-paced+ high-pressure environment. Excellent problem-solving+ debugging+ and analytical skills.

- Develop innovative solutions using data analysis - Lead Machine Learning projects - Coach junior data scientists - Design predictive models/algorithms - Apply machine learning for process optimization - Work with analytics software (SQL, R, Python) Required Candidate profile Technical skills : Machine Learning, Data Science, Python, Spark SQL, Microsoft Azure Cloud, Databricks, Mlflow Job Experience: 5+ years in Data Science/Machine Learning

We are seeking an experienced PySpark Developer / Data Engineer to design, develop, and optimize big data processing pipelines using Apache Spark and Python (PySpark). The ideal candidate should have expertise in distributed computing, ETL workflows, data lake architectures, and cloud-based big data solutions. Key Responsibilities: Develop and optimize ETL/ELT data pipelines using PySpark on distributed computing platforms (Hadoop, Databricks, EMR, HDInsight). Work with structured and unstructured data to perform data transformation, cleansing, and aggregation. Implement data lake and data warehouse solutions on AWS (S3, Glue, Redshift), Azure (ADLS, Synapse), or GCP (BigQuery, Dataflow). Optimize PySpark jobs for performance tuning, partitioning, and caching strategies. Design and implement real-time and batch data processing solutions. Integrate data pipelines with Kafka, Delta Lake, Iceberg, or Hudi for streaming and incremental updates. Ensure data security, governance, and compliance with industry best practices. Work with data scientists and analysts to prepare and process large-scale datasets for machine learning models. Collaborate with DevOps teams to deploy, monitor, and scale PySpark jobs using CI/CD pipelines, Kubernetes, and containerization. Perform unit testing and validation to ensure data integrity and reliability. Required Skills & Qualifications: 6+ years of experience in big data processing, ETL, and data engineering. Strong hands-on experience with PySpark (Apache Spark with Python). Expertise in SQL, DataFrame API, and RDD transformations. Experience with big data platforms (Hadoop, Hive, HDFS, Spark SQL). Knowledge of cloud data processing services (AWS Glue, EMR, Databricks, Azure Synapse, GCP Dataflow). Proficiency in writing optimized queries, partitioning, and indexing for performance tuning. Experience with workflow orchestration tools like Airflow, Oozie, or Prefect. Familiarity with containerization and deployment using Docker, Kubernetes, and CI/CD pipelines. Strong understanding of data governance, security, and compliance (GDPR, HIPAA, CCPA, etc.). Excellent problem-solving, debugging, and performance optimization skills.

Role: Senior Member of Technical Staff Experience: 4 to 9 years Location: Manyata Tech Park, Bangalore Work mode: Hybrid Skills: Design Data Analytics, Data Engineering, Data Warehouse, Java, SQL, Spark & Spark-SQL, Autonomous Data Warehouse, Object Store, Data Flow/DIS (or similar tech stack), exposure to cloud Short description : Looking for a Software Developer (Data Engineer) who are willing to perform Data Engineering tasks with skills sets like SQL, Spark, Big Data Development, Java & Cloud experience. Job description: Design, develop, troubleshoot and debug software programs for databases, applications etc. Strong knowledge of SQL is required. You should be having programming experience in any of the Java, Python, Scala etc. You should have a good understanding of the data, so that you can perform the data engineering tasks which involves analyzing the data, doing required transformations and loading the data into the Data Warehouse. The Reporting and Visualizations knowledge is a very key for this role. You should be able to develop some visualizations using reporting tools when needed. The experience of development in Cloud like working on processing nodes, storing the data in Object Store buckets, knowledge of databases and performance tuning of data engineering pipelines (batch and real time) is very much needed. Deployment of code in Cloud environment is a required skill. You should have an experience in Agile development. You should be able to perform the Continuous Integration and Continuous deployment of the code we develop. Responsibilities: As a member of the software engineering division, you will apply basic to intermediate knowledge of software architecture to perform software development tasks associated with developing, debugging or designing software applications or operating systems according to provided design specifications. Build enhancements within an existing software architecture and occasionally suggest improvements to the architecture.

Spark Developer/ Engineer: Expert proficiency in Spark Ability to design and implement efficient data processing workflows Experience with Spark SQL and DataFrames Good exposure to Big Data architectures and good understanding of Big Data eco system Experience with some framework building experience on Hadoop Good with DB knowledge with SQL tuning experience. Good to have experience with Python, APIs and exposure to Kafka.