93 Apache Spark Jobs - Page 3

About the Role: Grade Level (for internal use): 11 The Team: Our team is responsible for the design, architecture, and development of our client facing applications using a variety of tools that are regularly updated as new technologies emerge. You will have the opportunity every day to work with people from a wide variety of backgrounds and will be able to develop a close team dynamic with coworkers from around the globe. The Impact: The work you do will be used every single day, its the essential code youll write that provides the data and analytics required for crucial, daily decisions in the capital and commodities markets. Whats in it for you: Build a career with a global company. Work on code that fuels the global financial markets. Grow and improve your skills by working on enterprise level products and new technologies. Responsibilities: Solve problems, analyze and isolate issues. Provide technical guidance and mentoring to the team and help them adopt change as new processes are introduced. Champion best practices and serve as a subject matter authority. Develop solutions to develop/support key business needs. Engineer components and common services based on standard development models, languages and tools Produce system design documents and lead technical walkthroughs Produce high quality code Collaborate effectively with technical and non-technical partners As a team-member should continuously improve the architecture Basic Qualifications: 9-12 years of experience designing/building data-intensive solutions using distributed computing. Proven experience in implementing and maintaining enterprise search solutions in large-scale environments. Experience working with business stakeholders and users, providing research direction and solution design and writing robust maintainable architectures and APIs. Experience developing and deploying Search solutions in a public cloud such as AWS. Proficient programming skills at a high-level languages - Java, Scala, Python Solid knowledge of at least one machine learning research frameworks Familiarity with containerization, scripting, cloud platforms, and CI/CD. 5+ years experience with Python, Java, Kubernetes, and data and workflow orchestration tools 4+ years experience with Elasticsearch, SQL, NoSQL,??Apache spark, Flink, Databricks and Mlflow. Prior experience with operationalizing data-driven pipelines for large scale batch and stream processing analytics solutions Good to have experience with contributing to GitHub and open source initiatives or in research projects and/or participation in Kaggle competitions Ability to quickly, efficiently, and effectively define and prototype solutions with continual iteration within aggressive product deadlines. Demonstrate strong communication and documentation skills for both technical and non-technical audiences. Preferred Qualifications: Search Technologies: Query and Indexing content for Apache Solr, Elastic Search, etc. Proficiency in search query languages (e.g., Lucene Query Syntax) and experience with data indexing and retrieval. Experience with machine learning models and NLP techniques for search relevance and ranking. Familiarity with vector search techniques and embedding models (e.g., BERT, Word2Vec). Experience with relevance tuning using A/B testing frameworks. Big Data Technologies: Apache Spark, Spark SQL, Hadoop, Hive, Airflow Data Science Search Technologies: Personalization and Recommendation models, Learn to Rank (LTR) Preferred Languages: Python, Java Database Technologies: MS SQL Server platform, stored procedure programming experience using Transact SQL. Ability to lead, train and mentor.

As a Backend Python Developer, you will be responsible for designing, developing, and deploying scalable, secure APIs and backend services. You will work closely with cross-functional teams to implement best practices in coding, testing, and deployment. The role also involves ensuring high-quality, modular code delivery while leveraging cloud technologies like AWS, Docker, and Kubernetes. You'll be expected to maintain code quality, collaborate in an Agile environment, and optimize backend systems for performance and scalability. Experience6+ years LocationChennai, Bangalore, Pune, Noida Primary Skills: Core Python, Linux, SQLSecondary skills REST API, Cloud, NoSQL Requirement:- 5+ years of solid experience as a backend Python developer- Experience with Python frameworks (e.g. Django, Flask, Bottle)- Should have good experience in both Python 2 and Python 3- Strong knowledge of Data Structures & Algorithms, OOP, Threads, Parallel-Processing- Experience building secure, complex, and scalable APIs, from design through deployment- Should be able to write clean, modular code. Solid understanding of writing and deliveringtestable quality code- Should have knowledge of SDLC best practices, including coding standards, code reviews,source control management, build processes, testing, and operations. Experience with GIT, Jira & Agile Methodology- Familiarity with Amazon Web Services (AWS) and REST API- Experience with Docker and Kubernetes is a big plus- Experience with SQLNice to Have:- Experience with streaming data and complex event processing systems- Experience in working with NoSQL technologies like Redis, MongoDB, Cassandra is a plus- Working knowledge in AWS, Kafka, Apache Spark, ElasticsearchJava knowledge is a plusJob locationChennai / Bangalore / Pune (Hybrid model - weekly 2 days working from office)QUALIFICATIONS: Bachelors Degree in Computer Science or any related field B.Tech, BE, BCA, etc. 6 to 8 years of experience in software industry

Key Skills: SCALA, AWS, AWS Cloud, Apache Spark, Architect, SparkSQL, Spark, Spring Boot, Java Roles and Responsibilities: Technical lead the team and project to meet deadlines. Lead efforts with team members to come up with software solutions. Optimize and maintain existing software. Recommend tech upgrades to company leaders. Build scalable, efficient, and high-performance pipelines and workflows that are capable of processing large amounts of batch and real-time data. Multidisciplinary work supporting real-time streams, ETL pipelines, data warehouses, and reporting services. Design and develop microservices and data applications that interact with other microservices. Use Big Data technologies such as Kafka, Data Lake on AWS S3, EMR, Spark, and related technologies to ingest, store, aggregate, transform, move, and query data. Follow coding best practices - Unit testing, design/code reviews, code coverage, documentation, etc. Performance analysis and capacity planning for every release. Work effectively as part of an Agile team. Bring new and innovative solutions to resolve challenging software issues as they may develop throughout the product lifecycle. Skills Required: Excellence in software design skills. Strong knowledge of design patterns, including performance optimization considerations. Proficient in writing high-quality, well-structured code in Java and Scala. Excellence in test-driven development approach and debugging software. Proficient in writing clear, concise, and organized documentation. Knowledge of Amazon cloud computing infrastructure (Aurora MySQL, DynamoDB, EMR, Lambda, Step Functions, and S3). Ability to excel in a team environment. Strong communication skills and the ability to discuss a solution with team members of varying technical sophistication. Ability to perform thoughtful and detailed code reviews, both for peers and Junior Developers. Familiarity with software engineering and project management tools. Following security protocols and best data governance practices. Able to construct KPIs and using metrics for process improvements. Minimum qualifications: 12+ years' experience in designing and developing enterprise-level software solutions. 5 years' experience developing Scala/Java applications and microservices using Spring Boot. 10 years' experience with large volume data processing and big data tools such as Apache Spark, Scala, and Hadoop technologies. 5 years' experience with SQL and Relational databases. 2 years' experience working with Agile/Scrum methodology. Education: Bachelor's Degree in related field

We are looking for a highly experienced Lead Data Engineer / Data Architect to lead the design, development, and implementation of scalable data pipelines, data Lakehouse, and data warehousing solutions. The ideal candidate will provide technical leadership to a team of data engineers, drive architectural decisions, and ensure best practices in data engineering. This role is critical in enabling data-driven decision-making and modernizing our data infrastructure. Key Responsibilities: Act as a technical leader responsible for guiding the design, development, and implementation of data pipelines, data Lakehouse, and data warehousing solutions. Lead a team of data engineers, ensuring adherence to best practices and standards. Drive the successful delivery of high-quality, scalable, and reliable data solutions. Play a key role in shaping data architecture, adopting modern data technologies, and enabling data-driven decision-making across the team. Provide technical vision, guidance, and mentorship to the team. Lead technical design discussions, perform code reviews, and contribute to architectural decisions.

Type: Contract-to-Hire (C2H) Job Summary We are looking for a skilled PySpark Developer with MUST 4+ YEARS hands-on experience in building scalable data pipelines and processing large datasets. The ideal candidate will have deep expertise in Apache Spark, Python, and working with modern data engineering tools in cloud environments such as AWS. Key Skills & Responsibilities Strong expertise in PySpark and Apache Spark for batch and real-time data processing. Experience in designing and implementing ETL pipelines, including data ingestion, transformation, and validation. Proficiency in Python for scripting, automation, and building reusable components. Hands-on experience with scheduling tools like Airflow or Control-M to orchestrate workflows. Familiarity with AWS ecosystem, especially S3 and related file system operations. Strong understanding of Unix/Linux environments and Shell scripting. Experience with Hadoop, Hive, and platforms like Cloudera or Hortonworks. Ability to handle CDC (Change Data Capture) operations on large datasets. Experience in performance tuning, optimizing Spark jobs, and troubleshooting. Strong knowledge of data modeling, data validation, and writing unit test cases. Exposure to real-time and batch integration with downstream/upstream systems. Working knowledge of Jupyter Notebook, Zeppelin, or PyCharm for development and debugging. Understanding of Agile methodologies, with experience in CI/CD tools (e.g., Jenkins, Git). Preferred Skills Experience in building or integrating APIs for data provisioning. Exposure to ETL or reporting tools such as Informatica, Tableau, Jasper, or QlikView. Familiarity with AI/ML model development using PySpark in cloud environments.

Role & responsibilities: Technical Skills: 1. Core Databricks Platform: Databricks workspace, clusters, jobs, notebooks, Unity Catalog 2. Big Data Technologies: Apache Spark (PySpark/Scala), Delta Lake, Apache Kafka 3. Programming Languages: Python (advanced), SQL (advanced), Scala (preferred) 4. Cloud Platforms: Azure (preferred) or AWS with Databricks integration 5. Data Pipeline Tools: Apache Airflow, Azure Data Factory, or similar orchestration tools 6. Version Control & CI/CD: Git, Azure DevOps, Jenkins, or GitHub Actions 7. Data Formats & Storage: Parquet, JSON, Avro, Azure Data Lake, S3 8. Monitoring & Observability: Databricks monitoring, custom metrics, alerting systems Leadership & Soft Skills: 1. Strong leadership and people management capabilities 2. Excellent communication skills with ability to explain complex technical concepts 3. Experience with Agile/Scrum methodologies 4. Problem-solving mindset with attention to detail 5. Ability to work in fast-paced, dynamic environments 6. 8+ years of overall experience in data engineering, software engineering, or related technical roles 7. 4+ years of hands-on experience with Databricks/big data platform and Apache Spark 8. 2+ years of team leadership or technical mentoring experience Preferred Qualifications: 1. Databricks certifications (Certified Data Engineer Associate/Professional) 2. Experience with MLOps and machine learning pipeline deployment 3. Knowledge of data mesh or data fabric architectures 4. Experience with streaming data processing using Spark Structured Streaming 5. Background in financial services, healthcare, or retail domains

Responsibilities Lead the design, development, and implementation of big data solutions using Apache Spark and Databricks. Architect and optimize data pipelines and workflows to process large volumes of data efficiently. Utilize Databricks features such as Delta Lake, Databricks SQL, and Databricks Workflows to enhance data processing and analytics capabilities. Collaborate with data engineers, data scientists, and business stakeholders to understand data requirements and deliver high-quality data solutions. Implement best practices for data engineering, including data quality, data governance, and data security. Monitor and troubleshoot performance issues in Spark jobs and Databricks clusters. Mentor and guide junior engineers in the team, promoting a culture of continuous learning and improvement. Stay up-to-date with the latest advancements in Spark and Databricks technologies and incorporate them into the team's practices.

We specialize in delivering high-quality human-curated data and AI-first scaled operations services Based in San Francisco and Hyderabad, we are a fast-moving team on a mission to build AI for Good, driving innovation and societal impact Role Overview: We are looking for a Data Scientist to join and build intelligent, data-driven solutions for our client that enable impactful decisions This role requires contributions across the data science lifecycle from data wrangling and exploratory analysis to building and deploying machine learning models Whether youre just getting started or have years of experience, were looking for individuals who are curious, analytical, and driven to make a difference with data Responsibilities: Design, develop, and deploy machine learning models and analytical solutions Conduct exploratory data analysis and feature engineering Own or contribute to the end-to-end data science pipeline: data cleaning, modeling, validation, and deployment Collaborate with cross-functional teams (engineering, product, business) to define problems and deliver measurable impact Translate business challenges into data science problems and communicate findings clearly Implement A/B tests, statistical tests, and experimentation strategies Support model monitoring, versioning, and continuous improvement in production environments Evaluate new tools, frameworks, and best practices to improve model accuracy and scalability Required Skills: Strong programming skills in Python including libraries such as pandas, NumPy, scikit-learn, matplotlib, seaborn Proficient in SQL, comfortable querying large, complex datasets Sound understanding of statistics, machine learning algorithms, and data modeling Experience building end-to-end ML pipelines Exposure to or hands-on experience with model deployment tools like FastAPI, Flask, MLflow Experience with data visualization and insight communication Familiarity with version control tools (eg, Git) and collaborative workflows Ability to write clean, modular code and document processes clearly Nice to Have: Experience with deep learning frameworks like TensorFlow or PyTorch Familiarity with data engineering tools like Apache Spark, Kafka, Airflow, dbt Exposure to MLOps practices and managing models in production environments Working knowledge of cloud platforms like AWS, GCP, or Azure (e, SageMaker, BigQuery, Vertex AI) Experience designing and interpreting A/B tests or causal inference models Prior experience in high-growth startups or cross-functional leadership roles Educational Qualifications: Bachelors or Masters degree in Computer Science, Data Science, Mathematics, Engineering, or a related field Location : - Mumbai, Delhi / NCR, Bengaluru , Kolkata, Chennai, Hyderabad, Ahmedabad, Pune,India

Role Purpose The purpose of the role is to support process delivery by ensuring daily performance of the Production Specialists, resolve technical escalations and develop technical capability within the Production Specialists. Do Oversee and support process by reviewing daily transactions on performance parameters Review performance dashboard and the scores for the team Support the team in improving performance parameters by providing technical support and process guidance Record, track, and document all queries received, problem-solving steps taken and total successful and unsuccessful resolutions Ensure standard processes and procedures are followed to resolve all client queries Resolve client queries as per the SLAs defined in the contract Develop understanding of process/ product for the team members to facilitate better client interaction and troubleshooting Document and analyze call logs to spot most occurring trends to prevent future problems Identify red flags and escalate serious client issues to Team leader in cases of untimely resolution Ensure all product information and disclosures are given to clients before and after the call/email requests Avoids legal challenges by monitoring compliance with service agreements Handle technical escalations through effective diagnosis and troubleshooting of client queries Manage and resolve technical roadblocks/ escalations as per SLA and quality requirements If unable to resolve the issues, timely escalate the issues to TA & SES Provide product support and resolution to clients by performing a question diagnosis while guiding users through step-by-step solutions Troubleshoot all client queries in a user-friendly, courteous and professional manner Offer alternative solutions to clients (where appropriate) with the objective of retaining customers and clients business Organize ideas and effectively communicate oral messages appropriate to listeners and situations Follow up and make scheduled call backs to customers to record feedback and ensure compliance to contract SLAs Build people capability to ensure operational excellence and maintain superior customer service levels of the existing account/client Mentor and guide Production Specialists on improving technical knowledge Collate trainings to be conducted as triage to bridge the skill gaps identified through interviews with the Production Specialist Develop and conduct trainings (Triages) within products for production specialist as per target Inform client about the triages being conducted Undertake product trainings to stay current with product features, changes and updates Enroll in product specific and any other trainings per client requirements/recommendations Identify and document most common problems and recommend appropriate resolutions to the team Update job knowledge by participating in self learning opportunities and maintaining personal networks Deliver NoPerformance ParameterMeasure1ProcessNo. of cases resolved per day, compliance to process and quality standards, meeting process level SLAs, Pulse score, Customer feedback, NSAT/ ESAT2Team ManagementProductivity, efficiency, absenteeism3Capability developmentTriages completed, Technical Test performance Mandatory Skills: Apache Spark. Experience: 5-8 Years.

Role Description: We are looking for highly motivated expert Data Engineer who can own the design & development of complex data pipelines, solutions and frameworks. The ideal candidate will be responsible to design, develop, and maintain data pipelines, data integration frameworks, and metadata-driven architectures that enable seamless data access and analytics. This role prefers deep expertise in big data processing, distributed computing, data modeling, and governance frameworks to support self-service analytics, AI-driven insights, and enterprise-wide data management. Roles & Responsibilities: Design, develop, and maintain complex ETL/ELT data pipelines in Databricks using PySpark, Scala, and SQL to process large-scale datasets Understand the biotech/pharma or related domains & build highly efficient data pipelines to migrate and deploy complex data across systems Design and Implement solutions to enable unified data access, governance, and interoperability across hybrid cloud environments Ingest and transform structured and unstructured data from databases (PostgreSQL, MySQL, SQL Server, MongoDB etc.), APIs, logs, event streams, images, pdf, and third-party platforms Ensuring data integrity, accuracy, and consistency through rigorous quality checks and monitoring Expert in data quality, data validation and verification frameworks Innovate, explore and implement new tools and technologies to enhance efficient data processing Proactively identify and implement opportunities to automate tasks and develop reusable frameworks Work in an Agile and Scaled Agile (SAFe) environment, collaborating with cross-functional teams, product owners, and Scrum Masters to deliver incremental value Use JIRA, Confluence, and Agile DevOps tools to manage sprints, backlogs, and user stories. Support continuous improvement, test automation, and DevOps practices in the data engineering lifecycle Collaborate and communicate effectively with the product teams, with cross-functional teams to understand business requirements and translate them into technical solutions Must-Have Skills: Hands-on experience in data engineering technologies such as Databricks, PySpark, SparkSQL Apache Spark, AWS, Python, SQL, and Scaled Agile methodologies. Proficiency in workflow orchestration, performance tuning on big data processing. Strong understanding of AWS services Ability to quickly learn, adapt and apply new technologies Strong problem-solving and analytical skills Excellent communication and teamwork skills Experience with Scaled Agile Framework (SAFe), Agile delivery practices, and DevOps practices. Good-to-Have Skills: Data Engineering experience in Biotechnology or pharma industry Experience in writing APIs to make the data available to the consumers Experienced with SQL/NOSQL database, vector database for large language models Experienced with data modeling and performance tuning for both OLAP and OLTP databases Experienced with software engineering best-practices, including but not limited to version control (Git, Subversion, etc.), CI/CD (Jenkins, Maven etc.), automated unit testing, and Dev Ops Education and Professional Certifications Minimum 5 to 8 years of Computer Science, IT or related field experience AWS Certified Data Engineer preferred Databricks Certificate preferred Scaled Agile SAFe certification preferred Soft Skills: Excellent analytical and troubleshooting skills. Strong verbal and written communication skills Ability to work effectively with global, virtual teams High degree of initiative and self-motivation. Ability to manage multiple priorities successfully. Team-oriented, with a focus on achieving team goals. Ability to learn quickly, be organized and detail oriented. Strong presentation and public speaking skills.

Whats in it for you? Pay above market standards The role is going to be contract based with project timelines from 2 12 months, or freelancing Be a part of an Elite Community of professionals who can solve complex AI challenges Work location could be: Remote (Highly likely) Onsite on client location Deccan AIs Office: Hyderabad or Bangalore Responsibilities: Design and architect enterprise-scale data platforms, integrating diverse data sources and tools Develop real-time and batch data pipelines to support analytics and machine learning Define and enforce data governance strategies to ensure security, integrity, and compliance along with optimizing data pipelines for high performance, scalability, and cost efficiency in cloud environments Implement solutions for real-time streaming data (Kafka, AWS Kinesis, Apache Flink) and adopt DevOps/DataOps best practices Required Skills: Strong experience in designing scalable, distributed data systems and programming (Python, Scala, Java) with expertise in Apache Spark, Hadoop, Flink, Kafka, and cloud platforms (AWS, Azure, GCP) Proficient in data modeling, governance, warehousing (Snowflake, Redshift, BigQuery), and security/compliance standards (GDPR, HIPAA) Hands-on experience with CI/CD (Terraform, Cloud Formation, Airflow, Kubernetes) and data infrastructure optimization (Prometheus, Grafana) Nice to Have: Experience with graph databases, machine learning pipeline integration, real-time analytics, and IoT solutions Contributions to open-source data engineering communities What are the next steps? Register on our Soul AI website

Responsibilities: Design and architect enterprise-scale data platforms, integrating diverse data sources and tools Develop real-time and batch data pipelines to support analytics and machine learning Define and enforce data governance strategies to ensure security, integrity, and compliance along with optimizing data pipelines for high performance, scalability, and cost efficiency in cloud environments Implement solutions for real-time streaming data (Kafka, AWS Kinesis, Apache Flink) and adopt DevOps/DataOps best practices Required Skills: Strong experience in designing scalable, distributed data systems and programming (Python, Scala, Java) with expertise in Apache Spark, Hadoop, Flink, Kafka, and cloud platforms (AWS, Azure, GCP) Proficient in data modeling, governance, warehousing (Snowflake, Redshift, Big Query), and security/compliance standards (GDPR, HIPAA) Hands-on experience with CI/CD (Terraform, Cloud Formation, Airflow, Kubernetes) and data infrastructure optimization (Prometheus, Grafana) Nice to Have: Experience with graph databases, machine learning pipeline integration, real-time analytics, and IoT solutions Contributions to open-source data engineering communities

We specialize in delivering high-quality human-curated data and AI-first scaled operations services Based in San Francisco and Hyderabad, we are a fast-moving team on a mission to build AI for Good, driving innovation and societal impact Role Overview: We are seeking a Data Engineer / Data Architect who will be responsible for designing, building, and maintaining scalable data infrastructure and systems for a client Youll play a key role in enabling efficient data flow, storage, transformation, and access across our organization or client ecosystems Whether youre just beginning or already an expert, we value strong technical skills, curiosity, and the ability to translate complex requirements into reliable data pipelines Responsibilities: Design and implement scalable, robust, and secure data pipelines Build ETL/ELT frameworks to collect, clean, and transform structured and unstructured data Collaborate with data scientists, analysts, and backend engineers to enable seamless data access and model integration Maintain data integrity, schema design, lineage, and quality monitoring Optimize performance and ensure reliability of data workflows in production environments Design and manage data warehousing and lakehouse architecture Set up and manage infrastructure using IaC (Infrastructure as Code) when applicable Required Skills: Strong programming skills in Python, SQL, and Shell scripting Hands-on experience with ETL tools and orchestration frameworks (e g, Airflow, Luigi, dbt) Proficiency in relational databases (e g , PostgreSQL, MySQL) and NoSQL databases (e g , MongoDB, Redis) Experience with big data technologies: Apache Spark, Kafka, Hive, Hadoop, etc Deep understanding of data modeling, schema design, and data warehousing concepts Proficient with cloud platforms (AWS/GCP/Azure) and services like Redshift, BigQuery, S3, Dataflow, or Databricks Knowledge of DevOps and CI/CD tools relevant to data infrastructure Nice to Have: Experience working in real-time streaming environments Familiarity with containerization and Kubernetes Exposure to MLOps and collaboration with ML teams Experience with security protocols, data governance, and compliance frameworks Educational Qualifications: Bachelors or Masters in Computer Science, Data Engineering, Information Systems, or a related technical field Location - Mumbai, Delhi / NCR, Bengaluru , Kolkata, Chennai, Hyderabad, Ahmedabad, Pune, India

We are looking for Indias top 1% Computer Vision Engineers for a unique job opportunity to work with the industry leaders Who can be a part of the community? We are looking for top-tier Computer Vision (CV) Engineers with expertise in image/video processing, object detection, and generative AI If you have experience in this field then this is your chance to collaborate with industry leaders Whats in it for you? Pay above market standards The role is going to be contract based with project timelines from 2 12 months, or freelancing Be a part of an Elite Community of professionals who can solve complex AI challenges Work location could be: Remote (Highly likely) Onsite on client location Deccan AIs Office: Hyderabad or Bangalore Responsibilities: Develop and optimize computer vision models for tasks like object detection, image segmentation, and multi-object tracking Lead research on novel techniques using deep learning frameworks (TensorFlow, PyTorch, JAX) Build efficient computer vision pipelines and optimize models for real-time performance Deploy models using microservices (Docker, Kubernetes) and cloud platforms (AWS, GCP, Azure) Lead MLOps practices, including CI/CD pipelines, model versioning, and training optimizations Required Skills: Expert in Python, OpenCV, NumPy, and deep learning architectures (eg, ViTs, YOLO, Mask R-CNN) Strong knowledge in computer vision fundamentals, including feature extraction and multi-view geometry with experience in deploying and optimizing models with TensorRT, Open VINO, and cloud/edge solutions Proficient with MLOps tools (ML flow, DVC), CI/CD, and distributed training frameworks Experience in 3D vision, AR/VR, or LiDAR processing is a plus Nice to Have: Experience with multi-camera vision systems, LiDAR, sensor fusion, and reinforcement learning for vision tasks Exposure to generative AI models (eg, Stable Diffusion, GANs) and large-scale image processing (Apache Spark, Dask) Research publications or patents in computer vision and deep learning Location-Delhi NCR,Bangalore,Chennai,Pune,Kolkata,Ahmedabad,Mumbai,Hyderabad

Lead Analyst/Senior Software Engineer - Data Engineer with Python, Apache Spark, HDFS Job Overview : CGI is looking for a talented and motivated Data Engineer with strong expertise in Python, Apache Spark, HDFS, and MongoDB to build and manage scalable, efficient, and reliable data pipelines and infrastructure Youll play a key role in transforming raw data into actionable insights, working closely with data scientists, analysts, and business teams. Key Responsibilities: Design, develop, and maintain scalable data pipelines using Python and Spark. Ingest, process, and transform large datasets from various sources into usable formats. Manage and optimize data storage using HDFS and MongoDB. Ensure high availability and performance of data infrastructure. Implement data quality checks, validations, and monitoring processes. Collaborate with cross-functional teams to understand data needs and deliver solutions. Write reusable and maintainable code with strong documentation practices. Optimize performance of data workflows and troubleshoot bottlenecks. Maintain data governance, privacy, and security best practices. Required qualifications to be successful in this role: Minimum 6 years of experience as a Data Engineer or similar role. Strong proficiency in Python for data manipulation and pipeline development. Hands-on experience with Apache Spark for large-scale data processing. Experience with HDFS and distributed data storage systems. Strong understanding of data architecture, data modeling, and performance tuning. Familiarity with version control tools like Git. Experience with workflow orchestration tools (e.g., Airflow, Luigi) is a plus. Knowledge of cloud services (AWS, GCP, or Azure) is preferred. Bachelors or Masters degree in Computer Science, Information Systems, or a related field. Preferred Skills: Experience with containerization (Docker, Kubernetes). Knowledge of real-time data streaming tools like Kafka. Familiarity with data visualization tools (e.g., Power BI, Tableau). Exposure to Agile/Scrum methodologies. Skills: Hadoop Hive Python SQL English Note This role will require- 8 weeks of in-office work after joining, after which we will transition to a hybrid working model, with 2 days per week in the office. Mode of interview F2F Time : Registration Window -9am to 12.30 pm. Candidates who are shortlisted will be required to stay throughout the day for subsequent rounds of interviews Notice Period: 0-45 Days

Roles & Responsibilities: Data Engineering Leadership & Strategy: Lead and mentor a team of data engineers, fostering a culture of technical excellence and collaboration. Define and implement data engineering best practices, standards, and processes. Data Pipeline Architecture & Development: Design, build, and maintain scalable, robust, and efficient data pipelines for ingestion, transformation, and loading of data from various sources. Optimize data pipelines for performance, reliability, and cost-effectiveness. Implement data quality checks and monitoring systems to ensure data integrity. Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions. Cloud-Based Data Infrastructure: Design, implement, and manage cloud-based data infrastructure using platforms like AWS, Azure, or GCP. Leverage cloud services (e.g., data lakes, data warehouses, serverless computing) to build scalable and cost-effective data solutions. Leverage opensource airbyte , mage ai and similar Ensure data security, governance, and compliance within the cloud environment. Data Modeling & Warehousing: Design and implement data models to support business intelligence, reporting, and analytics. Optimize data warehouse performance for efficient querying and reporting. Collaboration & Communication: Collaborate effectively with cross-functional teams including product managers, software engineers, and business stakeholders. Requirements: Bachelor's or master's degree in computer science, Engineering, or a related field. 8+ years of proven experience in data engineering, with at least 3+ years in a lead role. Expertise in building and maintaining data pipelines using tools such as Apache Spark, Apache Kafka, Apache Beam, or similar. Proficiency in SQL and one or more programming languages like Python, Java, or Scala. Hands-on experience with cloud-based data platforms (AWS, Azure, GCP) and services. Locations : Mumbai, Delhi NCR, Bengaluru , Kolkata, Chennai, Hyderabad, Ahmedabad, Pune, Remote Work Timings: 2.30 pm - 11.30 pm IST

Key Responsibilities: Backend Service Development: Design and implement robust, scalable, and maintainable backend services using Python. Utilize appropriate frameworks and libraries to streamline development and enhance productivity. Integrate AI models and algorithms into backend services, ensuring efficient and reliable communication. AI Model Integration: Collaborate with data scientists and AI engineers to understand AI model requirements and specifications. Develop APIs and interfaces to facilitate seamless integration of AI models into backend services. Cloud Infrastructure Management: Deploy and manage backend services on cloud platforms (e.g., AWS, Azure & GCP). Leverage cloud-native technologies and services to optimize infrastructure costs and performance. Ensure the security and compliance of cloud infrastructure. Collaboration and Mentorship: Work collaboratively with a cross-functional team of engineers, data scientists, and project stakeholders. Provide technical guidance and mentorship to junior engineers. Qualifications and Skills: Bachelor's or Master's degree in Computer Science, Engineering, or a related field. 5+ years of experience in Python programming, with a focus on backend development. Strong understanding of object-oriented programming (OOP) principles and design patterns. Experience with Python web frameworks (e.g., Django, Flask) and RESTful API development. Proficiency in cloud technologies (e.g., AWS, Azure & GCP) and containerization (e.g., Docker & Kubernetes). Familiarity with AI principles, machine learning algorithms, and deep learning frameworks (e.g., TensorFlow, PyTorch). Preferred Qualifications: Experience with large-scale distributed systems and microservices architectures. Knowledge of data engineering principles and big data technologies (e.g., Apache Spark). Experience : 5+Years Job Location : Remote, Delhi NCR, Bangalore, Chennai, Pune, Kolkata, Ahmedabad, Mumbai, Hyderabad Work Timings 2.30 pm -11.30 pm

Working Model : Our flexible work arrangement combines both remote and in-office work, optimizing flexibility and productivity. This position will be part of Sapiens Digital (Data Suite) division, for more information about it, click here: Designation: Senior Developer Must Skills 4-5 Years Experience in Data Bricks, PySpark, SQL, Data warehousing Criterias Job Requirements General Job Description A seasoned, experienced professional with a full understanding of area of specialization; resolves a wide range of issues in creative ways. This job is the fully qualified, career-oriented, journey-level position. Pre - requisites Knowledge & Experience B.E (or equivalent) Extensive hands-on experience in Java development, including strong knowledge of core Java concepts, data structures, and algorithms. In-depth understanding of distributed data processing frameworks like Apache Spark , with specific expertise in Databricks . Proficiency in designing and building data pipelines for data extraction, transformation, and loading (ETL). Familiarity with big data technologies and concepts, including Hadoop, Hive, and HDFS. Proven experience in building scalable and high-performance data solutions for large datasets. Solid understanding of data modelling, database design, and data warehousing concepts. Knowledge of both SQL and NoSQL databases , and ability to choose the right database type based on project requirements. Demonstrated ability to write clean, maintainable, and efficient Java code for data processing and integration tasks. Experience with Java libraries commonly used in data engineering, such as Apache Kafka for streaming data. Extensive hands-on experience with Databricks for big data processing and analytics. Ability to set up and configure Databricks clusters and optimize their performance. Proficiency in Spark Data Frame and Spark SQL for data manipulation and querying. Understanding of data architecture principles and experience in designing data solutions that meet scalability and reliability requirements. Familiarity with cloud-based data platforms like AWS or Azure. Problem-Solving and Analytical Skills: Strong problem-solving skills and the ability to analyse complex data-related issues. Capacity to propose innovative and efficient solutions to data engineering challenges. Excellent communication skills, both verbal and written, with the ability to convey technical concepts to non-technical stakeholders effectively. Experience working collaboratively in cross-functional teams, including Data Scientists, Data Analysts, and business stakeholders. A strong inclination to stay updated with the latest advancements in data engineering, Java, and Databricks technologies. Adaptability to new tools and technologies to support evolving data requirements. Required Product/project Knowledge Ability to work in an agile development environment. Hand on experience in technical design document preparation Proven experience in fine tuning and identifying the potential bottle necks on the applications Required Skills Ability to work on tasks (POCs, Stories, CR's, Defects etc.) without taking much help. Technical ability includes Programming, Debugging and Logical skills. Ability to technically guide juniors in completion of POC, Stories, CR's, Defects etc Common Tasks Come up and follow process for: Technical compliance and documentation Code review Unit & Functional testing Deployment Ensures that the team is also following the process properly. Able to write at least two technical paper or present one tech talk in a year 100% Compliance to Sprint Plan. Required Soft Skills Providing technical leadership and mentoring to junior developers Collaboration and teamwork skills Self-motivated with strong initiative and excellent Communication Skills Abilities of becoming a technical activity leader Proactive and initiative approach Self-motivated, flexible and a team player Have good understanding of the requirements in the area of functionality being developed

We are looking for a Senior GCP Data Engineer / GCP Technical Lead with strong expertise in Google Cloud Platform (GCP), Apache Spark, and Python to join our growing data engineering team. The ideal candidate will have extensive experience working with GCP data services and should be capable of leading technical teams, designing robust data pipelines, and interacting directly with clients to gather requirements and ensure project delivery. Project Duration : 1 year and extendable Role & responsibilities Design, develop, and deploy scalable data pipelines and solutions using GCP services like DataProc and BigQuery. Lead and mentor a team of data engineers to ensure high-quality deliverables. Collaborate with cross-functional teams and client stakeholders to define technical requirements and deliver solutions aligned with business goals. Optimize data processing and transformation workflows for performance and cost-efficiency. Ensure adherence to best practices in cloud data architecture, data security, and governance. Mandatory Skills: Google Cloud Platform (GCP) especially DataProc and BigQuery Apache Spark Python Programming Preferred Skills: Experience in working with large-scale data processing frameworks. Exposure to DevOps/CI-CD practices in a cloud environment. Hands-on experience with other GCP tools like Cloud Composer, Pub/Sub, or Cloud Storage is a plus. Soft Skills: Strong communication and client interaction skills. Ability to work independently and as part of a distributed team. Excellent problem-solving and team management capabilities.

Hands on experience with Snowflake and Python a must. Hands on experience with Apache Spark a must. Hands on experience with DBT preferred. Experience with performance tuning SQL queries, Spark job, and stored procedures. An understanding of E-R data models (conceptual, logical, and physical).

Role & responsibilities Designing and building optimized data pipelines using cutting-edge technologies in a cloud environment to drive analytical insights. Constructing infrastructure for efficient ETL processes from various sources and storage systems. Leading the implementation of algorithms and prototypes to transform raw data into useful information. Architecting, designing, and maintaining database pipeline architectures, ensuring readiness for AI/ML transformations. Creating innovative data validation methods and data analysis tools. Ensuring compliance with data governance and security policies. Interpreting data trends and patterns to establish operational alerts. Developing analytical tools, programs, and reporting mechanisms. Conducting complex data analysis and presenting results effectively. Preparing data for prescriptive and predictive modeling. Continuously exploring opportunities to enhance data quality and reliability. Applying strong programming and problem-solving skills to develop scalable solutions.

Role: Performance Testing Specialist Databricks Pipelines Job Seniority: Advanced (4-6 years) OR Experienced (3-4 years) Location: Magarpatta City,Pune Unit: Amdocs Data and Intelligence Mandatory SKills: All Skills must be in the resume in the roles and responsibilities Strong understanding of Databricks, Apache Spark, and performance tuning techniques for distributed data processing systems. Hands-on experience in Spark (PySpark/Scala) performance profiling, partitioning strategies, and job parallelization. 2+ year s of experience in performance testing and load simulation of data pipelines. Solid skills in SQL, Snowflake, and analyzing performance via query plans and optimization hints. Familiarity with Azure Databricks, Azure Monitor, Log Analytics, or similar observability tools. Proficient in scripting (Python/Shell) for test automation and pipeline instrumentation. Experience with DevOps tools such as Azure DevOps, GitHub Actions, or Jenkins for automated testing. Comfortable working in Unix/Linux environments and writing shell scripts for monitoring and debugging. Notice Period: Only Serving NP candidate who can join in the month of June ( 15 days to Immediate) Excellent Communication SKills This is C2H role. Interested Candidate Share Resume at dipti.bhaisare@in.experis.com

We at Acxiom Technologies are hiring for Pyspark Developer for Mumbai Location Relevant Experience : 1 to 4 Years Location : Mumbai Mode of Work : Work From Office Notice Period : Upto 20 days. Job Description: Proven experience as a Pyspark Developer . Hands-on expertise with AWS Redshift . Strong proficiency in Pyspark , Spark , Python , and Hive . Solid experience with SQL . Excellent communication skills. Benefits of working at Acxiom: - Statutory Benefits - Paid Leaves - Phenomenal Career Growth - Exposure to Banking Domain About Acxiom Technologies: Acxiom Technologies is a leading software solutions services company that provides consulting services to global firms and has established itself as one of the most sought-after consulting organizations in the field of Data Management and Business Intelligence. Also here is our website address https://www.acxtech.co.in/ to give you a detailed overview of our company. Interested Candidates can share their resumes on 7977418669 Thank you.

JD: • Good experience in Apache Iceberg, Apache Spark, Trino • Proficiency in SQL and data modeling • Experience with open Data Lakehouse using Apache Iceberg • Experience with Data Lakehouse architecture with Apache Iceberg and Trino

Job Description About Oracle APAC ISV Business Oracle APAC ISV team is one of the fastest-growing and high-performing business units in APAC. We are a prime team that operates to serve a broad range of customers across the APAC region. ISVs are at the forefront of today's fastest-growing industries. Much of this growth stems from enterprises shifting toward adopting cloud-native ISV SaaS solutions. This transformation drives ISVs to evolve from traditional software vendors to SaaS service providers. Industry analysts predict exponential growth in the ISV market over the coming years, making it a key growth pillar for every hyperscaler. Our Cloud engineering team works on pitch-to-production scenarios of bringing ISVs solutions on the Oracle cloud (#oci) with an aim to provide a cloud platform for running their business which is better performant, more flexible, more secure, compliant to open-source technologies and offers multiple innovation options yet being most cost effective. The team walks along the path with our customers and are being regarded as a trusted techno-business advisors by them. Required Skills/Experience Your versatility and hands-on expertise will be your greatest asset as you deliver on time bound implementation work items and empower our customers to harness the full power of OCI. We also look for: Bachelor's degree in Computer Science, Information Technology, or a related field. Relevant certifications in AI Services on OCI and/or other cloud platforms (AWS, Azure, Google Cloud) 8+ years of professional work experience Proven experience with end-to-end AI solution implementation, from data integration to model deployment and optimization. Experience in design, build, and deployment of end-to-end AI solutions with a focus on LLMs and RAG workflows. Proficiency in frameworks such as TensorFlow, PyTorch, scikit-learn, Keras and programming languages such as Python, R, or SQL.Experience with data wrangling, data pipelines, and data integration tools. Hands-on experience with LLM frameworks and plugins, such as LangChain, LlamaIndex, VectorStores and Retrievers, LLM Cache, LLMOps (MLFlow), LMQL, Guidance, etc. Knowledge of containerization technologies such as Docker and orchestration tools like Kubernetes to scale AI models. Expertise in analytics platforms like Power BI, Tableau, or other business intelligence tools. Experience working with cloud platforms, particularly for AI and analytics workloads. Familiarity with cloud-based AI services like OCI AI, AWS SageMaker etc Experience with building and optimizing data pipelines for large-scale AI/ML applications using tools like Apache Kafka, Apache Spark, Apache Airflow, or similar. Excellent communication skills, with the ability to clearly explain complex AI and analytics concepts to non-technical stakeholders. Proven ability to work with diverse teams and manage client expectations Solid experience managing multiple implementation projects simultaneously while maintaining high-quality standards. Ability to develop and manage project timelines, resources, and budgets. Career Level - IC4 Responsibilities What Youll Do As a solution specialist, you will work closely with our cloud architects and key stakeholders of ISVs to propagate awareness and drive implementation of OCI native as well as open-source cloud-native technologies by ISV customers. Design, implement, and optimize AI and analytics solutions using OCI AI & Analytics Services that enable advanced analytics and AI use cases. Assist clients to architect & deploy AI systems that integrate seamlessly with existing client infrastructure, ensuring scalability, performance, and security. Support the deployment of machine learning models, including model training, testing, and fine-tuning. Ensure scalability, robustness, and performance of AI models in production environments. Design, build, and deploy end-to-end AI solutions with a focus on LLMs and Agentic AI workflows (including Proactive, Reactive, RAG etc.). Help customer migrate from other Cloud vendors AI platform or bring their own AI/ML models leveraging OCI AI services and Data Science platform. Design, propose and implement solution on OCI that helps customers move seamlessly when adopting OCI for their AI requirements Provides direction and specialist knowledge to clients in developing AI chatbots using ODA (Oracle digital Assistance), OIC (Oracle integration cloud) and OCI GenAI services. Configure, integrate, and customize analytics platforms and dashboards on OCI. Implement data pipelines and ensure seamless integration with existing IT infrastructure. Drive discussions on OCI GenAI and AI Platform across the region and accelerate implementation of OCI AI services into Production