Home
Jobs

3895 Pyspark Jobs - Page 37

Filter Interviews
Min: 0 years
Max: 25 years
Min: ₹0
Max: ₹10000000
Setup a job Alert
Filter
JobPe aggregates results for easy application access, but you actually apply on the job portal directly.

3.0 - 6.0 years

0 Lacs

Gurugram, Haryana, India

On-site

Linkedin logo

Summary Position Summary Strategy & Analytics AI & Data In this age of disruption, organizations need to navigate the future with confidence, embracing decision making with clear, data-driven choices that deliver enterprise value in a dynamic business environment. The AI & Data team leverages the power of data, analytics, robotics, science and cognitive technologies to uncover hidden relationships from vast troves of data, generate insights, and inform decision-making. Together with the Strategy practice, our Strategy & Analytics portfolio helps clients transform their business by architecting organizational intelligence programs and differentiated strategies to win in their chosen markets. AI & Data will work with our clients to: Implement large-scale data ecosystems including data management, governance and the integration of structured and unstructured data to generate insights leveraging cloud-based platforms Leverage automation, cognitive and science-based techniques to manage data, predict scenarios and prescribe actions Drive operational efficiency by maintaining their data ecosystems, sourcing analytics expertise and providing As-a-Service offerings for continuous insights and improvements PySpark Consultant The position is suited for individuals who have demonstrated ability to work effectively in a fast paced, high volume, deadline driven environment. Education And Experience Education: B.Tech/M.Tech/MCA/MS 3-6 years of experience in design and implementation of migrating an Enterprise legacy system to Big Data Ecosystem for Data Warehousing project. Required Skills Must have excellent knowledge in Apache Spark and Python programming experience Deep technical understanding of distributed computing and broader awareness of different Spark version Strong UNIX operating system concepts and shell scripting knowledge Hands-on experience using Spark & Python Deep experience in developing data processing tasks using PySpark such as reading data from external sources, merge data, perform data enrichment and load in to target data destinations. Experience in deployment and operationalizing the code, knowledge of scheduling tools like Airflow, Control-M etc. is preferred Working experience on AWS ecosystem, Google Cloud, BigQuery etc. is an added advantage Hands on experience with AWS S3 Filesystem operations Good knowledge of Hadoop, Hive and Cloudera/ Hortonworks Data Platform Should have exposure with Jenkins or equivalent CICD tool & Git repository Experience handling CDC operations for huge volume of data Should understand and have operating experience with Agile delivery model Should have experience in Spark related performance tuning Should be well versed with understanding of design documents like HLD, TDD etc Should be well versed with Data historical load and overall Framework concepts Should have participated in different kinds of testing like Unit Testing, System Testing, User Acceptance Testing, etc Preferred Skills Exposure to PySpark, Cloudera/ Hortonworks, Hadoop and Hive. Exposure to AWS S3/EC2 and Apache Airflow Participation in client interactions/meetings is desirable. Participation in code-tuning is desirable. Recruiting tips From developing a stand out resume to putting your best foot forward in the interview, we want you to feel prepared and confident as you explore opportunities at Deloitte. Check out recruiting tips from Deloitte recruiters. Benefits At Deloitte, we know that great people make a great organization. We value our people and offer employees a broad range of benefits. Learn more about what working at Deloitte can mean for you. Our people and culture Our inclusive culture empowers our people to be who they are, contribute their unique perspectives, and make a difference individually and collectively. It enables us to leverage different ideas and perspectives, and bring more creativity and innovation to help solve our clients' most complex challenges. This makes Deloitte one of the most rewarding places to work. Our purpose Deloitte’s purpose is to make an impact that matters for our people, clients, and communities. At Deloitte, purpose is synonymous with how we work every day. It defines who we are. Our purpose comes through in our work with clients that enables impact and value in their organizations, as well as through our own investments, commitments, and actions across areas that help drive positive outcomes for our communities. Professional development From entry-level employees to senior leaders, we believe there’s always room to learn. We offer opportunities to build new skills, take on leadership opportunities and connect and grow through mentorship. From on-the-job learning experiences to formal development programs, our professionals have a variety of opportunities to continue to grow throughout their career. Requisition code: 300028 Show more Show less

Posted 1 week ago

Apply

0 years

0 Lacs

Hyderabad, Telangana, India

On-site

Linkedin logo

Job description: Skills: Python, Pyspark and Palantir Need a strong hands on lead engineer. Onsite in Hyderabad. Tasks and Responsibilities: Leads data engineering activities on moderate to complex data and analytics-centric problems which have broad impact and require in-depth analysis to obtain desired results; assemble, enhance, maintain, and optimize current, enable cost savings and meet individual project or enterprise maturity objectives. advanced working knowledge of SQL, Python, and PySpark Experience using tools such as: Git/Bitbucket, Jenkins/CodeBuild, CodePipeline Experience with platform monitoring and alerts tools Work closely with Subject Matter Experts (SMEs) to design and develop Foundry front end applications with the ontology (data model) and data pipelines supporting the applications Implement data transformations to derive new datasets or create Foundry Ontology Objects necessary for business applications Implement operational applications using Foundry Tools (Workshop, Map, and/or Slate) Actively participate in agile/scrum ceremonies (stand ups, planning, retrospectives, etc.) Create and maintain documentation describing data catalog and data objects Maintain applications as usage grows and requirements change Promote a continuous improvement mindset by engaging in after action reviews and sharing learnings Use communication skills, especially for explaining technical concepts to nontechnical business leader Show more Show less

Posted 1 week ago

Apply

6.0 - 9.0 years

0 Lacs

Greater Kolkata Area

On-site

Linkedin logo

Summary Position Summary Strategy & Analytics AI & Data In this age of disruption, organizations need to navigate the future with confidence, embracing decision making with clear, data-driven choices that deliver enterprise value in a dynamic business environment. The AI & Data team leverages the power of data, analytics, robotics, science and cognitive technologies to uncover hidden relationships from vast troves of data, generate insights, and inform decision-making. Together with the Strategy practice, our Strategy & Analytics portfolio helps clients transform their business by architecting organizational intelligence programs and differentiated strategies to win in their chosen markets. AI & Data will work with our clients to: Implement large-scale data ecosystems including data management, governance and the integration of structured and unstructured data to generate insights leveraging cloud-based platforms Leverage automation, cognitive and science-based techniques to manage data, predict scenarios and prescribe actions Drive operational efficiency by maintaining their data ecosystems, sourcing analytics expertise and providing As-a-Service offerings for continuous insights and improvements PySpark Sr. Consultant The position is suited for individuals who have demonstrated ability to work effectively in a fast paced, high volume, deadline driven environment. Education And Experience Education: B.Tech/M.Tech/MCA/MS 6-9 years of experience in design and implementation of migrating an Enterprise legacy system to Big Data Ecosystem for Data Warehousing project. Required Skills Must have excellent knowledge in Apache Spark and Python programming experience Deep technical understanding of distributed computing and broader awareness of different Spark version Strong UNIX operating system concepts and shell scripting knowledge Hands-on experience using Spark & Python Deep experience in developing data processing tasks using PySpark such as reading data from external sources, merge data, perform data enrichment and load in to target data destinations. Experience in deployment and operationalizing the code, knowledge of scheduling tools like Airflow, Control-M etc. is preferred Working experience on AWS ecosystem, Google Cloud, BigQuery etc. is an added advantage Hands on experience with AWS S3 Filesystem operations Good knowledge of Hadoop, Hive and Cloudera/ Hortonworks Data Platform Should have exposure with Jenkins or equivalent CICD tool & Git repository Experience handling CDC operations for huge volume of data Should understand and have operating experience with Agile delivery model Should have experience in Spark related performance tuning Should be well versed with understanding of design documents like HLD, TDD etc Should be well versed with Data historical load and overall Framework concepts Should have participated in different kinds of testing like Unit Testing, System Testing, User Acceptance Testing, etc Preferred Skills Exposure to PySpark, Cloudera/ Hortonworks, Hadoop and Hive. Exposure to AWS S3/EC2 and Apache Airflow Participation in client interactions/meetings is desirable. Participation in code-tuning is desirable. Recruiting tips From developing a stand out resume to putting your best foot forward in the interview, we want you to feel prepared and confident as you explore opportunities at Deloitte. Check out recruiting tips from Deloitte recruiters. Benefits At Deloitte, we know that great people make a great organization. We value our people and offer employees a broad range of benefits. Learn more about what working at Deloitte can mean for you. Our people and culture Our inclusive culture empowers our people to be who they are, contribute their unique perspectives, and make a difference individually and collectively. It enables us to leverage different ideas and perspectives, and bring more creativity and innovation to help solve our clients' most complex challenges. This makes Deloitte one of the most rewarding places to work. Our purpose Deloitte’s purpose is to make an impact that matters for our people, clients, and communities. At Deloitte, purpose is synonymous with how we work every day. It defines who we are. Our purpose comes through in our work with clients that enables impact and value in their organizations, as well as through our own investments, commitments, and actions across areas that help drive positive outcomes for our communities. Professional development From entry-level employees to senior leaders, we believe there’s always room to learn. We offer opportunities to build new skills, take on leadership opportunities and connect and grow through mentorship. From on-the-job learning experiences to formal development programs, our professionals have a variety of opportunities to continue to grow throughout their career. Requisition code: 300041 Show more Show less

Posted 1 week ago

Apply

3.0 - 6.0 years

0 Lacs

Greater Kolkata Area

On-site

Linkedin logo

Summary Position Summary Strategy & Analytics AI & Data In this age of disruption, organizations need to navigate the future with confidence, embracing decision making with clear, data-driven choices that deliver enterprise value in a dynamic business environment. The AI & Data team leverages the power of data, analytics, robotics, science and cognitive technologies to uncover hidden relationships from vast troves of data, generate insights, and inform decision-making. Together with the Strategy practice, our Strategy & Analytics portfolio helps clients transform their business by architecting organizational intelligence programs and differentiated strategies to win in their chosen markets. AI & Data will work with our clients to: Implement large-scale data ecosystems including data management, governance and the integration of structured and unstructured data to generate insights leveraging cloud-based platforms Leverage automation, cognitive and science-based techniques to manage data, predict scenarios and prescribe actions Drive operational efficiency by maintaining their data ecosystems, sourcing analytics expertise and providing As-a-Service offerings for continuous insights and improvements PySpark Consultant The position is suited for individuals who have demonstrated ability to work effectively in a fast paced, high volume, deadline driven environment. Education And Experience Education: B.Tech/M.Tech/MCA/MS 3-6 years of experience in design and implementation of migrating an Enterprise legacy system to Big Data Ecosystem for Data Warehousing project. Required Skills Must have excellent knowledge in Apache Spark and Python programming experience Deep technical understanding of distributed computing and broader awareness of different Spark version Strong UNIX operating system concepts and shell scripting knowledge Hands-on experience using Spark & Python Deep experience in developing data processing tasks using PySpark such as reading data from external sources, merge data, perform data enrichment and load in to target data destinations. Experience in deployment and operationalizing the code, knowledge of scheduling tools like Airflow, Control-M etc. is preferred Working experience on AWS ecosystem, Google Cloud, BigQuery etc. is an added advantage Hands on experience with AWS S3 Filesystem operations Good knowledge of Hadoop, Hive and Cloudera/ Hortonworks Data Platform Should have exposure with Jenkins or equivalent CICD tool & Git repository Experience handling CDC operations for huge volume of data Should understand and have operating experience with Agile delivery model Should have experience in Spark related performance tuning Should be well versed with understanding of design documents like HLD, TDD etc Should be well versed with Data historical load and overall Framework concepts Should have participated in different kinds of testing like Unit Testing, System Testing, User Acceptance Testing, etc Preferred Skills Exposure to PySpark, Cloudera/ Hortonworks, Hadoop and Hive. Exposure to AWS S3/EC2 and Apache Airflow Participation in client interactions/meetings is desirable. Participation in code-tuning is desirable. Recruiting tips From developing a stand out resume to putting your best foot forward in the interview, we want you to feel prepared and confident as you explore opportunities at Deloitte. Check out recruiting tips from Deloitte recruiters. Benefits At Deloitte, we know that great people make a great organization. We value our people and offer employees a broad range of benefits. Learn more about what working at Deloitte can mean for you. Our people and culture Our inclusive culture empowers our people to be who they are, contribute their unique perspectives, and make a difference individually and collectively. It enables us to leverage different ideas and perspectives, and bring more creativity and innovation to help solve our clients' most complex challenges. This makes Deloitte one of the most rewarding places to work. Our purpose Deloitte’s purpose is to make an impact that matters for our people, clients, and communities. At Deloitte, purpose is synonymous with how we work every day. It defines who we are. Our purpose comes through in our work with clients that enables impact and value in their organizations, as well as through our own investments, commitments, and actions across areas that help drive positive outcomes for our communities. Professional development From entry-level employees to senior leaders, we believe there’s always room to learn. We offer opportunities to build new skills, take on leadership opportunities and connect and grow through mentorship. From on-the-job learning experiences to formal development programs, our professionals have a variety of opportunities to continue to grow throughout their career. Requisition code: 300028 Show more Show less

Posted 1 week ago

Apply

5.0 - 10.0 years

15 - 30 Lacs

Vadodara

Remote

Naukri logo

We are seeking an experienced Senior Data Engineer to join our team. The ideal candidate will have a strong background in data engineering and AWS infrastructure, with hands-on experience in building and maintaining data pipelines and the necessary infrastructure components. The role will involve using a mix of data engineering tools and AWS services to design, build, and optimize data architecture. Key Responsibilities: Design, develop, and maintain data pipelines using Airflow and AWS services. Implement and manage data warehousing solutions with Databricks and PostgreSQL. Automate tasks using GIT / Jenkins. Develop and optimize ETL processes, leveraging AWS services like S3, Lambda, AppFlow, and DMS. Create and maintain visual dashboards and reports using Looker. Collaborate with cross-functional teams to ensure smooth integration of infrastructure components. Ensure the scalability, reliability, and performance of data platforms. Work with Jenkins for infrastructure automation. Technical and functional areas of expertise: Working as a senior individual contributor on a data intensive project Strong experience in building high performance, resilient & secure data processing pipelines preferably using Python based stack. Extensive experience in building data intensive applications with a deep understanding of querying and modeling with relational databases preferably on time-series data. Intermediate proficiency in AWS services (S3, Airflow) Proficiency in Python and PySpark Proficiency with ThoughtSpot or Databricks. Intermediate proficiency in database scripting (SQL) Basic experience with Jenkins for task automation Nice to Have : Intermediate proficiency in data analytics tools (Power BI / Tableau / Looker / ThoughSpot) Experience working with AWS Lambda, Glue, AppFlow, and other AWS transfer services. Exposure to PySpark and data automation tools like Jenkins or CircleCI. Familiarity with Terraform for infrastructure-as-code. Experience in data quality testing to ensure the accuracy and reliability of data pipelines. Proven experience working directly with U.S. client stakeholders. Ability to work independently and take the lead on tasks. Education and experience: Bachelors or masters in computer science or related fields. 5+ years of experience Stack/Skills needed: Databricks PostgreSQL Python & Pyspark AWS Stack Power BI / Tableau / Looker / ThoughSpot Familiarity with GIT and/or CI/CD tools

Posted 1 week ago

Apply

0 years

0 Lacs

Chennai, Tamil Nadu, India

On-site

Linkedin logo

Exp :6yrs + Location : Chennai/Hyderabad Skillset: - SQL, Python, Pyspark, Databricks (strong in core databricks), AWS (AWS is mandate Show more Show less

Posted 1 week ago

Apply

4.0 - 9.0 years

20 - 35 Lacs

Mumbai, Navi Mumbai, Pune

Work from Office

Naukri logo

Job Summary: We are looking for a highly skilled Data Scientist with deep expertise in time series forecasting, particularly in demand forecasting and customer lifecycle analytics (CLV). The ideal candidate will be proficient in Python or PySpark, have hands-on experience with tools like Prophet and ARIMA, and be comfortable working in Databricks environments. Familiarity with classic ML models and optimization techniques is a plus. Key Responsibilities • Develop, deploy, and maintain time series forecasting models (Prophet, ARIMA, etc.) for demand forecasting and customer behavior modeling. • Design and implement Customer Lifetime Value (CLV) models to drive customer retention and engagement strategies. • Process and analyze large datasets using PySpark or Python (Pandas). • Partner with cross-functional teams to identify business needs and translate them into data science solutions. • Leverage classic ML techniques (classification, regression) and boosting algorithms (e.g., XGBoost, LightGBM) to support broader analytics use cases. • Use Databricks for collaborative development, data pipelines, and model orchestration. • Apply optimization techniques where relevant to improve forecast accuracy and business decision-making. • Present actionable insights and communicate model results effectively to technical and non-technical stakeholders. Required Qualifications • Strong experience in Time Series Forecasting, with hands-on knowledge of Prophet, ARIMA, or equivalent Mandatory. • Proven track record in Demand Forecasting Highly Preferred. • Experience in modeling Customer Lifecycle Value (CLV) or similar customer analytics use cases – Highly Preferred. • Proficiency in Python (Pandas) or PySpark – Mandatory. • Experience with Databricks – Mandatory. • Solid foundation in statistics, predictive modeling, and machine learning Locations: Mumbai/Pune/Noida/Bangalore/Jaipur/Hyderabad

Posted 1 week ago

Apply

7.0 years

0 Lacs

Gurugram, Haryana, India

On-site

Linkedin logo

Job Summary: We are seeking a skilled Data Engineer to join our dynamic team. In this role, will be responsible for implementing and maintaining scalable data pipelines and infrastructure on AWS cloud platform. The ideal candidate will have experience with AWS services, particularly in the realm of big data processing and analytics. The role involves working closely with cross-functional teams to support data-driven decision-making and focus on delivering business objectives while improving efficiency and ensuring high service quality. Key Responsibilities: Design, develop, and maintain large-scale data pipelines that can handle large datasets from multiple sources. Knowledge of real-time data replication and batch processing of data using distributed computing platforms like Spark, Kafka, etc. Optimize performance of data processing jobs and ensure system scalability and reliability. Collaborate with DevOps teams to manage infrastructure, including cloud environments like AWS Collaborate with data scientists, analysts, and business stakeholders to develop tools and platforms that enable advanced analytics and reporting. Lead and mentor junior data engineers, providing guidance on best practices, code reviews, and technical solutions. Evaluating and implementing new frameworks, tools for data engineering Strong analytical and problem-solving skills with attention to detail. To maintain a healthy working relationship with the business partners/users and other MLI departments Responsible for overall performance, cost and delivery of technology solutions Key Technical competencies/skills required: Hands-on experience with AWS services such as S3, DMS, Lambda, EMR, Glue, Redshift,RDS (Postgres) Athena, Kinesics, etc. Expertise in data modelling and knowledge of modern file and table formats. Proficiency in programming languages such as Python, PySpark, SQL/PLSQL for implementing data pipelines and ETL processes. Experience data architecting or deploying Cloud/Virtualization solutions (Like Data Lake, EDW, Mart ) in enterprise Knowledge of modern data stack and keeping the technology stack refreshed. Knowledge of DevOps to perform CI/CD for data pipelines. Knowledge of Data Observability, automated data lineage and metadata management would be an added advantage. Cloud/hybrid cloud (preferable AWS) solution for data strategy for Data lake, BI and Analytics Set-up logging, monitoring, alerting, dashboards for cloud solution and data solution Experience with data warehousing concepts. Desired qualifications and experience: Bachelor’s degree in Computer Science, Engineering, or related field (Master’s preferred). Proven experience of 7+ years as a Data Engineer or similar role with a strong focus on AWS cloud Strong analytical and problem-solving skills with attention to detail. Excellent communication and collaboration skills. AWS certifications (e.g., AWS Certified Big Data - Specialty) are a plus Show more Show less

Posted 1 week ago

Apply

3.0 - 7.0 years

15 - 20 Lacs

Hyderabad, Gurugram

Work from Office

Naukri logo

Role: Hadoop Data Engineer Location: Gurgaon / Hyderabad Work Mode: Hybrid Employment Type: Full-Time Interview Mode: First Video then In Person Job Description Job Overview: We are looking for experienced Data Engineers proficient in Hadoop, Hive, Python, SQL, and Pyspark/Spark to join our dynamic team. Candidates will be responsible for designing, developing, and maintaining scalable big data solutions. Key Responsibilities: Develop and optimize data pipelines for large-scale data processing. Work with structured and unstructured datasets to derive actionable insights. Collaborate with cross-functional teams to enhance data-driven decision-making. Ensure the performance, scalability, and reliability of data architectures. Implement best practices for data security and governance.

Posted 1 week ago

Apply

8.0 years

0 Lacs

Chennai, Tamil Nadu, India

On-site

Linkedin logo

Senior Data Engineer Position Summary The Senior Data Engineer leads complex data engineering projects working on designing data architectures that align with business requirements This role focuses on optimizing data workflows managing data pipelines and ensuring the smooth operation of data systems Minimum Qualifications 8 Years overall IT experience with minimum 5 years of work experience in below tech skills Tech Skill Strong experience in Python Scripting and PySpark for data processing Proficiency in SQL dealing with big data over Informatica ETL Proven experience in Data quality and data optimization of data lake in Iceberg format with strong understanding of architecture Experience in AWS Glue jobs Experience in AWS cloud platform and its data services S3 Redshift Lambda EMR Airflow Postgres SNS Event bridge Expertise in BASH Shell scripting Strong understanding of healthcare data systems and experience leading data engineering teams Experience in Agile environments Excellent problem solving skills and attention to detail Effective communication and collaboration skills Responsibilities Leads development of data pipelines and architectures that handle large scale data sets Designs constructs and tests data architecture aligned with business requirements Provides technical leadership for data projects ensuring best practices and high quality data solutions Collaborates with product finance and other business units to ensure data pipelines meet business requirements Work with DBT Data Build Tool for transforming raw data into actionable insights Oversees development of data solutions that enable predictive and prescriptive analytics Ensures the technical quality of solutions managing data as it moves across environments Aligns data architecture to Healthfirst solution architecture Show more Show less

Posted 1 week ago

Apply

2025.0 years

0 Lacs

Chennai, Tamil Nadu, India

On-site

Linkedin logo

Greetings from TATA CONSULTANCY SERVICES! WALK-IN-DRIVE "ETL Testing" Interview Date: 14th June 2025 Years of Experience: 6+ years Location: Chennai Job Description:- Should be strong in Azure and ETL Testing(Highly Importance), SQL and good Knowledge in Data Warehousing (DWH) Concepts Able to work individually and meet the testing delivery expectation from End to End. Able to analyze the requirement, pro-actively identify the scenarios, co-ordinate with business team and get it clarified. Able to understand, convert and verify the business transformation logic into technical terms Should be willing and ready to put in additional effort to learn SAS Should be willing and ready to put in additional effort to learn Python and PySpark Thanks, Ayushi Gupta Show more Show less

Posted 1 week ago

Apply

5.0 years

5 - 8 Lacs

Hyderābād

On-site

GlassDoor logo

Category: Software Development/ Engineering Main location: India, Andhra Pradesh, Hyderabad Position ID: J0625-0219 Employment Type: Full Time Position Description: Company Profile: Founded in 1976, CGI is among the largest independent IT and business consulting services firms in the world. With 94,000 consultants and professionals across the globe, CGI delivers an end-to-end portfolio of capabilities, from strategic IT and business consulting to systems integration, managed IT and business process services and intellectual property solutions. CGI works with clients through a local relationship model complemented by a global delivery network that helps clients digitally transform their organizations and accelerate results. CGI Fiscal 2024 reported revenue is CA$14.68 billion and CGI shares are listed on the TSX (GIB.A) and the NYSE (GIB). Learn more at cgi.com. Your future duties and responsibilities: Position: Senior Software Engineer Experience: 5-10 years Category: Software Development/ Engineering Shift Timings: 1:00 pm to 10:00 pm Main location: Hyderabad Work Type: Work from office Skill: Spark (PySpark), Python and SQL Employment Type: Full Time Position ID: J0625-0219 Required qualifications to be successful in this role: Must have Skills: 5+ yrs. Development experience with Spark (PySpark), Python and SQL. Extensive knowledge building data pipelines Hands on experience with Databricks Devlopment Strong experience with Strong experience developing on Linux OS. Experience with scheduling and orchestration (e.g. Databricks Workflows,airflow, prefect, control-m). Good to have skills: Solid understanding of distributed systems, data structures, design principles. Agile Development Methodologies (e.g. SAFe, Kanban, Scrum). Comfortable communicating with teams via showcases/demos. Play key role in establishing and implementing migration patterns for the Data Lake Modernization project. Actively migrate use cases from our on premises Data Lake to Databricks on GCP. Collaborate with Product Management and business partners to understand use case requirements and reporting. Adhere to internal development best practices/lifecycle (e.g. Testing, Code Reviews, CI/CD, Documentation) . Document and showcase feature designs/workflows. Participate in team meetings and discussions around product development. Stay up to date on industry latest industry trends and design patterns. 3+ years experience with GIT. 3+ years experience with CI/CD (e.g. Azure Pipelines). Experience with streaming technologies, such as Kafka, Spark. Experience building applications on Docker and Kubernetes. Cloud experience (e.g. Azure, Google). Skills: English Python SQLite What you can expect from us: Together, as owners, let’s turn meaningful insights into action. Life at CGI is rooted in ownership, teamwork, respect and belonging. Here, you’ll reach your full potential because… You are invited to be an owner from day 1 as we work together to bring our Dream to life. That’s why we call ourselves CGI Partners rather than employees. We benefit from our collective success and actively shape our company’s strategy and direction. Your work creates value. You’ll develop innovative solutions and build relationships with teammates and clients while accessing global capabilities to scale your ideas, embrace new opportunities, and benefit from expansive industry and technology expertise. You’ll shape your career by joining a company built to grow and last. You’ll be supported by leaders who care about your health and well-being and provide you with opportunities to deepen your skills and broaden your horizons. Come join our team—one of the largest IT and business consulting services firms in the world.

Posted 1 week ago

Apply

5.0 years

6 - 9 Lacs

Hyderābād

On-site

GlassDoor logo

About the job We are seeking an experienced Data Engineering Specialist interested in challenging the status quo to ensure the seamless creation and operation of the data pipelines that are needed by Sanofi’s advanced analytic, AI and ML initiatives for the betterment of our global patients and customers. Sanofi has recently embarked into a vast and ambitious digital transformation program. A cornerstone of this roadmap is the acceleration of its data transformation and of the adoption of artificial intelligence (AI) and machine learning (ML) solutions, to accelerate R&D, manufacturing and commercial performance and bring better drugs and vaccines to patients faster, to improve health and save lives Main Responsibilities: Establish technical designs to meet Sanofi requirements aligned with the architectural and Data standards Ownership of the entire back end of the application, including the design, implementation, test, and troubleshooting of the core application logic, databases, data ingestion and transformation, data processing and orchestration of pipelines, APIs, CI/CD integration and other processes Fine-tune and optimize queries using Snowflake platform and database techniques Optimize ETL/data pipelines to balance performance, functionality, and other operational requirements. Assess and resolve data pipeline issues to ensure performance and timeliness of execution Assist with technical solution discovery to ensure technical feasibility. Assist in setting up and managing CI/CD pipelines and development of automated tests Developing and managing microservices using python Conduct peer reviews for quality, consistency, and rigor for production level solution Design application architecture for efficient concurrent user handling, ensuring optimal performance during high usage periods Own all areas of the product lifecycle: design, development, test, deployment, operation, and support About you Qualifications: 5+ years of relevant experience developing backend, integration, data pipelining, and infrastructure Expertise in database optimization and performance improvement Expertise in Python, PySpark, and Snowpark Experience data warehousing and object-relational database (Snowflake and PostgreSQL) and writing efficient SQL queries Experience in cloud-based data platforms (Snowflake, AWS) Proficiency in developing robust, reliable APIs using Python and FastAPI Framework Expert in ELT and ETL & experience working with large data sets and performance and query optimization. IICS is a plus Understanding of data structures and algorithms Understanding of DBT is a plus Experience in modern testing framework (SonarQube, K6 is a plus) Strong collaboration skills, willingness to work with others to ensure seamless integration of the server-side and client-side Knowledge of DevOps best practices and associated tools is a plus, especially in the setup, configuration, maintenance, and troubleshooting of associated tools: Containers and containerization technologies (Kubernetes, Argo, Red Hat OpenShift) Infrastructure as code (Terraform) Monitoring and Logging (CloudWatch, Grafana) CI/CD Pipelines (JFrog Artifactory) Scripting and automation (Python, GitHub, Github actions) Experience with JIRA & Confluence Workflow orchestration (Airflow) Message brokers (RabbitMQ) Education: Bachelor’s degree in computer science, engineering, or similar quantitative field of study Why choose us? Bring the miracles of science to life alongside a supportive, future-focused team. Discover endless opportunities to grow your talent and drive your career, whether it’s through a promotion or lateral move, at home or internationally. Enjoy a thoughtful, well-crafted rewards package that recognizes your contribution and amplifies your impact. Take good care of yourself and your family, with a wide range of health and wellbeing benefits including high-quality healthcare, prevention and wellness programs and at least 14 weeks’ gender-neutral parental leave. Opportunity to work in an international environment, collaborating with diverse business teams and vendors, working in a dynamic team, and fully empowered to propose and implement innovative ideas. Pursue Progress . Discover Extraordinary . Progress doesn’t happen without people – people from different backgrounds, in different locations, doing different roles, all united by one thing: a desire to make miracles happen. You can be one of those people. Chasing change, embracing new ideas and exploring all the opportunities we have to offer. Let’s pursue Progress. And let’s discover Extraordinary together. At Sanofi, we provide equal opportunities to all regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, or gender identity. Watch our ALL IN video and check out our Diversity Equity and Inclusion actions at sanofi.com! Languages: English is a must

Posted 1 week ago

Apply

4.0 years

2 - 8 Lacs

Hyderābād

On-site

GlassDoor logo

Overview: Provide data science / analytics support for the Perfect Store group who works with AMESA Sectors, a part of the broader Global Capability Center in Hyderabad, India. This role will help to enable accelerated growth for PepsiCo by building Retailer Value Offer and Shopper Value offer, aligning data, and performing advance analytics approaches for PepsiCo to drive actionable insights at Business Unit, store level. Key responsibilities will be to build and manage advance analytics-deep dives in a cloud environment, manage data and prepare data to be used for advanced analytics, artificial intelligence, machine learning, and deep learning projects. Responsibilities: Support Perfect Store (Demand Accelerator) team with delivery of Retail Value Offer, Shopper Value Offer framework for AMESA sector Work within cloud environment (e.g., Microsoft Azure) Build and maintain codes for use in advanced analytics, artificial intelligence, and machine learning projects Clean and prepare data for use in advanced analytics, artificial intelligence, and machine learning projects Build deep dive analysis reports in cloud environment (using Pyspark and Python) to support BU Ask Develop, maintain, and apply statistical techniques to business questions - including Distribution, Outliers, visualizations etc. Support relationships with the key end-user stakeholders with Business Units-AMESA Own flawless execution AND Quality Check of analytics exercises Responsible for managing multiple priorities; being able to manage deadlines and deliverables Lead communication with Business Partners and potentially end-users on matters such as available capacity, changes of scope of existing projects and planning of future projects Deliver outputs in line with the agreed timelines and formats while updating existing project management tools Flag and monitor any business risks related to delivering the requested outputs Qualifications: An experienced analytics profession with 4 years+ experience Education: B.Tech or any bachelor’s degree. Masters are optional Proficient with Python, SQL, Excel and PowerBI Plus, to have knowledge on Machine Learning algorithms Good to have Retail experience Strong collaborator: Interested and motivated by working with others. Owns the full responsibility of deliverables, quality check thoroughly, look for and work on improvements in the process Actively creates and participates in opportunities to co-create solutions across markets. Willing and able to embrace Responsive Ways of Working

Posted 1 week ago

Apply

0 years

0 Lacs

Chennai, Tamil Nadu, India

On-site

Linkedin logo

Should lead key initiatives in GenAI COE on new tech trends, R and D on new product POC Work on proposals, Provide solutions architecture design solution on different business use cases Ability to work across various different GenAI Models and cloud providers Should have a strong experience in implementing projects in Digital Engineering Or Data Analytics Technically lead a team of developers and groom talent Extensive implementation experience in data analytics space or a senior developer role in one of the modern technology stack Excellent programming skills and proficiency in at least one of the major programming scripting languages used in Gen AI orchestration such as Python or PySpark or Java Ability to build API based scalable solutions and debug & troubleshoot software or design issues Hands on exposure to integrating atleast one of the popular LLMs(Open AI GPT, PaLM 2, Dolly, Claude 2, Cohere etc.) using API endpoints. Thorough understanding of prompt engineering; implementation exposure to LLM agents like LangChain & vector databases Pinecone or Chroma or FAISS Basic data engineering skills to load structured & unstructured data from source systems to target data stores. Build and maintain data pipelines and infrastructure to support Hands on exposure to using cloud(Azure/GCP/AWS) services for storage, serverless-logic, search, transcription and chat Extensive experience with data engineering and should be hands on in using Agentic AI Framework, RAG Show more Show less

Posted 1 week ago

Apply

6.0 - 8.0 years

2 - 9 Lacs

Hyderābād

On-site

GlassDoor logo

Job description Some careers shine brighter than others. If you’re looking for a career that will help you stand out, join HSBC and fulfil your potential. Whether you want a career that could take you to the top, or simply take you in an exciting new direction, HSBC offers opportunities, support and rewards that will take you further. HSBC is one of the largest banking and financial services organisations in the world, with operations in 64 countries and territories. We aim to be where the growth is, enabling businesses to thrive and economies to prosper, and, ultimately, helping people to fulfil their hopes and realise their ambitions. We are currently seeking an experienced professional to join our team in the role of Senior Consultant Specialist In this role, you will: Design, deploy, and manage scalable applications using Kubernetes Helm-charts. Python-Spark development and should be able to Work with Agile application dev team to implement data strategies, build data flows. Troubleshoot and resolve technical issues optimizing system performance ensuring reliability. Create and maintain technical documentation for new and existing system ensuring that information is accessible to the team. Take part in process improvements in automation and CI/CD Create short-term plans to deliver environments to support sprint-based development. Implementing and monitoring solutions that identify both system bottlenecks and production issues. Creation of software deployment that allow DevOps engineers to successfully deploy sites/software in any environment. Requirements To be successful in this role, you should meet the following requirements: Minimum 6-8 years of hands-on experience of Design, deploy, and manage scalable applications using Kubernetes Helm-charts and ensure the availability, performance, and readiness of the Kubernetes infrastructure. Python-Spark development and should be able to Work with Agile application dev team to implement data strategies, build data flows. Solid understanding and hands-on experience with Kubernetes and container orchestration. Preferably IKP (Internal Kubernetes Platform) Solid experience with Docker containerization and orchestration. Understanding and experience on DevOps, automation and CI/CD Experience with monitoring tools like Splunk, Prometheus, Grafana, EFK, etc. Monitor the health of Kubernetes clusters and troubleshoot any issues. Understanding of microservice design and architectural patterns Implementing security measures and ensuring compliance with security standards within Kubernetes environments. Certifications like CKD, CKA, and CKS are desirable. Experience with Infrastructure as Code and configuration management, using tools like Terraform and Ansible is desirable. Strong communication skills and ability to document processes and configurations clearly. HSBC is committed to building a culture where all employees are valued, respected and opinions count. We take pride in providing a workplace that fosters continuous professional development, flexible working and opportunities to grow within an inclusive and diverse environment. Personal data held by the Bank relating to employment applications will be used in accordance with our Privacy Statement, which is available on our website. Issued by – HSBC Software Development India

Posted 1 week ago

Apply

0 years

0 Lacs

Chennai, Tamil Nadu, India

On-site

Linkedin logo

Extensive implementation experience in data analytics space or a senior developer role in one of the modern technology stack Excellent programming skills and proficiency in at least one of the major programming scripting languages used in Gen AI orchestration such as Python or PySpark or Java Ability to build API based scalable solutions and debug troubleshoot software or design issues Hands on exposure to integrating atleast one of the popular LLMs Open AI GPT PaLM 2 Dolly Claude 2 Cohere etc using API endpoints Thorough understanding of prompt engineering; implementation exposure to LLM agents like LangChain vector databases Pinecone or Chroma or FAISS Ability to quickly conduct experiments and analyze the features and capabilities of newer versions of the LLM models as they come into market Basic data engineering skills to load structured unstructured data from source systems to target data stores Work closely with Gen AI leads and other team members to address requirements from the product backlog Build and maintain data pipelines and infrastructure to support AI Solutions Desirable Hands on exposure to using cloud Azure GCP AWS services for storage serverless logic search transcription and chat Extensive experience with data engineering and ETL tools is a big plus Masters Bachelors degree in Computer Science or Statistics or Mathematics Show more Show less

Posted 1 week ago

Apply

4.6 years

4 - 8 Lacs

Hyderābād

On-site

GlassDoor logo

Category: Software Development/ Engineering Main location: India, Andhra Pradesh, Hyderabad Position ID: J1024-1413 Employment Type: Full Time Position Description: At CGI, we’re a team of builders. We call our employees members because all who join CGI are building their own company - one that has grown to 72,000 professionals located in 40 countries. Founded in 1976, CGI is a leading IT and business process services firm committed to helping clients succeed. We have the global resources, expertise, stability and dedicated professionals needed to achieve. At CGI, we’re a team of builders. We call our employees members because all who join CGI are building their own company - one that has grown to 72,000 professionals located in 40 countries. Founded in 1976, CGI is a leading IT and business process services firm committed to helping clients succeed. We have the global resources, expertise, stability and dedicated professionals needed to achieve results for our clients - and for our members. Come grow with us. Learn more at www.cgi.com. This is a great opportunity to join a winning team. CGI offers a competitive compensation package with opportunities for growth and professional development. Benefits for full-time, permanent members start on the first day of employment and include a paid time-off program and profit participation and stock purchase plans. We wish to thank all applicants for their interest and effort in applying for this position, however, only candidates selected for interviews will be contacted. No unsolicited agency referrals please. Job Title: Python Developer Position: Senior Software Engineer Experience: 4.6 – 6 Years Category: Software Development/ Engineering Main location: Bangalore/Hyderabad/Chennai Position ID: J1024-1413 Employment Type: Full Time Works independently under limited supervision and applies knowledge of subject matter in Applications Development. Possess sufficient knowledge and skills to effectively deal with issues, challenges within field of specialization to develop simple applications solutions. Second level professional with direct impact on results and outcome. Qualification: Bachelor's degree in Computer Science or related field or higher with minimum 4 years of relevant experience. Your future duties and responsibilities: Completes the delivery of design, code or testing for modules or multiple functions related to IS development initiatives. Prepares requirement definition, design, technical specifications. Provides coding, testing and implementation support for identified technical platform (i.e., Mainframe, Mid-range, Distributed or Web) Analyzes user requirements, and defines technical project scope and assumptions for assigned tasks. Creates business and/or technical designs for new systems, and/or modifications to existing systems. Required qualifications to be successful in this role: Position: Senior Software Engineer Experience: 4.6 - 6 Years Main location: Hyderabad and Bangalore Must-Have Skills: 4+ years of development experience with Python/AWS technologies. Hands on experience with Python, Pyspark, AWS and SQL AWS services required: S3, Lemda,Dynamo DB and etc.. Working experience in TDD and BDD frameworks. Provide technical direction on design considerations, including performance, scalability, availability, maintainability, and auditability. Strong customer facing experience. Propose and Design the solution approach to cater business requirements by building/enhancing re-usable components Working experience SAFe delivery model. Good organizational and written/verbal communication skills Good presentation skills Positive attitude and team focus are required Good-to-Have Skills: Experience with IBM Spectrum Conductor is an added advantage. Python utilities like interacting with FLASK API, Sharepoint API is an added advantage. Skills: Python What you can expect from us: Together, as owners, let’s turn meaningful insights into action. Life at CGI is rooted in ownership, teamwork, respect and belonging. Here, you’ll reach your full potential because… You are invited to be an owner from day 1 as we work together to bring our Dream to life. That’s why we call ourselves CGI Partners rather than employees. We benefit from our collective success and actively shape our company’s strategy and direction. Your work creates value. You’ll develop innovative solutions and build relationships with teammates and clients while accessing global capabilities to scale your ideas, embrace new opportunities, and benefit from expansive industry and technology expertise. You’ll shape your career by joining a company built to grow and last. You’ll be supported by leaders who care about your health and well-being and provide you with opportunities to deepen your skills and broaden your horizons. Come join our team—one of the largest IT and business consulting services firms in the world.

Posted 1 week ago

Apply

1.0 - 3.0 years

2 - 5 Lacs

Hyderabad

Work from Office

Naukri logo

What you will do In this vital role you will be responsible to design, develop, and optimize data pipelines, data integration frameworks, and metadata-driven architectures that enable seamless data access and analytics. This role prefers deep expertise in big data processing, distributed computing, data modeling, and governance frameworks to support self-service analytics, AI-driven insights, and enterprise-wide data management. Design, develop, and maintain complex ETL/ELT data pipelines in Databricks using PySpark, Scala, and SQL to process large-scale datasets Understand the biotech/pharma or related domains & build highly efficient data pipelines to migrate and deploy complex data across systems Design and Implement solutions to enable unified data access, governance, and interoperability across hybrid cloud environments Ingest and transform structured and unstructured data from databases (PostgreSQL, MySQL, SQL Server, MongoDB etc.), APIs, logs, event streams, images, pdf, and third-party platforms Ensuring data integrity, accuracy, and consistency through rigorous quality checks and monitoring Expert in data quality, data validation and verification frameworks Innovate, explore and implement new tools and technologies to enhance efficient data processing Proactively identify and implement opportunities to automate tasks and develop reusable frameworks Work in an Agile and Scaled Agile (SAFe) environment, collaborating with cross-functional teams, product owners, and Scrum Masters to deliver incremental value Use JIRA, Confluence, and Agile DevOps tools to manage sprints, backlogs, and user stories Support continuous improvement, test automation, and DevOps practices in the data engineering lifecycle Collaborate and communicate effectively with the product teams, with cross-functional teams to understand business requirements and translate them into technical solutions What we expect of you We are all different, yet we all use our unique contributions to serve patients. We are looking for highly motivated expert Data Engineer who can own the design & development of complex data pipelines, solutions and frameworks. Basic Qualifications: Masters degree and 1 to 3 years of Computer Science, IT or related field experience OR Bachelors degree and 3 to 5 years of Computer Science, IT or related field experience OR Diploma and 7 to 9 years of Computer Science, IT or related field experience Hands-on experience in data engineering technologies such as Databricks, PySpark, SparkSQL Apache Spark, AWS, Python, SQL, and Scaled Agile methodologies Proficiency in workflow orchestration, performance tuning on big data processing Strong understanding of AWS services Ability to quickly learn, adapt and apply new technologies Strong problem-solving and analytical skills Excellent communication and teamwork skills Experience with Scaled Agile Framework (SAFe), Agile delivery practices, and DevOps practices Preferred Qualifications: AWS Certified Data Engineer preferred Databricks Certificate preferred Scaled Agile SAFe certification preferred Data Engineering experience in Biotechnology or pharma industry Experience in writing APIs to make the data available to the consumers Experienced with SQL/NOSQL database, vector database for large language models Experienced with data modeling and performance tuning for both OLAP and OLTP databases Experienced with software engineering best-practices, including but not limited to version control (Git, Subversion, etc.), CI/CD (Jenkins, Maven etc.), automated unit testing, and DevOps Soft Skills: Excellent analytical and troubleshooting skills. Strong verbal and written communication skills Ability to work effectively with global, virtual teams High degree of initiative and self-motivation. Ability to manage multiple priorities successfully. Team-oriented, with a focus on achieving team goals. Ability to learn quickly, be organized and detail oriented. Strong presentation and public speaking skills.

Posted 1 week ago

Apply

2.0 years

0 Lacs

Bengaluru, Karnataka, India

On-site

Linkedin logo

Line of Service Advisory Industry/Sector Not Applicable Specialism Emerging Technologies Management Level Manager Job Description & Summary At PwC, our people in software and product innovation focus on developing cutting-edge software solutions and driving product innovation to meet the evolving needs of clients. These individuals combine technical experience with creative thinking to deliver innovative software products and solutions. In emerging technology at PwC, you will focus on exploring and implementing cutting-edge technologies to drive innovation and transformation for clients. You will work in areas such as artificial intelligence, blockchain, and the internet of things (IoT). Why PWC At PwC, you will be part of a vibrant community of solvers that leads with trust and creates distinctive outcomes for our clients and communities. This purpose-led and values-driven work, powered by technology in an environment that drives innovation, will enable you to make a tangible impact in the real world. We reward your contributions, support your wellbeing, and offer inclusive benefits, flexibility programmes and mentorship that will help you thrive in work and life. Together, we grow, learn, care, collaborate, and create a future of infinite experiences for each other. Learn more about us. At PwC, we believe in providing equal employment opportunities, without any discrimination on the grounds of gender, ethnic background, age, disability, marital status, sexual orientation, pregnancy, gender identity or expression, religion or other beliefs, perceived differences and status protected by law. We strive to create an environment where each one of our people can bring their true selves and contribute to their personal growth and the firm’s growth. To enable this, we have zero tolerance for any discrimination and harassment based on the above considerations. " Job Description & Summary: A career in our New Technologies practice, within Application and Emerging Technology services, will provide you with a unique opportunity to help our clients identify and prioritize emerging technologies that can help solve their business problems. We help clients design approaches to integrate new technologies, skills, and processes so they can drive business results and innovation. Our team helps organizations to embrace emerging technologies to remain competitive and improve their business by solving complex questions. Our team focuses on identifying and prioritizing emerging technologies, breaking into new markets, and preparing clients to get the most out of their emerging technology investments. Responsibilities Expertise with at least one Object Oriented JavaScript Framework (React, Angular, Ember, Dojo, Node, Next etc.) Atleast 2 years hands-on experience in Python (including PySpark and Flask), Java (Springboot and Springbatch), .Net Proficiency with Object Oriented Programming, multi-threading, data serialization and REST API to connect applications to back-end services. Proficiency in Docker, Kubernetes (k8s), Jenkins, and GitHub Actions is essential for this role. Proven cloud development experience AWS or AZURE or Hybrid Cloud Understanding of IT life cycle methodology & processes - SDLC/ Agile WoW Experience working with Microservices/Service Oriented Architecture Frameworks: Kubernetes, Express.js or ASP.Net core Good Understanding on Middleware technologies : Express.js or Django or ASP.Net Possess expertise in at least one unit testing framework: Junit, Selenium etc. Experience in android and iOS native is preferable Experience in Flutter or React Native is preferable Hands-on knowledge in Golang (good to have) Experience in SQL and NoSQL (MongoDB, CosmosDB, PostGre etc.) Mandatory Skill Sets Java/Python/Mern Preferred Skill Sets Express JS, Flutter, React Native, SQL and NoSQL Years Of Experience Required 10 to 12 years Education Qualification B.E/B.Tech Education (if blank, degree and/or field of study not specified) Degrees/Field of Study required: Bachelor Degree, Bachelor of Engineering Degrees/Field Of Study Preferred Certifications (if blank, certifications not specified) Required Skills Java, Python (Programming Language) Optional Skills Accepting Feedback, Accepting Feedback, Active Listening, Analytical Thinking, Artificial Intelligence, Business Planning and Simulation (BW-BPS), Coaching and Feedback, Communication, Competitive Advantage, Conducting Research, Creativity, Digital Transformation, Embracing Change, Emotional Regulation, Empathy, Implementing Technology, Inclusion, Innovation Processes, Intellectual Curiosity, Internet of Things (IoT), Learning Agility, Optimism, Product Development, Product Testing, Professional Courage {+ 14 more} Desired Languages (If blank, desired languages not specified) Travel Requirements Not Specified Available for Work Visa Sponsorship? No Government Clearance Required? No Job Posting End Date Show more Show less

Posted 1 week ago

Apply

0 years

0 Lacs

Trivandrum, Kerala, India

Remote

Linkedin logo

Description Data Engineer Responsibilities : Deliver end-to-end data and analytics capabilities, including data ingest, data transformation, data science, and data visualization in collaboration with Data and Analytics stakeholder groups Design and deploy databases and data pipelines to support analytics projects Develop scalable and fault-tolerant workflows Clearly document issues, solutions, findings and recommendations to be shared internally & externally Learn and apply tools and technologies proficiently, including: Languages: Python, PySpark, ANSI SQL, Python ML libraries Frameworks/Platform: Spark, Snowflake, Airflow, Hadoop , Kafka Cloud Computing: AWS Tools/Products: PyCharm, Jupyter, Tableau, PowerBI Performance optimization for queries and dashboards Develop and deliver clear, compelling briefings to internal and external stakeholders on findings, recommendations, and solutions Analyze client data & systems to determine whether requirements can be met Test and validate data pipelines, transformations, datasets, reports, and dashboards built by team Develop and communicate solutions architectures and present solutions to both business and technical stakeholders Provide end user support to other data engineers and analysts Candidate Requirements Expert experience in the following[Should have/Good to have]: SQL, Python, PySpark, Python ML libraries. Other programming languages (R, Scala, SAS, Java, etc.) are a plus Data and analytics technologies including SQL/NoSQL/Graph databases, ETL, and BI Knowledge of CI/CD and related tools such as Gitlab, AWS CodeCommit etc. AWS services including EMR, Glue, Athena, Batch, Lambda CloudWatch, DynamoDB, EC2, CloudFormation, IAM and EDS Exposure to Snowflake and Airflow. Solid scripting skills (e.g., bash/shell scripts, Python) Proven work experience in the following: Data streaming technologies Big Data technologies including, Hadoop, Spark, Hive, Teradata, etc. Linux command-line operations Networking knowledge (OSI network layers, TCP/IP, virtualization) Candidate should be able to lead the team, communicate with business, gather and interpret business requirements Experience with agile delivery methodologies using Jira or similar tools Experience working with remote teams AWS Solutions Architect / Developer / Data Analytics Specialty certifications, Professional certification is a plus Bachelor Degree in Computer Science relevant field, Masters Degree is a plus Show more Show less

Posted 1 week ago

Apply

3.0 years

25 Lacs

Gurgaon

On-site

GlassDoor logo

Position Title: Hadoop Data Engineer Location: Hyderabd and Gurgaon Position Type: Full-Time Required Experience: 3+ Years Job Overview: We are looking for experienced Data Engineers proficient in Hadoop, Hive, Python, SQL, and Pyspark/Spark to join our dynamic team. Candidates will be responsible for designing, developing, and maintaining scalable big data solutions. Key Responsibilities: Develop and optimize data pipelines for large-scale data processing. Work with structured and unstructured datasets to derive actionable insights. Collaborate with cross-functional teams to enhance data-driven decision-making. Ensure the performance, scalability, and reliability of data architectures. Implement best practices for data security and governance. Interview Process: L1: Virtual interview. L2: Face-to-Face interview at office. L3: Final round (Face-to-Face or Virtual). Job Types: Full-time, Permanent Pay: Up to ₹2,500,000.00 per year Benefits: Health insurance Provident Fund Location Type: In-person Schedule: Day shift Monday to Friday Morning shift Application Question(s): How many years of experience do you have in Python? How many years of experience do you have in Hive? Experience: total work: 3 years (Preferred) Hadoop: 3 years (Preferred) Work Location: In person

Posted 1 week ago

Apply

0 years

15 - 19 Lacs

Gurgaon

On-site

GlassDoor logo

NEW OPPORTUNITY || IMMEDIATE TO 45 DAYS JOINERS REQUIRED || Role: Hadoop Data Engineer Location: Gurgaon / Hyderabad Work Mode: Hybrid Employment Type: Full-Time Interview Mode: First Video then In Person Job Description Job Overview: We are looking for experienced Data Engineers proficient in Hadoop, Hive, Python, SQL, and Pyspark/Spark to join our dynamic team. Candidates will be responsible for designing, developing, and maintaining scalable big data solutions. Key Responsibilities: Develop and optimize data pipelines for large-scale data processing. Work with structured and unstructured datasets to derive actionable insights. Collaborate with cross-functional teams to enhance data-driven decision-making. Ensure the performance, scalability, and reliability of data architectures. Implement best practices for data security and governance. Job Type: Full-time Pay: ₹1,597,042.36 - ₹1,988,639.54 per year Schedule: Day shift Application Question(s): How many years of total experience do you have? How many years of relevant experience as Hadoop Data Engineer do you have? Do you have good hands-on-experience in all the skills - Hadoop, Hive, Python, SQL, Pyspark/Spark? Can you join within 45 days? Are you comfortable with F2F interview at Gurgaon/ Hyderabad? Work Location: In person

Posted 1 week ago

Apply

0 years

15 - 19 Lacs

Haryāna

On-site

GlassDoor logo

Role: Hadoop Data Engineer Location: Gurgaon, HR Work Mode: Hybrid Employment Type: Full-Time Interview Mode: First Video then In Person Job Description Job Overview: We are looking for experienced Data Engineers proficient in Hadoop, Hive, Python, SQL, and Pyspark/Spark to join our dynamic team. Candidates will be responsible for designing, developing, and maintaining scalable big data solutions. Key Responsibilities: Develop and optimize data pipelines for large-scale data processing. Work with structured and unstructured datasets to derive actionable insights. Collaborate with cross-functional teams to enhance data-driven decision-making. Ensure the performance, scalability, and reliability of data architectures. Implement best practices for data security and governance. Job Type: Full-time Pay: ₹1,507,675.01 - ₹1,926,524.53 per year Schedule: Day shift Work Location: In person

Posted 1 week ago

Apply

0 years

0 Lacs

Bangalore Urban, Karnataka, India

On-site

Linkedin logo

Should lead key initiatives in GenAI COE on new tech trends, R and D on new product POC Work on proposals, Provide solutions architecture design solution on different business use cases Ability to work across various different GenAI Models and cloud providers Should have a strong experience in implementing projects in Digital Engineering Or Data Analytics Technically lead a team of developers and groom talent Extensive implementation experience in data analytics space or a senior developer role in one of the modern technology stack Excellent programming skills and proficiency in at least one of the major programming scripting languages used in Gen AI orchestration such as Python or PySpark or Java Ability to build API based scalable solutions and debug & troubleshoot software or design issues Hands on exposure to integrating atleast one of the popular LLMs(Open AI GPT, PaLM 2, Dolly, Claude 2, Cohere etc.) using API endpoints. Thorough understanding of prompt engineering; implementation exposure to LLM agents like LangChain & vector databases Pinecone or Chroma or FAISS Basic data engineering skills to load structured & unstructured data from source systems to target data stores. Build and maintain data pipelines and infrastructure to support Hands on exposure to using cloud(Azure/GCP/AWS) services for storage, serverless-logic, search, transcription and chat Extensive experience with data engineering and should be hands on in using Agentic AI Framework, RAG Show more Show less

Posted 1 week ago

Apply

Exploring PySpark Jobs in India

PySpark, a powerful data processing framework built on top of Apache Spark and Python, is in high demand in the job market in India. With the increasing need for big data processing and analysis, companies are actively seeking professionals with PySpark skills to join their teams. If you are a job seeker looking to excel in the field of big data and analytics, exploring PySpark jobs in India could be a great career move.

Top Hiring Locations in India

Here are 5 major cities in India where companies are actively hiring for PySpark roles: 1. Bangalore 2. Pune 3. Hyderabad 4. Mumbai 5. Delhi

Average Salary Range

The estimated salary range for PySpark professionals in India varies based on experience levels. Entry-level positions can expect to earn around INR 6-8 lakhs per annum, while experienced professionals can earn upwards of INR 15 lakhs per annum.

Career Path

In the field of PySpark, a typical career progression may look like this: 1. Junior Developer 2. Data Engineer 3. Senior Developer 4. Tech Lead 5. Data Architect

Related Skills

In addition to PySpark, professionals in this field are often expected to have or develop skills in: - Python programming - Apache Spark - Big data technologies (Hadoop, Hive, etc.) - SQL - Data visualization tools (Tableau, Power BI)

Interview Questions

Here are 25 interview questions you may encounter when applying for PySpark roles:

  • Explain what PySpark is and its main features (basic)
  • What are the advantages of using PySpark over other big data processing frameworks? (medium)
  • How do you handle missing or null values in PySpark? (medium)
  • What is RDD in PySpark? (basic)
  • What is a DataFrame in PySpark and how is it different from an RDD? (medium)
  • How can you optimize performance in PySpark jobs? (advanced)
  • Explain the difference between map and flatMap transformations in PySpark (basic)
  • What is the role of a SparkContext in PySpark? (basic)
  • How do you handle schema inference in PySpark? (medium)
  • What is a SparkSession in PySpark? (basic)
  • How do you join DataFrames in PySpark? (medium)
  • Explain the concept of partitioning in PySpark (medium)
  • What is a UDF in PySpark? (medium)
  • How do you cache DataFrames in PySpark for optimization? (medium)
  • Explain the concept of lazy evaluation in PySpark (medium)
  • How do you handle skewed data in PySpark? (advanced)
  • What is checkpointing in PySpark and how does it help in fault tolerance? (advanced)
  • How do you tune the performance of a PySpark application? (advanced)
  • Explain the use of Accumulators in PySpark (advanced)
  • How do you handle broadcast variables in PySpark? (advanced)
  • What are the different data sources supported by PySpark? (medium)
  • How can you run PySpark on a cluster? (medium)
  • What is the purpose of the PySpark MLlib library? (medium)
  • How do you handle serialization and deserialization in PySpark? (advanced)
  • What are the best practices for deploying PySpark applications in production? (advanced)

Closing Remark

As you explore PySpark jobs in India, remember to prepare thoroughly for interviews and showcase your expertise confidently. With the right skills and knowledge, you can excel in this field and advance your career in the world of big data and analytics. Good luck!

cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

Featured Companies