49 Web Crawling Jobs

We are seeking an experienced Web Scraping Engineer with deep expertise in Scrapy to develop, maintain, and optimize web crawlers. The ideal candidate will have a strong background in extracting, processing, and managing large-scale web data efficiently. Responsibilities : - Write and maintain web scraping scripts using Python Optimize custom web scraping tools and workflows. - Familiarity with Python and web scraping frameworks (Scrapy, Selenium, BeautifulSoup, Requests, Playwright). - Troubleshoot and resolve scraping challenges, including CAPTCHAs, rate limiting, and IP blocking. - Experience with proxy management (Rotating IPs, VPNs, Residential proxies, etc.) - Handle dynamic content using headless browsers and solve CAPTCHAs & IP bans. - Collaborate with senior developers to improve code quality and efficiency. Skills Required : - Python and scraping libraries such as Scrapy and BeautifulSoup - Experience with proxy management, CAPTCHA bypass techniques, and anti-bot evasion. - Ability to optimize crawlers for performance and minimize website detection risks. - Strong background in databases and data storage systems . - Prior Experience of scraping large-scale E-Commerce websites would be plus. - Willingness to learn and take feedback constructively. - Excellent communication and leadership abilities.

You will be responsible for developing the Policies & Next Gen Rule Engine for Digital Lending. This includes working on various RestFul APIs and integrating them with different data sources. Your role will also involve developing various reports and automating tasks necessary for the business operations. The ideal candidate should possess good programming skills using Python and DB. Experience in Numpy and Pandas is highly preferred. Knowledge of AWS or other Cloud Platforms would be advantageous for this role. Additionally, familiarity with Web Scraping or Crawling would also be a plus. In this position, strong logical and analytical skills are essential. Good soft skills and communication abilities will also be beneficial for effective collaboration within the team and across departments.,

About The Role Entity :- Accenture Strategy & Consulting Team :- Strategy & Consulting Global Network Practice :- Responsible AI COE Title :- Responsible AI Specialist/ Sr. Analyst Job location :- Bangalore/Gurgaon/Pune/ Hyderabad/Chennai/Mumbai The rapid development of AI is creating new opportunities to improve the lives of people around the world, from business to healthcare to education. As a result, it is also raising new questions about the best way to build fairness, interpretability, privacy and security into these systems. The Data and AI revolution is changing everything. Its everywhere transforming how we work and play. Join Accenture and help transform leading companies and communities around the world. Accenture is driving these exciting changes and bringing them to life across 40 industries in more than 120 countries. The sheer scale of our capabilities and client engagements and the way we collaborate with the ecosystem, operate, and deliver value provides an unparalleled opportunity to grow and advance. Accentures S&C Global Network Data & AI team covers the range of skills, from Strategy, Data Science, Data Architecture, AI Engineering and Visual Insights. When combined with our broader Strategy and Consulting practice, we bring a unique ability to drive end to end business change through the application of Data and AI. At the forefront of the industry, youll help make our Responsible AI vision a reality for clients looking to better serve their customers and operate always-on enterprises. Were not just focused on increasing revenues our technologies and innovations are making millions of lives easier and more comfortable. But above all, were doing this responsibly and inclusively to make sure AI technology is used equitably and in a way that is both ethically and technically sound. Join us and become an integral part of our global Responsible AI team with the credibility, expertise and insight clients depend on. There will never be a typical day at Accenture, but thats why people love it here. You will be working with famous brands and household names no worrying about how to explain what you do to your family again! We are looking for experienced and motivated individuals who will be a part of the Responsible AI Centre of Excellence (COE) within Accenture to join our multi-disciplinary Responsible AI team . We are committed to help people build AI products/solutions/services in a responsible and trustworthy manner. The ideal candidate should have a strong client-facing consulting background in data science/analytics with an ability to pick up new technologies very quickly . He/she will be passionate about understanding the impact of AI systems on people and society and will have a track record in using tools to undertake assessments such as Fairness/Bias, Explainability, Model Validation and Robustness, to assess model behaviour through a Responsibility lens. Being a part of Accenture will help you grow both professionally and personally as you help shape our thinking and approaches to Responsible AI, working alongside world-class academics, industry leaders and practitioners. Responsible AI is a key strategic priority for Accenture and were looking for the very best in the field to help us meet our ambitious goals in this space. Responsibilities: As a client-facing in Responsible AI, you will consult with Accentures clients on how todesign & develop reliable, effective user-centered AI systems in adherence to general best practices for software systems, together with practices that address responsibility considerations unique to AI & machine learning. You will also be expected to contribute to research on how AI systems can be designed holistically with fairness, interpretability, privacy, security, safety, and robustness built in by design. As part of our team, you will Be a subject matter expert on technical aspects of Responsible AI & Data Collaborate with colleagues to develop best practices, frameworks, tools for scalable implementation of Responsible AI in enterprises Conduct research on Responsible AI policies, principles, issues, risk identification, risk remediation, regulatory requirements, latest trends etc. Bring a strong conceptual understanding of Responsible AI, principles & tools with experience of using these tools in close collaboration with internal and external stakeholders/clients Evaluate and implement technical best practices and tools for fairness, explainability, transparency, accountability, and other relevant aspects of Responsible AI Develop a clear understanding of clients business issues to adopt the best approach to implement the Responsible AI Framework Establish a consistent and collaborative presence by partnering with clients to understand the wider business goals, objectives & competitive constraints Provide thought leadership by publishing in public forums/conferences/blogs on Responsible AI products, research or developments Leading diverse and well-qualified RAI team. Qualification Skillset Requirements: Education :- PhD / Masters / Bachelors degree in Statistics / Economics / Mathematics /Computer Science / Physics or related disciplines from Premier Colleges in India or Abroad. Specialization in Data Science. Must Have 3 10 years of Hands-on Data science experience in solving real life complex business problems Minimum 1 years experience in enhancing AI systems to meet Responsible AI principles - Explainability, Fairness, Accountability, etc. Passionate about understanding the impact of AI systems on people and society Hands-on experience of using techniques such as data bias testing (e.g. for under-represented groups, proxy variables, recall bias, skew etc), Explainability (e.g. SHAP values, LIME), sensitivity testing, repeatability and similar to understand model limitations. Demonstrated experience in writing reports that summarize analysis / assessments into simple and concise actionable points Strong conceptual knowledge and practical experience in the Development, Validation, and Deployment of ML/AL models such as: Supervised Learning - regression, classification techniques Unsupervised Learning clustering techniques Recommender Systems Reinforcement Learning Deep Learning Sequence models (RNN, GRU, LSTM etc.), CNN, GAN etc Econometric models Exploratory Data analysis, Hypothesis testing etc. Comfortable in ingestion of technical whitepapers, legal policies, government regulations etc. in relation into Responsible AI and work with Academic partners to convert them into practice Ability to learn and develop new methods, strategies and frameworks to proactively identify potential loopholes. Comfortable with ambiguity, believe in first principles and have the skill to transform broad ideas into action plans Excellent written and verbal communication skills with ability to clearly communicate ideas and results to both technical and non-technical business audience, such as senior leaders Good time management skills to manage day-to-day work progress and ensure timely and high-quality deliverables Self-motivated with ability to work independently across multiple projects and set priorities and Strong analytical bent of mind. Good to have Cloud Certifications (Azure / AWS / GCP) Knowledge of AWS SageMaker Clarify / Azure Responsible ML and Fairlearn SDK / GCP AI Explanations Experience in Chatbot Analytics, Web Crawling Experience in MLOps tools like MLflow or Kubeflow Knowledge of cybersecurity, vulnerability assessment, risk remediation etc.

About The Role Entity :- Accenture Strategy & Consulting Team :- Strategy & Consulting Global Network Practice :- Responsible AI COE Title :- Responsible AI Manager Job location :- Bangalore/Gurgaon/Pune/ Hyderabad/Chennai/Mumbai The rapid development of AI is creating new opportunities to improve the lives of people around the world, from business to healthcare to education. As a result, it is also raising new questions about the best way to build fairness, interpretability, privacy and security into these systems. The Data and AI revolution is changing everything. Its everywhere transforming how we work and play. Join Accenture and help transform leading companies and communities around the world. Accenture is driving these exciting changes and bringing them to life across 40 industries in more than 120 countries. The sheer scale of our capabilities and client engagements and the way we collaborate with the ecosystem, operate, and deliver value provides an unparalleled opportunity to grow and advance. Accentures S&C Global Network Data & AI team covers the range of skills, from Strategy, Data Science, Data Architecture, AI Engineering and Visual Insights. When combined with our broader Strategy and Consulting practice, we bring a unique ability to drive end to end business change through the application of Data and AI. At the forefront of the industry, youll help make our Responsible AI vision a reality for clients looking to better serve their customers and operate always-on enterprises. Were not just focused on increasing revenues our technologies and innovations are making millions of lives easier and more comfortable. But above all, were doing this responsibly and inclusively to make sure AI technology is used equitably and in a way that is both ethically and technically sound. Join us and become an integral part of our global Responsible AI team with the credibility, expertise and insight clients depend on. There will never be a typical day at Accenture, but thats why people love it here. You will be working with famous brands and household names no worrying about how to explain what you do to your family again! We are looking for experienced and motivated individuals who will be a part of the Responsible AI Centre of Excellence (COE) within Accenture to join our multi-disciplinary Responsible AI team . We are committed to help people build AI products/solutions/services in a responsible and trustworthy manner. The ideal candidate should have a strong client-facing consulting background in data science/analytics with an ability to pick up new technologies very quickly . He/she will be passionate about understanding the impact of AI systems on people and society and will have a track record in using tools to undertake assessments such as Fairness/Bias, Explainability, Model Validation and Robustness, to assess model behaviour through a Responsibility lens. Being a part of Accenture will help you grow both professionally and personally as you help shape our thinking and approaches to Responsible AI, working alongside world-class academics, industry leaders and practitioners. Responsible AI is a key strategic priority for Accenture and were looking for the very best in the field to help us meet our ambitious goals in this space. Responsibilities: As a client-facing in Responsible AI, you will consult with Accentures clients on how todesign & develop reliable, effective user-centered AI systems in adherence to general best practices for software systems, together with practices that address responsibility considerations unique to AI & machine learning. You will also be expected to contribute to research on how AI systems can be designed holistically with fairness, interpretability, privacy, security, safety, and robustness built in by design. As part of our team, you will Be a subject matter expert on technical aspects of Responsible AI & Data Collaborate with colleagues to develop best practices, frameworks, tools for scalable implementation of Responsible AI in enterprises Conduct research on Responsible AI policies, principles, issues, risk identification, risk remediation, regulatory requirements, latest trends etc. Bring a strong conceptual understanding of Responsible AI, principles & tools with experience of using these tools in close collaboration with internal and external stakeholders/clients Evaluate and implement technical best practices and tools for fairness, explainability, transparency, accountability, and other relevant aspects of Responsible AI Develop a clear understanding of clients business issues to adopt the best approach to implement the Responsible AI Framework Establish a consistent and collaborative presence by partnering with clients to understand the wider business goals, objectives & competitive constraints Provide thought leadership by publishing in public forums/conferences/blogs on Responsible AI products, research or developments Leading diverse and well-qualified RAI team. Qualification Skillset Requirements: Education :- PhD / Masters / Bachelors degree in Statistics / Economics / Mathematics /Computer Science / Physics or related disciplines from Premier Colleges in India or Abroad. Specialization in Data Science. Must Have 7+ years of Hands-on Data science experience in solving real life complex business problems Minimum 3 years experience in enhancing AI systems to meet Responsible AI principles - Explainability, Fairness, Accountability, etc. Passionate about understanding the impact of AI systems on people and society Hands-on experience of using techniques such as data bias testing (e.g. for under-represented groups, proxy variables, recall bias, skew etc), Explainability (e.g. SHAP values, LIME), sensitivity testing, repeatability and similar to understand model limitations. Demonstrated experience in writing reports that summarize analysis / assessments into simple and concise actionable points Strong conceptual knowledge and practical experience in the Development, Validation, and Deployment of ML/AL models such as: Supervised Learning - regression, classification techniques Unsupervised Learning clustering techniques Recommender Systems Reinforcement Learning Deep Learning Sequence models (RNN, GRU, LSTM etc.), CNN, GAN etc Econometric models Exploratory Data analysis, Hypothesis testing etc. Comfortable in ingestion of technical whitepapers, legal policies, government regulations etc. in relation into Responsible AI and work with Academic partners to convert them into practice Ability to learn and develop new methods, strategies and frameworks to proactively identify potential loopholes. Comfortable with ambiguity, believe in first principles and have the skill to transform broad ideas into action plans Excellent written and verbal communication skills with ability to clearly communicate ideas and results to both technical and non-technical business audience, such as senior leaders Good time management skills to manage day-to-day work progress and ensure timely and high-quality deliverables Self-motivated with ability to work independently across multiple projects and set priorities and Strong analytical bent of mind. Good to have Cloud Certifications (Azure / AWS / GCP) Knowledge of AWS SageMaker Clarify / Azure Responsible ML and Fairlearn SDK / GCP AI Explanations Experience in Chatbot Analytics, Web Crawling Experience in MLOps tools like MLflow or Kubeflow Knowledge of cybersecurity, vulnerability assessment, risk remediation etc.

About The Role Entity :- Accenture Strategy & Consulting Team :- Strategy & Consulting Global Network Practice :- Responsible AI COE Title :- Responsible AI Specialist/ Sr. Analyst Job location :- Bangalore/Gurgaon/Pune/ Hyderabad/Chennai/Mumbai The rapid development of AI is creating new opportunities to improve the lives of people around the world, from business to healthcare to education. As a result, it is also raising new questions about the best way to build fairness, interpretability, privacy and security into these systems. The Data and AI revolution is changing everything. Its everywhere transforming how we work and play. Join Accenture and help transform leading companies and communities around the world. Accenture is driving these exciting changes and bringing them to life across 40 industries in more than 120 countries. The sheer scale of our capabilities and client engagements and the way we collaborate with the ecosystem, operate, and deliver value provides an unparalleled opportunity to grow and advance. Accentures S&C Global Network Data & AI team covers the range of skills, from Strategy, Data Science, Data Architecture, AI Engineering and Visual Insights. When combined with our broader Strategy and Consulting practice, we bring a unique ability to drive end to end business change through the application of Data and AI. At the forefront of the industry, youll help make our Responsible AI vision a reality for clients looking to better serve their customers and operate always-on enterprises. Were not just focused on increasing revenues our technologies and innovations are making millions of lives easier and more comfortable. But above all, were doing this responsibly and inclusively to make sure AI technology is used equitably and in a way that is both ethically and technically sound. Join us and become an integral part of our global Responsible AI team with the credibility, expertise and insight clients depend on. There will never be a typical day at Accenture, but thats why people love it here. You will be working with famous brands and household names no worrying about how to explain what you do to your family again! We are looking for experienced and motivated individuals who will be a part of the Responsible AI Centre of Excellence (COE) within Accenture to join our multi-disciplinary Responsible AI team . We are committed to help people build AI products/solutions/services in a responsible and trustworthy manner. The ideal candidate should have a strong client-facing consulting background in data science/analytics with an ability to pick up new technologies very quickly . He/she will be passionate about understanding the impact of AI systems on people and society and will have a track record in using tools to undertake assessments such as Fairness/Bias, Explainability, Model Validation and Robustness, to assess model behaviour through a Responsibility lens. Being a part of Accenture will help you grow both professionally and personally as you help shape our thinking and approaches to Responsible AI, working alongside world-class academics, industry leaders and practitioners. Responsible AI is a key strategic priority for Accenture and were looking for the very best in the field to help us meet our ambitious goals in this space. Qualification Responsibilities: As a client-facing in Responsible AI, you will consult with Accentures clients on how todesign & develop reliable, effective user-centered AI systems in adherence to general best practices for software systems, together with practices that address responsibility considerations unique to AI & machine learning. You will also be expected to contribute to research on how AI systems can be designed holistically with fairness, interpretability, privacy, security, safety, and robustness built in by design. As part of our team, you will Be a subject matter expert on technical aspects of Responsible AI & Data Collaborate with colleagues to develop best practices, frameworks, tools for scalable implementation of Responsible AI in enterprises Conduct research on Responsible AI policies, principles, issues, risk identification, risk remediation, regulatory requirements, latest trends etc. Bring a strong conceptual understanding of Responsible AI, principles & tools with experience of using these tools in close collaboration with internal and external stakeholders/clients Evaluate and implement technical best practices and tools for fairness, explainability, transparency, accountability, and other relevant aspects of Responsible AI Develop a clear understanding of clients business issues to adopt the best approach to implement the Responsible AI Framework Establish a consistent and collaborative presence by partnering with clients to understand the wider business goals, objectives & competitive constraints Provide thought leadership by publishing in public forums/conferences/blogs on Responsible AI products, research or developments Leading diverse and well-qualified RAI team. Skillset Requirements: Education :- PhD / Masters / Bachelors degree in Statistics / Economics / Mathematics /Computer Science / Physics or related disciplines from Premier Colleges in India or Abroad. Specialization in Data Science. Must Have 3 10 years of Hands-on Data science experience in solving real life complex business problems Minimum 1 years experience in enhancing AI systems to meet Responsible AI principles - Explainability, Fairness, Accountability, etc. Passionate about understanding the impact of AI systems on people and society Hands-on experience of using techniques such as data bias testing (e.g. for under-represented groups, proxy variables, recall bias, skew etc), Explainability (e.g. SHAP values, LIME), sensitivity testing, repeatability and similar to understand model limitations. Demonstrated experience in writing reports that summarize analysis / assessments into simple and concise actionable points Strong conceptual knowledge and practical experience in the Development, Validation, and Deployment of ML/AL models such as: Supervised Learning - regression, classification techniques Unsupervised Learning clustering techniques Recommender Systems Reinforcement Learning Deep Learning Sequence models (RNN, GRU, LSTM etc.), CNN, GAN etc Econometric models Exploratory Data analysis, Hypothesis testing etc. Comfortable in ingestion of technical whitepapers, legal policies, government regulations etc. in relation into Responsible AI and work with Academic partners to convert them into practice Ability to learn and develop new methods, strategies and frameworks to proactively identify potential loopholes. Comfortable with ambiguity, believe in first principles and have the skill to transform broad ideas into action plans Excellent written and verbal communication skills with ability to clearly communicate ideas and results to both technical and non-technical business audience, such as senior leaders Good time management skills to manage day-to-day work progress and ensure timely and high-quality deliverables Self-motivated with ability to work independently across multiple projects and set priorities and Strong analytical bent of mind. Good to have Cloud Certifications (Azure / AWS / GCP) Knowledge of AWS SageMaker Clarify / Azure Responsible ML and Fairlearn SDK / GCP AI Explanations Experience in Chatbot Analytics, Web Crawling Experience in MLOps tools like MLflow or Kubeflow Knowledge of cybersecurity, vulnerability assessment, risk remediation etc.

Highly skilled security researcher / reverse engineer expertise in fast Python development . Skill to analyze platforms, identify fresh access points, and implement ultra-fast scrapers that feed our real-time WebSocket/raw pipeline, proxy Experience Annual bonus

Entity :- Accenture Strategy & Consulting Team :- Strategy & Consulting Global Network Practice :- Responsible AI COE Title :- Responsible AI Specialist/ Sr. Analyst Job location :- Bangalore/Gurgaon/Pune/ Hyderabad/Chennai/Mumbai The rapid development of AI is creating new opportunities to improve the lives of people around the world, from business to healthcare to education. As a result, it is also raising new questions about the best way to build fairness, interpretability, privacy and security into these systems. The Data and AI revolution is changing everything. Its everywhere transforming how we work and play. Join Accenture and help transform leading companies and communities around the world. Accenture is driving these exciting changes and bringing them to life across 40 industries in more than 120 countries. The sheer scale of our capabilities and client engagements and the way we collaborate with the ecosystem, operate, and deliver value provides an unparalleled opportunity to grow and advance. Accentures S&C Global Network Data & AI team covers the range of skills, from Strategy, Data Science, Data Architecture, AI Engineering and Visual Insights. When combined with our broader Strategy and Consulting practice, we bring a unique ability to drive end to end business change through the application of Data and AI. At the forefront of the industry, youll help make our Responsible AI vision a reality for clients looking to better serve their customers and operate always-on enterprises. Were not just focused on increasing revenues our technologies and innovations are making millions of lives easier and more comfortable. But above all, were doing this responsibly and inclusively to make sure AI technology is used equitably and in a way that is both ethically and technically sound. Join us and become an integral part of our global Responsible AI team with the credibility, expertise and insight clients depend on. There will never be a typical day at Accenture, but thats why people love it here. You will be working with famous brands and household names no worrying about how to explain what you do to your family again! We are looking for experienced and motivated individuals who will be a part of the Responsible AI Centre of Excellence (COE) within Accenture to join our multi-disciplinary Responsible AI team . We are committed to help people build AI products/solutions/services in a responsible and trustworthy manner. The ideal candidate should have a strong client-facing consulting background in data science/analytics with an ability to pick up new technologies very quickly . He/she will be passionate about understanding the impact of AI systems on people and society and will have a track record in using tools to undertake assessments such as Fairness/Bias, Explainability, Model Validation and Robustness, to assess model behaviour through a Responsibility lens. Being a part of Accenture will help you grow both professionally and personally as you help shape our thinking and approaches to Responsible AI, working alongside world-class academics, industry leaders and practitioners. Responsible AI is a key strategic priority for Accenture and were looking for the very best in the field to help us meet our ambitious goals in this space.Qualification Responsibilities: As a client-facing in Responsible AI, you will consult with Accentures clients on how todesign & develop reliable, effective user-centered AI systems in adherence to general best practices for software systems, together with practices that address responsibility considerations unique to AI & machine learning. You will also be expected to contribute to research on how AI systems can be designed holistically with fairness, interpretability, privacy, security, safety, and robustness built in by design. As part of our team, you will: Be a subject matter expert on technical aspects of Responsible AI & Data Collaborate with colleagues to develop best practices, frameworks, tools for scalable implementation of Responsible AI in enterprises Conduct research on Responsible AI policies, principles, issues, risk identification, risk remediation, regulatory requirements, latest trends etc. Bring a strong conceptual understanding of Responsible AI, principles & tools with experience of using these tools in close collaboration with internal and external stakeholders/clients Evaluate and implement technical best practices and tools for fairness, explainability, transparency, accountability, and other relevant aspects of Responsible AI Develop a clear understanding of clients business issues to adopt the best approach to implement the Responsible AI Framework Establish a consistent and collaborative presence by partnering with clients to understand the wider business goals, objectives & competitive constraints Provide thought leadership by publishing in public forums/conferences/blogs on Responsible AI products, research or developments Leading diverse and well-qualified RAI team. Skillset : Education :- PhD / Masters / Bachelors degree in Statistics / Economics / Mathematics /Computer Science / Physics or related disciplines from Premier Colleges in India or Abroad. Specialization in Data Science. Must Have 3 10 years of Hands-on Data science experience in solving real life complex business problems Minimum 1 years experience in enhancing AI systems to meet Responsible AI principles - Explainability, Fairness, Accountability, etc. Passionate about understanding the impact of AI systems on people and society Hands-on experience of using techniques such as data bias testing (e.g. for under-represented groups, proxy variables, recall bias, skew etc), Explainability (e.g. SHAP values, LIME), sensitivity testing, repeatability and similar to understand model limitations. Demonstrated experience in writing reports that summarize analysis / assessments into simple and concise actionable points Strong conceptual knowledge and practical experience in the Development, Validation, and Deployment of ML/AL models such as: Supervised Learning - regression, classification techniques Unsupervised Learning clustering techniques Recommender Systems Reinforcement Learning Deep Learning Sequence models (RNN, GRU, LSTM etc.), CNN, GAN etc Econometric models Exploratory Data analysis, Hypothesis testing etc. Comfortable in ingestion of technical whitepapers, legal policies, government regulations etc. in relation into Responsible AI and work with Academic partners to convert them into practice Ability to learn and develop new methods, strategies and frameworks to proactively identify potential loopholes. Comfortable with ambiguity, believe in first principles and have the skill to transform broad ideas into action plans Excellent written and verbal communication skills with ability to clearly communicate ideas and results to both technical and non-technical business audience, such as senior leaders Good time management skills to manage day-to-day work progress and ensure timely and high-quality deliverables Self-motivated with ability to work independently across multiple projects and set priorities and Strong analytical bent of mind. Good to have Cloud Certifications (Azure / AWS / GCP) Knowledge of AWS SageMaker Clarify / Azure Responsible ML and Fairlearn SDK / GCP AI Explanations Experience in Chatbot Analytics, Web Crawling Experience in MLOps tools like MLflow or Kubeflow Knowledge of cybersecurity, vulnerability assessment, risk remediation etc.

Responsibilities: * Design, develop & maintain web scrapers using Scrapy framework. * Optimize crawl speed & handle anti-scraping measures. * Ensure data accuracy & compliance with privacy policies. Work from home

We are seeking an experienced Web Scraping Engineer with deep expertise in Scrapy to develop, maintain, and optimize web crawlers. The ideal candidate will have a strong background in extracting, processing, and managing large-scale web data efficiently. Responsibilities : - Write and maintain web scraping scripts using Python Optimize custom web scraping tools and workflows. - Familiarity with Python and web scraping frameworks (Scrapy, Selenium, BeautifulSoup, Requests, Playwright). - Troubleshoot and resolve scraping challenges, including CAPTCHAs, rate limiting, and IP blocking. - Experience with proxy management (Rotating IPs, VPNs, Residential proxies, etc.) - Handle dynamic content using headless browsers and solve CAPTCHAs & IP bans. - Collaborate with senior developers to improve code quality and efficiency. Skills Required : - Python and scraping libraries such as Scrapy and BeautifulSoup - Experience with proxy management, CAPTCHA bypass techniques, and anti-bot evasion. - Ability to optimize crawlers for performance and minimize website detection risks. - Strong background in databases and data storage systems . - Prior Experience of scraping large-scale E-Commerce websites would be plus. - Willingness to learn and take feedback constructively. - Excellent communication and leadership abilities.

You will be responsible for developing and executing comprehensive digital marketing strategies across various channels such as SEO, PPC, social media, content marketing, and email marketing. Your tasks will include using web crawling and data scraping techniques to gather insights for targeted campaigns, managing and optimizing campaigns for improved ROI, conducting competitor analysis and keyword research, and creating engaging content for blogs, websites, and social media platforms. Additionally, you will analyze performance data and provide actionable insights using analytics tools to optimize strategies, as well as collaborate with teams across the company to align digital marketing with overall business goals. To be successful in this role, you must have proven experience in digital marketing with a track record of successful campaigns and a degree in any related field. Strong proficiency in web crawling and data scraping tools is essential, along with expertise in SEO, Google Ads, social media advertising, email marketing platforms, and analytics tools. You should also have an in-depth understanding of performance tracking, A/B testing, and conversion rate optimization, as well as solid knowledge of HTML, CSS, and web analytics. Excellent communication and collaboration skills, the ability to work independently, prioritize tasks, and manage time effectively are also required. Candidates only from Kerala will be considered for this position. In return, you will have opportunities for career growth and advancement within the company, professional development opportunities, a dynamic and collaborative work environment, a competitive salary based on experience and performance, and team-building activities and work-life balance support. We believe in investing in our employees" success and well-being and are committed to providing a supportive and rewarding work environment. Join us and be part of our journey to success! You should possess a strong ability to convince and close deals, excellent verbal and written communication skills, the ability to work diligently under pressure, eagerness to learn about software and technology, and resilience in facing challenges and rejections. A post-graduation in Business, Marketing, or a related field is a plus.,

As a Python Developer & Web Crawling professional at Global Checks Pvt Ltd, you will collaborate with cross-functional teams to define, design, and implement new features. Your role will involve ensuring the performance, quality, and responsiveness of web crawling systems. You will be responsible for identifying and correcting bottlenecks, fixing bugs in web crawling processes, and helping maintain code quality, organization, and automation. It is essential to stay up-to-date with the latest industry trends and technologies. With 3+ years of experience in Web Scraping or Crawling through frameworks like Scrapy, Selenium, or other related libraries, you should be an expert in the latest version of Python. Your expertise will be crucial in fetching data from multiple online sources, cleansing it, and building APIs on top of it. A good understanding of data structure and algorithms, as well as knowledge in bypassing Bot Detection techniques, is required. Experience in Web RestFul APIs/Microservices Development is preferred. In this role, you will need to think deeply about developing large-scale scraping tools, including data integrity, health, and monitoring systems. It is important to develop a deep understanding of vast data sources on the web and know how, when, and which data to scrape, parse, and store. Working with SQL and NoSQL databases to store raw data, developing frameworks for automating and maintaining a constant flow of data from multiple sources, and having knowledge of distributed technologies are essential. You should possess a strong passion for coding, take quality, security, and performance seriously, and be able to work independently with little supervision to research and test innovative solutions. Collaboration with other engineers and cross-teams is necessary, and excellent communication skills are required to present effectively to both business and technical stakeholders. Joining Global Checks Pvt Ltd offers various perks and benefits, including learning directly from industry experts with over 20 years of experience, the opportunity to work in a dynamic and collaborative environment, professional development and growth opportunities, exposure to world-class management practices, and insurance benefits (Medical and Accidental) for all employees. Additionally, there are multi-level rewards programs for all employees. This role provides a challenging yet rewarding opportunity to contribute to a cutting-edge web crawling system and work with a talented team to drive innovation and excellence in data scraping and API development.,

Role & responsibilities As a Data Engineer focused on web crawling and platform data acquisition, you will design,develop, and maintain large-scale web scraping pipelines to extract valuable platform data. Youwill be responsible for implementing scalable and resilient data extraction solutions, ensuringseamless data retrieval while working with proxy management, anti-bot bypass techniques, anddata parsing. Optimizing scraping workflows for performance, reliability, and efficiency will be akey part of your role. Additionally, you will ensure that all extracted data maintains high qualityand integrity. Preferred candidate profile We are seeking candidates with: Strong experience in Python and web scraping frameworks such as Scrapy, Selenium, Playwright, or BeautifulSoup. Knowledge of distributed web crawling architectures and job scheduling. Familiarity with headless browsers, CAPTCHA-solving techniques, and proxy management to handle dynamic web challenges. Experience with data storage solutions, including SQL, and cloud storage. Understanding of big data technologies like Spark and Kafka (a plus). Strong debugging skills to adapt to website structure changes and blockers. A proactive, problem-solving mindset and ability to work effectively in a team-driven environment.

As a Web Scraper, you will be responsible for leveraging your expertise to collect data from various online sources. Your duties will include developing robust web scrapers and parsers for different websites, extracting both structured and unstructured data, and storing it in SQL or NoSQL databases. You will collaborate closely with Project, Business, and Research teams to provide them with the scraped data for analysis. Your role will also involve maintaining the scraping projects that have been deployed to production, as well as creating frameworks to automate and ensure a continuous flow of data from multiple sources. Working independently with minimal supervision, you will need to gain a deep understanding of web data sources and be able to identify, extract, parse, and store relevant data effectively. ### Responsibilities: - Apply your knowledge to fetch data from multiple online sources - Develop reliable web scrapers and parsers for various websites - Extract structured/unstructured data and store it in SQL/NoSQL databases - Collaborate with Project/Business/Research teams to provide scraped data for analysis - Maintain scraping projects in production - Create frameworks for automating and ensuring a constant flow of data from multiple sources - Work independently with minimal supervision - Develop a deep understanding of web data sources to identify, extract, parse, and store data effectively ### Required Skills And Experience: - 1 to 3 years of experience as a Web Scraper - Proficient in Python and familiar with Web Crawling/Web scraping using Python Requests, Beautifulsoup or URLlib, Selenium, and Playwright - Strong knowledge of basic Linux commands for system navigation, management, and troubleshooting - Expertise in proxy usage for secure and efficient network operations - Experience with captcha-solving techniques for automation and data extraction - Knowledge of data parsing techniques - Proficiency in Regular Expressions, HTML, CSS, DOM, and XPATH - Familiarity with JavaScript is a plus - Ability to access, manipulate, and transform data from various database and flat file sources - Essential skills in MongoDB & MySQL - Ability to develop reusable code-based scraping products for broader use - Proficiency in GIT for version control and collaborative development workflows - Experience with cloud servers on platforms like AWS, GCP, and LEAPSWITCH for scalable and reliable infrastructure management - Strong problem-solving skills and innovative thinking - Ability to deliver results in a clear and understandable manner for clients ### Note: This job description is sourced from hirist.tech.,

1. To ensure proper storage of scrap & surplus generated in the hospital which has the salvage value or which can be used later on 2. Make sure that allocated scrap area is properly arranged & labelled, if not then coordinate with user dept. 3. To maintain places or godown (Outside Hospital) required to store these materials in proper condition in terms of cleanliness and Safety. 4. Gathering scrap materials from designated areas and with help of mathadis/scrap vendor loading them into vehicles, and transporting them to the disposal site. 5. Separating different types of scrap (metals, plastics, paper, etc.) and preparing them for recycling or disposal with help of mathadis/scrap vendor. 6. Maintaining accurate records of materials received, processed, and disposed of, including quantities and types. 7. Ensuring all activities comply with safety regulations and environmental policies. 8. To maintain database of scrap vendors and update the same on periodic basis. 9. To call for competitive bids from Scrap vendors for various scrap materials. 10. Carry out the proper Checking of Scrap items & fill in Scrap Sale Check List & take a signature of respective representatives in Scrap Sale Check List form. 11. To carry out actual selling of scrap after obtaining necessary approval both for regular scrap items as well as scrapped capital assets as per SOP. 12. Preparation of NRGP for sale of Scrap items along with record note against approved list and submit to purchase dept. for further process and maintaining the record of same. 13. To provide “Mathadi” manpower to any of the department of the hospital for heavy physical movement activity. 14. To supervise and maintain any of the items in legal disputes i.e. CESTAT/Octroi etc 15. Interacting with clients and the public, addressing inquiries and ensuring smooth operations. 16. For Capex items : Checking of items against “ ASSET SCRAP PROPPOSAL FORM “ in locations mentioned on note and tagging of same and along with Matadi workers and shifting items to allocated area against Material Transfer Note. Roles and Responsibilities 1. To ensure proper storage of scrap & surplus generated in the hospital which has the salvage value or which can be used later on 2. Make sure that allocated scrap area is properly arranged & labelled, if not then coordinate with user dept. 3. To maintain places or godown (Outside Hospital) required to store these materials in proper condition in terms of cleanliness and Safety. 4. Gathering scrap materials from designated areas and with help of mathadis/scrap vendor loading them into vehicles, and transporting them to the disposal site. 5. Separating different types of scrap (metals, plastics, paper, etc.) and preparing them for recycling or disposal with help of mathadis/scrap vendor. 6. Maintaining accurate records of materials received, processed, and disposed of, including quantities and types. 7. Ensuring all activities comply with safety regulations and environmental policies. 8. To maintain database of scrap vendors and update the same on periodic basis. 9. To call for competitive bids from Scrap vendors for various scrap materials. 10. Carry out the proper Checking of Scrap items & fill in Scrap Sale Check List & take a signature of respective representatives in Scrap Sale Check List form. 11. To carry out actual selling of scrap after obtaining necessary approval both for regular scrap items as well as scrapped capital assets as per SOP. 12. Preparation of NRGP for sale of Scrap items along with record note against approved list and submit to purchase dept. for further process and maintaining the record of same. 13. To provide “Mathadi” manpower to any of the department of the hospital for heavy physical movement activity. 14. To supervise and maintain any of the items in legal disputes i.e. CESTAT/Octroi etc 15. Interacting with clients and the public, addressing inquiries and ensuring smooth operations. 16. For Capex items : Checking of items against “ ASSET SCRAP PROPPOSAL FORM “ in locations mentioned on note and tagging of same and along with Matadi workers and shifting items to allocated area against Material Transfer Note.

Role & responsibilities As a Data Engineer focused on web crawling and platform data acquisition, you will design, develop, and maintain large-scale web scraping pipelines to extract valuable platform data. You will be responsible for implementing scalable and resilient data extraction solutions, ensuring seamless data retrieval while working with proxy management, anti-bot bypass techniques, and data parsing. Optimizing scraping workflows for performance, reliability, and efficiency will be a key part of your role. Additionally, you will ensure that all extracted data maintains high quality and integrity. Preferred candidate profile We are seeking candidates with: Strong experience in Python and web scraping frameworks such as Scrapy, Selenium, Playwright, or BeautifulSoup. Knowledge of distributed web crawling architectures and job scheduling. Familiarity with headless browsers, CAPTCHA-solving techniques, and proxy management to handle dynamic web challenges. Experience with data storage solutions, including SQL, and cloud storage. Understanding of big data technologies like Spark and Kafka (a plus). Strong debugging skills to adapt to website structure changes and blockers. A proactive, problem-solving mindset and ability to work effectively in a team-driven environment.

As a Senior Software Engineer specializing in .NET and Angular development at CUBE, a pioneering RegTech company based in Chennai with a hybrid work setup, you will have the opportunity to contribute to the advancement of machine automated compliance solutions. Recently recognized as a "RegTech Top Performer" in Market Fintech's RegTech Supplier Performance Report, we are dedicated to setting the gold standard for regulatory intelligence and change within the financial services industry. Through our innovative SaaS platform powered by AI and proprietary data ontology, we simplify the complex landscape of compliance for our global clientele. At CUBE, we value strong teamwork, customer commitment, and a forward-thinking approach to shaping the future. With a rapidly expanding footprint in the industry, we are actively seeking talented individuals who are eager to grow professionally and potentially step into leadership roles within our organization. Your responsibilities will include collaborating closely with our lead engineer to develop product features using .NET and Cloud-based technologies. You will play a key role in expanding and optimizing our content processing solutions to meet the growing demands of our clients. Additionally, you will be involved in designing and implementing new products, enhancing existing features, and troubleshooting issues as part of the product sustenance process, all while thriving in a fast-paced environment. Essential skills for this role include having over 5 years of experience as an Application Engineer delivering high-performance, scalable web applications, proficiency in .NET/C#, familiarity with serverless programming (Azure Functions or AWS Lambdas), understanding of Containerization, K8, and AKS, preference for No-SQL DB experience, knowledge of Web Crawling, and expertise in Asynchronous, Parallel, and Concurrent Programming (TPL, Async/Await). Strong communication skills, self-motivation, organizational abilities, and the capacity to work effectively both independently and in a team will be essential. You should also be comfortable working in a challenging delivery environment with tight deadlines. Beneficial skills that would be advantageous for this role include experience with Azure cloud services, Python, Angular, React, Linux, Docker containers, Kubernetes, unit testing, integration testing, and event-driven messaging architecture (Azure ESB or AWS SQS). CUBE's product offerings include RegPlatform, a technology platform that simplifies regulatory change management by providing firms with a continuously updated inventory of global regulations, and RegBrain, which allows customers to leverage CUBE's AI models for faster release cycles. Our innovative AI services cover a range of functions such as structural detection, classification, entity extraction, summarization, and recommendations. Joining CUBE means being part of a unique global team where growth, progression, international collaboration, innovative technology, and work-life balance are prioritized. As a remote-first business, you will have the flexibility to design your home office and choose your work equipment, while also having opportunities for in-person meet-ups to foster collaboration and idea-sharing with colleagues worldwide. If you are an ambitious individual looking to have a real impact, advance your career, and contribute to reshaping the regulatory compliance landscape, CUBE offers an environment where your ideas and efforts can truly make a difference.,

As a Senior Software Engineer at CUBE, you will be part of a pioneering RegTech business that is revolutionizing machine automated compliance. Our global presence in the financial services industry is defined by our SaaS platform, powered by AI and proprietary data ontology, simplifying regulatory intelligence for our clients. You will collaborate closely with our lead engineer to develop product features using .NET and Cloud-based technologies. Your responsibilities will include expanding and optimizing content processing solutions, designing and implementing new products and features, troubleshooting, and investigating for product sustainment, all while thriving in a fast-paced environment. To excel in this role, you must have over 5 years of experience as an Application Engineer delivering high throughput, scalable web-based applications. Proficiency in .NET/C#, serverless programming (Azure Functions or AWS Lambdas), containerization, No-SQL databases, web crawling, and asynchronous programming is essential. Strong communication skills, self-motivation, and the ability to work both independently and within a team are crucial for success in this position. Additionally, beneficial skills include knowledge of Azure cloud services, Python, Angular, React, Linux, Docker containers, Kubernetes, unit testing, integration testing, and event-driven messaging architecture. Our flagship products, RegPlatform and RegBrain, are industry-leading technology platforms that streamline regulatory change management and provide AI-driven solutions for regulatory content management. Joining CUBE means being part of a globally renowned RegTech leader with a culture focused on growth, progression, collaboration, innovation, and work-life balance. As we continue to shape the future of regulatory compliance, we invite ambitious individuals to join us and contribute to our dynamic and forward-thinking workforce.,

As a Python Developer at Innefu Lab, you will play a crucial role in the software development life cycle, contributing from requirements analysis to deployment. Working in collaboration with diverse teams, you will design and implement solutions that align with client requirements and industry standards. Your responsibilities encompass various key areas: Software Development: You will be responsible for creating, testing, and deploying high-quality Python applications and scripts. Code Optimization: Your role involves crafting efficient, reusable, and modular code while enhancing existing codebases for optimal performance. Database Integration: You will integrate Python applications with databases to ensure data integrity and efficient data retrieval. API Development: Designing and implementing RESTful APIs to enable seamless communication between different systems. Collaboration: Working closely with UI/UX designers, backend developers, and stakeholders to ensure effective integration of Python components. Testing and Debugging: Thoroughly testing applications, identifying and rectifying bugs, and ensuring software reliability. Documentation: Creating and maintaining comprehensive technical documentation for code, APIs, and system architecture. Continuous Learning: Staying updated on industry trends, best practices, and emerging technologies related to Python development. Required Skills: - Proficient in Python, Django, Flask - Strong knowledge of Regular Expressions, Pandas, Numpy - Excellent expertise in Web Crawling and Web Scraping - Experience with scraping modules like Selenium, Scrapy, Beautiful Soup, or URLib - Familiarity with text processing, Elasticsearch, and Graph-Based Databases such as Neo4j (optional) - Proficient in data mining, Natural Language Processing (NLP), and Optical Character Recognition (OCR) - Basic understanding of databases - Strong troubleshooting and debugging capabilities - Effective interpersonal, verbal, and written communication skills - Ability to extract data from structured and unstructured sources, analyze text, images, and videos, and utilize NLP frameworks for data enrichment - Skilled in collecting and extracting intelligence from data, utilizing regular expressions, and extracting information from RDBMS databases - Experience in web scraping frameworks like Scrapy for data extraction from websites Join us at Innefu Lab, where innovative offerings and cutting-edge technologies converge to deliver exceptional security solutions. Be part of our dynamic team driving towards excellence and growth in the cybersecurity domain.,

Expertise in CRM/ERP systems Strong experience in system migration from legacy PHP platforms Odoo(Community Edition)/.NET-based platforms,REST APIs,AI/ML integration Experience with CRM & configurator integration (automotive or manufacturing context) Required Candidate profile API integrations (REST/SOAP) AI-based data crawlers integrate GPT-based or ML-based modules automotive industry platforms or product configurators SaaS platform architecture and admin-user separation

Preferred candidate profile Overall 5+ years of IT experience Proven experience in Python programming language Experience in Object Oriented programming using Python Development experience using RESTful services and Web frameworks Automation experience using Python - Including shell scripting, web crawling, parsing PDF/word documents, pattern recognition & regular expression Strong debugging skills Packaging code for release and deployment Experience in Agile development , CI/CD tools Go-getter with Excellent communication and problem-solving skills Experience working in individual contributor role

1.Develop and maintain service that extracts websites data using scrapers and APIs 2.Extract structured / unstructured data 3.Manipulate data through text processing, image processing, regular expressions etc. . Free meal Health insurance Accidental insurance Provident fund

Software Engineer is mainly responsible for core development of the product using best software development practices. We are looking for a highly competent and self-motivated individual for this role who can write effective, reusable, and modular code along with unit tests. And a person who can work with minimal supervision. Requirements: MCA or BE/BTech in Computer Science. Having 1-3 years of exp working in Microsoft technologies stack (C#/.NET Core, SQL Server) Sound knowledge of XPATH and HTML DOM. Develop scripts to extract data from websites and APIs. Experience with Puppeteer or Selenium. Good in algorithms and data structures. Worked in NoSQL database like Mongo DB, will be an added advantage. Knowledge of python will be added advantage. Worked in large scale distributed applications and familiar with event-based programming. Have knowledge of using various cloud services, mainly Azure. Must be familiar with Scrum methodology, CI/CD, Git, Branching/Merging and test-driven software development. Candidate worked in product-based company will be preferred. Good verbal and written communication skills.

1. Conducts comprehensive patent analysis, prepare patent landscape reports and understands the scope of patent protection 2. Analysis of patent documents utilizing a knowledge of chemistry/biotechnology and chemical nomenclature coupled with knowledge of international pharmaceutical patent law 3. Monitors and analyzes patent legal status (patent expiry, withdrawals, lapses and rejections) 4. Monitors and analyzes other sources of intellectual property and regulatory monopoly, including patent extensions, marketing and data exclusivity and patent legal status (patent expiry, reassignments, withdrawals, lapses and rejections) 5. Ongoing monitoring of patent publications in accordance with standards, company priorities and the production cycle 6. Creation of Excel/VBA/Access/SQL based automation tools to increase overall process efficiency 7. Efficiently manage multiple tasks on an ongoing basis which would typically include web crawling, process improvements, quality check, advanced data analytics etc.

Description: Loiking For Senior Software Engineer with 4+ years of experience in Javascript .HTML & CSS. Requirements: CSS Selector HTML, DOM, BOM Advance JavaScript OOPs Concept JSON Chrome developer tool Internet protocols API calls, http headers, CORS & authentication Code optimization and maintainability Code review Good knowledge of writing query in SQL/other databases Appropriate documentation and code commenting -Must have hands on experience on scripting as desired on Advanced JavaScript, HTML, CSS selectors, web crawling etc - Sound knowledge of any Programming language like C/C++/Python and any other scripting language -Excellent written and verbal communication skills, including public speaking -Creative problem-solving and analysis skills -Ability to analyze and come up with efficient workaround -Proficiency in MS Excel, Powerpoint, and Word -In-depth experience with website developers tools -Should have experience in verifying peer work -Experience in training design and delivery, preferably corporate training -Quick learner with proven ability to lead and develop team/ members -Ability to clearly and effectively articulate thoughts and points independently Good to Have Knowledge of Analytical Skills to analyze web sites Latest trends and best practices for debugging Data extraction and normalization Job Responsibilities: Build team structure, recognize leadership potential, and develop enhanced skill sets within the team -Deliver new scripts/crawlers etc in a timely manner to the highest possible quality standard -Design and optimize existing processes, to ensure we optimize for efficiency and quality of output, and provide ongoing feedback on tools -Provide regular reports on growth and performance of the department, and develop metrics to measure this growth -Training curriculum development and delivery for the teams within the office -Coordinate effectively with other Team Leads across Verticals -Assist in the evaluation of candidates -Manage the relationship with the client and all stakeholders -Report and escalate to management as needed -Critical thinker, ability to negotiate with key stakeholders and maintain positive attitude What We Offer: Exciting Projects: We focus on industries like High-Tech, communication, media, healthcare, retail and telecom. Our customer list is full of fantastic global brands and leaders who love what we build for them. Collaborative Environment: You Can expand your skills by collaborating with a diverse team of highly talented people in an open, laidback environment — or even abroad in one of our global centers or client facilities! Work-Life Balance: GlobalLogic prioritizes work-life balance, which is why we offer flexible work schedules, opportunities to work from home, and paid time off and holidays. Professional Development: Our dedicated Learning & Development team regularly organizes Communication skills training(GL Vantage, Toast Master),Stress Management program, professional certifications, and technical and soft skill trainings. Excellent Benefits: We provide our employees with competitive salaries, family medical insurance, Group Term Life Insurance, Group Personal Accident Insurance , NPS(National Pension Scheme ), Periodic health awareness program, extended maternity leave, annual performance bonuses, and referral bonuses. Fun Perks: We want you to love where you work, which is why we host sports events, cultural activities, offer food on subsidies rates, Corporate parties. Our vibrant offices also include dedicated GL Zones, rooftop decks and GL Club where you can drink coffee or tea with your colleagues over a game of table and offer discounts for popular stores and restaurants!

What do you need for this opportunity? Must have skills required: TensorFlow, PyTorch, rag, LangChain Forbes Advisor is Looking for: Location - Remote (For candidate's from Chennai or Mumbai it's hybrid The Data Extraction Team is a brand-new team who plays a crucial role in our organization by designing, implementing, and overseeing advanced web scraping frameworks. Their core function involves creating and refining tools and methodologies to efficiently gather precise and meaningful data from a diverse range of digital platforms. Additionally, this team is tasked with constructing robust data pipelines and implementing Extract, Transform, Load (ETL) processes. These processes are essential for seamlessly transferring the harvested data into our data storage systems, ensuring its ready availability for analysis and utilization. A typical day in the life of a Data Research Engineer will involve coming up with ideas regarding how the company/team can best harness the power of AI/LLM, and use it not only simplify operations within the team, but also to streamline the work of the research team in gathering/retrieving large sets of data. The role is that of a leader who sets a vision for the future of AI/LLMs use within the team and the company. They think outside the box and are proactive in engaging with new technologies and developing new ideas for the team to move forward in the AI/LLM field. The candidate should also at least be willing to acquire some basic skills in scraping and data pipelining. Responsibilities: Develop methods to leverage the potential of LLM and AI within the team. Proactive at finding new solutions to engage the team with AI/LLM, and streamline processes in the team. Be a visionary with AI/LLM tools and predict how the use of future technologies could be harnessed early on so that when these technologies come out, the team is ahead of the game regarding how it could be used. Assist in acquiring and integrating data from various sources, including web crawling and API integration. Stay updated with emerging technologies and industry trends. Explore third-party technologies as alternatives to legacy approaches for efficient data pipelines. Contribute to cross-functional teams in understanding data requirements. Assume accountability for achieving development milestones. Prioritize tasks to ensure timely delivery, in a fast-paced environment with rapidly changing priorities. Collaborate with and assist fellow members of the Data Research Engineering Team as required. Leverage online resources effectively like StackOverflow, ChatGPT, Bard, etc., while considering their capabilities and limitations. Skills and Experience Bachelor's degree in Computer Science, Data Science, or a related field. Higher qualifications is a plus. Think proactively and creatively regarding the next AI/LLM technologies and how to use them to the teams and companys benefits. Think outside the box mentality. Experience prompting LLMs in a streamlined way, taking into account how the LLM can potentially hallucinate and return wrong information. Experience building agentic AI platforms with modular capabilities and autonomous task execution. (crewai, lagchain, etc.) Proficient in implementing Retrieval-Augmented Generation (RAG) pipelines for dynamic knowledge integration. (chromadb, pinecone, etc) Experience managing a team of AI/LLM experts is a plus: this includes setting up goals and objectives for the team and fine-tuning complex models. Strong proficiency in Python programming Proficiency in SQL and data querying is a plus. Familiarity with web crawling techniques and API integration is a plus but not a must. Experience in AI/ML engineering and data extraction Experience with LLMs, NLP frameworks (spaCy, NLTK, Hugging Face, etc.) Strong understanding of machine learning frameworks (TensorFlow, PyTorch) Design and build AI models using LLMs Integrate LLM solutions with existing systems via APIs Collaborate with the team to implement and optimize AI solutions Monitor and improve model performance and accuracy Familiarity with Agile development methodologies is a plus. Strong problem-solving and analytical skills with attention to detail. Creative and critical thinking. Ability to work collaboratively in a team environment. Good and effective communication skills. Experience with version control systems, such as Git, for collaborative development. Ability to thrive in a fast-paced environment with rapidly changing priorities. Comfortable with autonomy and ability to work independently.