106 Web Crawling Jobs - Page 2

About ClearDemand: Clear Demand is the leader in AI-driven price and promotion optimization for retailers . Our platform transforms pricing from a challenge to a competitive advantage, helping retailers make smarter, data-backed decisions across the entire pricing lifecycle. By integrating competitive intelligence, pricing rules, and demand modelling , we enable retailers to maximize profit, drive growth, and enhance customer loyalty all while maintaining pricing compliance and brand integrity. With Clear Demand, retailers stay ahead of the market, automate complex pricing decisions, and unlock new opportunities for growth. Key Responsibilities: People Management Lead, coach, and develop a h...

Role & responsibilities Use AI and automation tools (e.g., ChatGPT, Perplexity, Phantombuster, Playwright, n8n, Zapier, or similar) to streamline data collection, cleaning, and validation. Perform data scraping, enrichment, and regular updates of law firm and attorney information. Identify process gaps and propose AI-based solutions to increase efficiency and accuracy. Work with the team to maintain structured data in spreadsheets, CRMs, and internal dashboards. Summarize and organize research findings clearly in written form. Collaborate with developers and analysts to test and improve automation workflows. Preferred candidate profile 13 years of experience in process operations research, o...

Entity :- Accenture Strategy & Consulting Team :- Strategy & Consulting Global Network Practice :- Responsible AI COE Title :- Responsible AI Specialist/ Sr. Analyst Job location :- Bangalore/Gurgaon/Pune/ Hyderabad/Chennai/Mumbai The rapid development of AI is creating new opportunities to improve the lives of people around the world, from business to healthcare to education. As a result, it is also raising new questions about the best way to build fairness, interpretability, privacy and security into these systems. The Data and AI revolution is changing everything. Its everywhere transforming how we work and play. Join Accenture and help transform leading companies and communities around t...

What We're Looking For We are seeking an Engineer with over 3 years of experience in Web Automation to take ownership of our existing automation workflows and enhance them for speed, resilience, and scalability. You will collaborate closely with our product team to improve systems that extract data from diverse and dynamic endpoints. The ideal candidate is a proactive problem-solver who thrives in fast-paced environments and knows how to develop production-ready automation systems. Your Responsibilities Implement web automation across various endpoints with different complexities. Enhance and modernize current workflows using tools like Playwright and Puppeteer. Design frameworks for abstrac...

About Forage AI: Forage AI is a pioneering AI-powered data extraction and automation company that transforms complex, unstructured web and document data into clean, structured intelligence. Our platform combines web crawling, NLP, LLMs, and agentic AI to deliver highly accurate firmographic and enterprise insights across numerous domains. Trusted by global clients in finance, real estate, and healthcare, Forage AI enables businesses to automate workflows, reduce manual rework, and access high-quality data at scale. About the Role: We are seeking a Data Pipeline Engineer to develop, optimize, and maintain production-grade data pipelines focused on web data extraction and ETL workflows. This i...

As a skilled Digital Marketing Analyst at Cascade Headway, you will play a crucial role in enhancing the company's online presence, managing impactful campaigns, and driving business growth across various digital channels. Key Responsibilities: - Develop and execute comprehensive digital marketing strategies encompassing SEO, PPC, social media, content marketing, and email marketing. - Utilize web crawling and data scraping techniques to gather insights for targeted campaigns and enhance SEO performance. - Manage and optimize campaigns to improve ROI, including both paid media and organic strategies. - Conduct competitor analysis, keyword research, and market trend analysis to refine campaig...

Company: Masthead Technologies Pvt. Ltd. Website: https://www.mastheadtechnologies.com/ Location: Noida(Sector-90) Employment Type: Full-time Designation: Lead Developer(Python) Key Responsibilities Lead a team of developers in the design, development, and deployment of scalable Python-based applications. Define technical architecture, ensure code quality, and enforce best practices. Collaborate with cross-functional teams (Product, DevOps, QA, UI/UX) to deliver high-quality solutions. Review, optimize, and refactor code to enhance performance and maintainability. Oversee project planning, timelines, and resource allocation. Mentor and guide team members through code reviews, technical sessi...

Role Overview: You will be responsible for developing the Policies & Next Gen Rule Engine for Digital Lending. Additionally, you will need to develop various RestFul APIs & integrate them with various data sources. Another key aspect of your role will involve developing various reports & automating tasks required for the business. Your proficiency in programming using Python & DB will be crucial for this role. Moreover, having experience in Numpy & Pandas will be beneficial. Key Responsibilities: - Develop Policies & Next Gen Rule Engine for Digital Lending - Develop various RestFul APIs & integrate with different data sources - Develop various reports & automate tasks necessary for the busi...

We are looking for a skilled Production Manager to lead our Scrap Ops team in the Hotels & Restaurants industry. The ideal candidate will have 3-7 years of experience and a strong background in production management. Roles and Responsibility Manage and oversee the scrap operations process to ensure efficient and effective waste management. Develop and implement strategies to reduce scrap and increase recycling rates. Collaborate with cross-functional teams to identify areas for improvement and optimize processes. Analyze data and metrics to measure performance and inform decision-making. Ensure compliance with regulatory requirements and industry standards. Lead and motivate a team of profes...

Collaborate with cross-functional teams to define, design, and implement new features. Ensure the performance, quality, and responsiveness of web crawling systems. Identify and correct bottlenecks and fix bugs in web crawling processes. Help maintain code quality, organization, and automation. Stay up-to-date with the latest industry trends and technologies. Skills: 3+ years of experience in Web Scraping or Crawling through Scrapy, Selenium or other frameworks and related libraries (like beautifulsoup, puppeteer). Should be expert on latest version of Python. Should have very good experience on fetching data from multiple online sources, cleanse it and build APIs on top of it. Good understan...

Overview Stats Perform is the market leader in sports tech. We provide the most trusted sports data to some of the world's biggest organizations, across sports, media, and broadcasting. Through the latest AI technologies and machine learning, we combine decades' worth of data with the latest in-game happenings. We then offer coaches, teams, professional bodies, and media channels around the world, access to the very best data, content, and insights. In turn, improving how sports fans interact with their favorite sports teams and competitions. How do they use it? Media outlets add a little magic to their coverage with our stats and graphics packages. Sportsbooks can offer better predictions a...

Role & responsibilities As a Data Engineer focused on web crawling and platform data acquisition, you will design, develop, and maintain large-scale web scraping pipelines to extract valuable platform data. You will be responsible for implementing scalable and resilient data extraction solutions, ensuring seamless data retrieval while working with proxy management, anti-bot bypass techniques, and data parsing. Optimizing scraping workflows for performance, reliability, and efficiency will be a key part of your role. Additionally, you will ensure that all extracted data maintains high quality and integrity. Preferred candidate profile We are seeking candidates with: Strong experience in Pytho...

Building and maintaining advanced web scraping and automation scripts across platforms (LinkedIn, Facebook, Twitter, Discord). Handle CAPTCHA, proxy, IP rotation, and anti-blocking. Ensure deployable, scalable, and monitored solutions using Docker.

We are seeking a skilled Python Developer with hands-on experience in web scraping to join our data engineering team. The ideal candidate will be responsible for building robust scraping pipelines, extracting structured data from various websites, and ensuring data quality and compliance with ethical scraping practices. Key Responsibilities Develop and maintain scalable web scraping scripts using Python libraries such as BeautifulSoup, Scrapy, Selenium, or Playwright. Handle dynamic content, pagination, and authentication mechanisms in scraping workflows. Clean, transform, and store scraped data in structured formats (CSV, JSON, databases). Monitor scraping jobs and troubleshoot failures or ...

This role entails building and managing the universe of web content resources that are of a high interest to AlphaSense clients. You will be responsible for the end-to-end process of discovering, onboarding, and monitoring web content ingestion across multiple business domains. You will also play a key role in evolving AlphaSense web processing capabilities. Responsibilities: Web content research : Perform research to discover new web sources based on the established content onboarding policies or client requests. Content Analysis: Evaluate requested web sources to identify content sets that need to be onboarded. Content Onboarding: Configure new web crawling/scraping jobs via admin interfac...

As a Senior Software Engineer at CUBE, you will have the opportunity to be part of a pioneering RegTech business that is transforming machine automated compliance in the financial services industry. You will work with our lead engineer to develop product features using .NET and Cloud-based technologies. Your responsibilities will involve expanding and optimizing content processing solutions, designing and implementing new products and features, troubleshooting, and investigating for product sustainment in a fast-paced environment. Key Responsibilities: - Collaborate with the lead engineer to develop product features using .NET and Cloud-based technologies - Expand and optimize content proces...

Disclaimer: Candidates with pure e-commerce or online retail operations experience will be preferred. Backgrounds limited to logistics or warehousing operations will not be considered for this role Position: Asst. Manager/ Deputy Manager / Manager / Sr. Manager E-Commerce Operations Drive Excellence, Optimize Operations, and Lead the Future of E-Commerce at Contexio! Job Summary Shape the Future of E-Commerce Operations with Contexio We are looking for a dynamic and results-driven Operations Manager Retail Solutions to strategize, manage, and execute high-impact projects in the e-commerce space. This role is ideal for a leader with a strong process management mindset, client servicing expert...

About The Role Entity :- Accenture Strategy & Consulting Team :- Strategy & Consulting Global Network Practice :- Responsible AI COE Title :- Responsible AI Manager Job location :- Bangalore/Gurgaon/Pune/ Hyderabad/Chennai/Mumbai The rapid development of AI is creating new opportunities to improve the lives of people around the world, from business to healthcare to education. As a result, it is also raising new questions about the best way to build fairness, interpretability, privacy and security into these systems. The Data and AI revolution is changing everything. Its everywhere transforming how we work and play. Join Accenture and help transform leading companies and communities around th...

About The Role Entity :- Accenture Strategy & Consulting Team :- Strategy & Consulting Global Network Practice :- Responsible AI COE Title :- Responsible AI Specialist/ Sr. Analyst Job location :- Bangalore/Gurgaon/Pune/ Hyderabad/Chennai/Mumbai The rapid development of AI is creating new opportunities to improve the lives of people around the world, from business to healthcare to education. As a result, it is also raising new questions about the best way to build fairness, interpretability, privacy and security into these systems. The Data and AI revolution is changing everything. Its everywhere transforming how we work and play. Join Accenture and help transform leading companies and commu...

About The Role Entity :- Accenture Strategy & Consulting Team :- Strategy & Consulting Global Network Practice :- Responsible AI COE Title :- Responsible AI Specialist/ Sr. Analyst Job location :- Bangalore/Gurgaon/Pune/ Hyderabad/Chennai/Mumbai The rapid development of AI is creating new opportunities to improve the lives of people around the world, from business to healthcare to education. As a result, it is also raising new questions about the best way to build fairness, interpretability, privacy and security into these systems. The Data and AI revolution is changing everything. Its everywhere transforming how we work and play. Join Accenture and help transform leading companies and commu...

Entity :- Accenture Strategy & Consulting Team :- Strategy & Consulting Global Network Practice :- Responsible AI COE Title :- Responsible AI Specialist/ Sr. Analyst Job location :- Bangalore/Gurgaon/Pune/ Hyderabad/Chennai/Mumbai The rapid development of AI is creating new opportunities to improve the lives of people around the world, from business to healthcare to education. As a result, it is also raising new questions about the best way to build fairness, interpretability, privacy and security into these systems. The Data and AI revolution is changing everything. Its everywhere transforming how we work and play. Join Accenture and help transform leading companies and communities around t...

We are seeking an experienced Web Scraping Engineer with deep expertise in Scrapy to develop, maintain, and optimize web crawlers. The ideal candidate will have a strong background in extracting, processing, and managing large-scale web data efficiently. Responsibilities : - Write and maintain web scraping scripts using Python Optimize custom web scraping tools and workflows. - Familiarity with Python and web scraping frameworks (Scrapy, Selenium, BeautifulSoup, Requests, Playwright). - Troubleshoot and resolve scraping challenges, including CAPTCHAs, rate limiting, and IP blocking. - Experience with proxy management (Rotating IPs, VPNs, Residential proxies, etc.) - Handle dynamic content us...

Role Overview: As a Web Scraper, your main responsibility is to apply your expertise to extract data from various online sources. You will be tasked with developing reliable web scrapers and parsers for different websites. Your role involves extracting structured/unstructured data and storing it in SQL/NoSQL data stores. Additionally, you will collaborate closely with Project/Business/Research teams to provide the scraped data for analysis. It will be your duty to maintain the scraping projects that are delivered to production and to create frameworks for automating and ensuring a continuous flow of data from multiple sources. You should be able to work independently with minimal supervision...

As an experienced candidate with 5-8 years in the field, your role will involve the following key responsibilities: - Integrate with AI models using FAST API/Flask API, including Generative AI models, to solve complex problems. - Experience using frameworks like LangChain, Llamalndex, and agent orchestration tools such as LangGraph, AutoGen, and CrewAI. - Deploy and manage applications in cloud environments (AWS, Azure, or GCP), ensuring scalability, resilience, and cost efficiency. - Experience in integrating with full-stack AI-driven applications and modern front-end frameworks like React, Angular, Vue, or Next.js. - Implement monitoring and observability solutions for real-time tracking, ...

Key Responsibilities1 Design, develop, and maintain robust Python scripts for data processing, cleaning, and transformation Work extensively with Pandas for complex data manipulation and reporting Automate Excel workflows using OpenPyXL and win32com libraries Build and optimize web scraping solutions using Selenium (and other scraping tools if required) Maintain and scale existing Python-based applications and automation pipeline Troubleshoot, debug, and optimize performance of data workflows and scraping processes Ensure data accuracy, reliability, and process scalability Collaborate with stakeholders to understand business requirements and deliver automated solutions Take ownership of exis...