74 Web Crawling Jobs

Role Overview: You will be responsible for developing the Policies & Next Gen Rule Engine for Digital Lending. Additionally, you will need to develop various RestFul APIs & integrate them with various data sources. Another key aspect of your role will involve developing various reports & automating tasks required for the business. Your proficiency in programming using Python & DB will be crucial for this role. Moreover, having experience in Numpy & Pandas will be beneficial. Key Responsibilities: - Develop Policies & Next Gen Rule Engine for Digital Lending - Develop various RestFul APIs & integrate with different data sources - Develop various reports & automate tasks necessary for the busi...

We are looking for a skilled Production Manager to lead our Scrap Ops team in the Hotels & Restaurants industry. The ideal candidate will have 3-7 years of experience and a strong background in production management. Roles and Responsibility Manage and oversee the scrap operations process to ensure efficient and effective waste management. Develop and implement strategies to reduce scrap and increase recycling rates. Collaborate with cross-functional teams to identify areas for improvement and optimize processes. Analyze data and metrics to measure performance and inform decision-making. Ensure compliance with regulatory requirements and industry standards. Lead and motivate a team of profes...

Collaborate with cross-functional teams to define, design, and implement new features. Ensure the performance, quality, and responsiveness of web crawling systems. Identify and correct bottlenecks and fix bugs in web crawling processes. Help maintain code quality, organization, and automation. Stay up-to-date with the latest industry trends and technologies. Skills: 3+ years of experience in Web Scraping or Crawling through Scrapy, Selenium or other frameworks and related libraries (like beautifulsoup, puppeteer). Should be expert on latest version of Python. Should have very good experience on fetching data from multiple online sources, cleanse it and build APIs on top of it. Good understan...

Overview Stats Perform is the market leader in sports tech. We provide the most trusted sports data to some of the world's biggest organizations, across sports, media, and broadcasting. Through the latest AI technologies and machine learning, we combine decades' worth of data with the latest in-game happenings. We then offer coaches, teams, professional bodies, and media channels around the world, access to the very best data, content, and insights. In turn, improving how sports fans interact with their favorite sports teams and competitions. How do they use it? Media outlets add a little magic to their coverage with our stats and graphics packages. Sportsbooks can offer better predictions a...

Role & responsibilities As a Data Engineer focused on web crawling and platform data acquisition, you will design, develop, and maintain large-scale web scraping pipelines to extract valuable platform data. You will be responsible for implementing scalable and resilient data extraction solutions, ensuring seamless data retrieval while working with proxy management, anti-bot bypass techniques, and data parsing. Optimizing scraping workflows for performance, reliability, and efficiency will be a key part of your role. Additionally, you will ensure that all extracted data maintains high quality and integrity. Preferred candidate profile We are seeking candidates with: Strong experience in Pytho...

Building and maintaining advanced web scraping and automation scripts across platforms (LinkedIn, Facebook, Twitter, Discord). Handle CAPTCHA, proxy, IP rotation, and anti-blocking. Ensure deployable, scalable, and monitored solutions using Docker.

We are seeking a skilled Python Developer with hands-on experience in web scraping to join our data engineering team. The ideal candidate will be responsible for building robust scraping pipelines, extracting structured data from various websites, and ensuring data quality and compliance with ethical scraping practices. Key Responsibilities Develop and maintain scalable web scraping scripts using Python libraries such as BeautifulSoup, Scrapy, Selenium, or Playwright. Handle dynamic content, pagination, and authentication mechanisms in scraping workflows. Clean, transform, and store scraped data in structured formats (CSV, JSON, databases). Monitor scraping jobs and troubleshoot failures or ...

This role entails building and managing the universe of web content resources that are of a high interest to AlphaSense clients. You will be responsible for the end-to-end process of discovering, onboarding, and monitoring web content ingestion across multiple business domains. You will also play a key role in evolving AlphaSense web processing capabilities. Responsibilities: Web content research : Perform research to discover new web sources based on the established content onboarding policies or client requests. Content Analysis: Evaluate requested web sources to identify content sets that need to be onboarded. Content Onboarding: Configure new web crawling/scraping jobs via admin interfac...

As a Senior Software Engineer at CUBE, you will have the opportunity to be part of a pioneering RegTech business that is transforming machine automated compliance in the financial services industry. You will work with our lead engineer to develop product features using .NET and Cloud-based technologies. Your responsibilities will involve expanding and optimizing content processing solutions, designing and implementing new products and features, troubleshooting, and investigating for product sustainment in a fast-paced environment. Key Responsibilities: - Collaborate with the lead engineer to develop product features using .NET and Cloud-based technologies - Expand and optimize content proces...

Disclaimer: Candidates with pure e-commerce or online retail operations experience will be preferred. Backgrounds limited to logistics or warehousing operations will not be considered for this role Position: Asst. Manager/ Deputy Manager / Manager / Sr. Manager E-Commerce Operations Drive Excellence, Optimize Operations, and Lead the Future of E-Commerce at Contexio! Job Summary Shape the Future of E-Commerce Operations with Contexio We are looking for a dynamic and results-driven Operations Manager Retail Solutions to strategize, manage, and execute high-impact projects in the e-commerce space. This role is ideal for a leader with a strong process management mindset, client servicing expert...

About The Role Entity :- Accenture Strategy & Consulting Team :- Strategy & Consulting Global Network Practice :- Responsible AI COE Title :- Responsible AI Manager Job location :- Bangalore/Gurgaon/Pune/ Hyderabad/Chennai/Mumbai The rapid development of AI is creating new opportunities to improve the lives of people around the world, from business to healthcare to education. As a result, it is also raising new questions about the best way to build fairness, interpretability, privacy and security into these systems. The Data and AI revolution is changing everything. Its everywhere transforming how we work and play. Join Accenture and help transform leading companies and communities around th...

About The Role Entity :- Accenture Strategy & Consulting Team :- Strategy & Consulting Global Network Practice :- Responsible AI COE Title :- Responsible AI Specialist/ Sr. Analyst Job location :- Bangalore/Gurgaon/Pune/ Hyderabad/Chennai/Mumbai The rapid development of AI is creating new opportunities to improve the lives of people around the world, from business to healthcare to education. As a result, it is also raising new questions about the best way to build fairness, interpretability, privacy and security into these systems. The Data and AI revolution is changing everything. Its everywhere transforming how we work and play. Join Accenture and help transform leading companies and commu...

About The Role Entity :- Accenture Strategy & Consulting Team :- Strategy & Consulting Global Network Practice :- Responsible AI COE Title :- Responsible AI Specialist/ Sr. Analyst Job location :- Bangalore/Gurgaon/Pune/ Hyderabad/Chennai/Mumbai The rapid development of AI is creating new opportunities to improve the lives of people around the world, from business to healthcare to education. As a result, it is also raising new questions about the best way to build fairness, interpretability, privacy and security into these systems. The Data and AI revolution is changing everything. Its everywhere transforming how we work and play. Join Accenture and help transform leading companies and commu...

Entity :- Accenture Strategy & Consulting Team :- Strategy & Consulting Global Network Practice :- Responsible AI COE Title :- Responsible AI Specialist/ Sr. Analyst Job location :- Bangalore/Gurgaon/Pune/ Hyderabad/Chennai/Mumbai The rapid development of AI is creating new opportunities to improve the lives of people around the world, from business to healthcare to education. As a result, it is also raising new questions about the best way to build fairness, interpretability, privacy and security into these systems. The Data and AI revolution is changing everything. Its everywhere transforming how we work and play. Join Accenture and help transform leading companies and communities around t...

We are seeking an experienced Web Scraping Engineer with deep expertise in Scrapy to develop, maintain, and optimize web crawlers. The ideal candidate will have a strong background in extracting, processing, and managing large-scale web data efficiently. Responsibilities : - Write and maintain web scraping scripts using Python Optimize custom web scraping tools and workflows. - Familiarity with Python and web scraping frameworks (Scrapy, Selenium, BeautifulSoup, Requests, Playwright). - Troubleshoot and resolve scraping challenges, including CAPTCHAs, rate limiting, and IP blocking. - Experience with proxy management (Rotating IPs, VPNs, Residential proxies, etc.) - Handle dynamic content us...

Role Overview: As a Web Scraper, your main responsibility is to apply your expertise to extract data from various online sources. You will be tasked with developing reliable web scrapers and parsers for different websites. Your role involves extracting structured/unstructured data and storing it in SQL/NoSQL data stores. Additionally, you will collaborate closely with Project/Business/Research teams to provide the scraped data for analysis. It will be your duty to maintain the scraping projects that are delivered to production and to create frameworks for automating and ensuring a continuous flow of data from multiple sources. You should be able to work independently with minimal supervision...

As an experienced candidate with 5-8 years in the field, your role will involve the following key responsibilities: - Integrate with AI models using FAST API/Flask API, including Generative AI models, to solve complex problems. - Experience using frameworks like LangChain, Llamalndex, and agent orchestration tools such as LangGraph, AutoGen, and CrewAI. - Deploy and manage applications in cloud environments (AWS, Azure, or GCP), ensuring scalability, resilience, and cost efficiency. - Experience in integrating with full-stack AI-driven applications and modern front-end frameworks like React, Angular, Vue, or Next.js. - Implement monitoring and observability solutions for real-time tracking, ...

Key Responsibilities1 Design, develop, and maintain robust Python scripts for data processing, cleaning, and transformation Work extensively with Pandas for complex data manipulation and reporting Automate Excel workflows using OpenPyXL and win32com libraries Build and optimize web scraping solutions using Selenium (and other scraping tools if required) Maintain and scale existing Python-based applications and automation pipeline Troubleshoot, debug, and optimize performance of data workflows and scraping processes Ensure data accuracy, reliability, and process scalability Collaborate with stakeholders to understand business requirements and deliver automated solutions Take ownership of exis...

**Job Description:** **Role Overview:** As a Solutions Consultant (Sales) at Turbolab Technologies, you will play a proactive and detail-oriented role in supporting Product Growth/Sales initiatives. This Mid-Senior level position is ideal for someone with a minimum of 3 years of experience, preferably in SaaS solutions, who is enthusiastic about learning, developing, and contributing within a fast-paced, data-driven environment. **Key Responsibilities:** - Identify and generate business development efforts through targeted client outreach, fostering relationships via personalized messaging, product education, or campaign tie-ins. - Conduct market research to pinpoint trends and consumer beha...

Responsibilities: Work closely with our lead engineer and develop product features using .NET and cloud-based technologies. Handle the expansion and optimization of our content processing solutions to accommodate growing needs. Undertake the design and implementation of new products, enhancements to existing products, as well as troubleshooting and investigations as part of product sustainment. Be effective in a fast-paced environment. Essential Skills: 5+ years of experience as an Application Engineer delivering high-throughput, scalable, complex web-based applications. Extensive experience in .NET/C#. Strong exposure to serverless programming (Azure Functions or AWS Lambdas). Understanding...

We are hiring Backend Python Developer, with a strong focus on web development and automation along with web crawling/scraping, Selenium, RESTful APIs/FastAPI, Docker, Kubernetes, MS SQL Server and MongoDB exp. Years of experience - 5 to 8 Location - Hyderabad Interested candidates pleas drop your resumes to praveenkumard@ritefit.ai

Total Experience : 1+ years Mode of Hire: Permanent Required Skills Set (Mandatory): JavaScript/Node.js/Python, Web Crawling (Puppeteer/Playwright), Strong knowledge of web technologies Desired Skills (Good if you have): HTML, Distributed Systems, Data pipelines, Proxy Management, Anti-bot Techniques Job Responsibilities Design scalable web crawling solutions using modern technologies like Puppeteer and Playwright. Develop optimal and cost-effective methods for efficiently crawling diverse websites at scale. Implement robust proxy management and sophisticated anti-bot techniques. Create comprehensive debugging and monitoring systems for crawler performance. Actively contribute to technical d...

About the company: Gigmo Solutions is a fast growing tech. startup with a mission to fundamentally disrupt customer support industry through perfectly tuned symphony of Artificial Intelligence based conversational bots and Gig workers. With our engineers spread out in 10+ countries across 4 continents, Gigmos is uniquely poised to fundamentally change the technical customers support industry. Website: https://Gigmos.com LinkedIn:https://www.linkedin.com/company/14543064 Work Location/Shift Timings- Gurgaon Sec-54 (Work from office) Mon to Fri, 9.30 AM to 6.30 PM Job description for a Python Developer with web scraping expertise: Design, develop, and maintain web scraping scripts using Python...

Role Overview: As a highly skilled Backend Python Developer with a strong background in web development and automation, you will play a key role in designing, developing, and maintaining scalable backend solutions that power critical applications and automation processes. Your expertise in web crawling/scraping, Selenium, RESTful APIs/FastAPI, Docker, Kubernetes, MS SQL Server, and MongoDB will be crucial in ensuring the success of the projects. Key Responsibilities: - Design, develop, and maintain scalable backend applications using Python and modern frameworks. - Build and optimize RESTful APIs and microservices with FastAPI. - Develop and maintain robust web crawling and scraping solution...

Must Have: Java, Javascript, Groovy & Grails, Play framework, MySQL, Solr, Kafka, Docker, Linux & Jasper Reports. Nice to have: web crawling, Bootstrap & responsive design experience. Hands-on with REST APIs and scalable systems.