619 Scrapy Jobs - Page 5

Role Overview: You will design, build, and operate software for data collection and processing at scale. The role is hands‑on, with emphasis on clean design, reliability, and performance. About Forage AI: ForageAI builds next‑generation systems for data collection and processing — large‑scale web crawling , document parsing , data pipelines, and automation. We work primarily in Python , leverage cloud‑native designs (mainly AWS , with exposure to GCP/Azure ), and increasingly apply GenAI and AI agents across our stack. Every developer owns their module and collaborates closely with peers in a high‑ownership, high‑trust environment. Location: Remote (Work from Home) Key Responsibilities: · De...

Company Profile: GTC is a leading end-to-end IT services company, helping clients harness the power of technology services and digital transformation to thrive in the evolving world of cloud, digital innovation. Our agile and collaborative approach to creating customized solutions across the digital value chain helps our clients to get faster in the market.Our deep domain expertise in application management turns IT into a strategic asset. Whether you need to run your business more efficiently or accelerate revenue growth, GTC can get you there with utmost qualitative, on-time, and cost-effective software solutions. ns. Position: Python Developer Website : https://gtcsys.com Roles and Respon...

Position: Lead Data Crawling Engineer Location: Sector 23 Dwarka, Delhi This is a Delhi-based position and work from office only!! Job Summary: We are looking for an experienced Lead Data Crawling Engineer with 4–6 years of hands-on expertise in designing, optimizing, and scaling data extraction systems. The ideal candidate should be highly skilled in Python, Selenium, Scrapy, API integrations, SQL, and handling complex, large-scale crawling architectures. You will be responsible for building advanced automated scraping solutions, leading optimization initiatives, and ensuring the highest standards of data quality, efficiency, and compliance. Key Responsibilities: Design, develop, and mainta...

You will be responsible for designing and developing web scraping and automation projects using Node.js and Python. Your key responsibilities will include: - Optimizing web scraping and automation processes for performance and scalability - Troubleshooting and debugging web scraping and automation issues - Collaborating with team members to enhance the overall architecture and performance of the web scraping and automation projects - Staying up-to-date with new technologies and advancements in web scraping and automation Qualifications required for this role: - 2+ years of experience working with web scraping and automation projects - Proficiency in Node.js, Python, JavaScript, and web scrap...

Requisition Id: 1662578 As a global leader in assurance, tax, transaction and advisory services, we hire and develop the most passionate people in their field to help build a better working world. This starts with a culture that believes in giving you the training, opportunities and creative freedom. At EY, we don't just focus on who you are now, but who you can become. We believe that it’s your career and ‘It’s yours to build’ which means potential here is limitless and we'll provide you with motivating and fulfilling experiences throughout your career to help you on the path to becoming your best professional self. The opportunity : Consultant-National-Forensics-ASU - Forensics - Investiga...

As a Senior Python Developer specializing in web scraping and automation, your role will involve designing, developing, and optimizing large-scale scraping solutions using Scrapy. You will collaborate with product managers, QA, and DevOps teams to ensure timely delivery and research and adopt new scraping technologies to improve performance and efficiency. Key Responsibilities: - Design, develop, and maintain scalable web scraping frameworks using Scrapy (mandatory). - Utilize additional libraries/tools such as BeautifulSoup, Selenium, Playwright, Requests, etc. - Implement robust error handling, data parsing, and data storage mechanisms (JSON, CSV, SQL/NoSQL databases). - Build and optimize...

Role & responsibilities As a Data Engineer focused on web crawling and platform data acquisition, you will design, develop, and maintain large-scale web scraping pipelines to extract valuable platform data. You will be responsible for implementing scalable and resilient data extraction solutions, ensuring seamless data retrieval while working with proxy management, anti-bot bypass techniques, and data parsing. Optimizing scraping workflows for performance, reliability, and efficiency will be a key part of your role. Additionally, you will ensure that all extracted data maintains high quality and integrity. Preferred candidate profile We are seeking candidates with: Strong experience in Pytho...

Python Automation & Web Scraping Engineer (WFH) Experience: 2–4 Years Location: Remote Mode of Engagement: Full-time No of Positions: 3 Educational Qualifications: Bachelor’s degree in Computer Science / Information Technology Industry: IT / Software Development Notice Period: Immediate Joiners Preferred What We Are Looking For: Strong hands-on experience in Python for automation and mid-scale web scraping projects. Experience with Selenium, BeautifulSoup, Requests , and automation-driven scraping workflows. Good understanding of HTML, CSS, DOM structures , and dynamic/JavaScript-rendered websites. Experience in backend/API development using Flask or Django . Ability to handle cookies, sessi...

As a GTM Engineer Intern, you will help us accelerate customer acquisition, improve funnel performance, and enable sales & marketing teams with data-driven automation . What you will do ? 1. Growth Engineering & Automation Build low-code/no-code automations across HubSpot, CRM, email, product, and website. Design & automate lead scoring, onboarding flows, and lifecycle journeys. 2. GTM Experiments & Funnel Optimization Run experiments across website, landing pages, ads, product onboarding, and outreach. Build funnel dashboards and analyze drop-off, activation, retention metrics. Contribute to inbound & outbound GTM motions. 3. Revenue Operations (RevOps) Set up attribution models, GTM dashbo...

Position: Data Engineer Experience: 6 Months to 3 Years Location: Remote/On-site (As applicable) Joining: Immediate Joiners Preferred About the Role: We are looking for a skilled Data Engineer with strong Python expertise and hands-on experience in handling large datasets, data cleaning, analysis, and visualization. The ideal candidate should be capable of building data pipelines, performing web scraping, and generating actionable insights through dashboards and reports. Key Responsibilities: Develop, test, and maintain efficient Python scripts for data collection and transformation. Design and implement web scraping solutions to extract structured and unstructured data from diverse sources....

You are sought after for the role of Senior Python Developer with expertise in web scraping and automation. Your responsibilities will include designing, developing, and optimizing large-scale scraping solutions using Scrapy and collaborating with various teams to ensure timely delivery. - Design, develop, and maintain scalable web scraping frameworks using Scrapy. - Utilize additional libraries/tools like BeautifulSoup, Selenium, Playwright, Requests, etc. - Implement robust error handling, data parsing, and storage mechanisms. - Build and optimize asynchronous scraping workflows, handling multithreading/multiprocessing. - Research and adopt new scraping technologies to enhance performance ...

Position Overview: We are seeking a highly skilled Web Scraping Architect to join our team. The successful candidate will be responsible for designing, implementing, and maintaining web scraping processes to gather data from various online sources efficiently and accurately. As a Web Scraping Specialist, you will play a crucial role in collecting data for competitor analysis, and other business intelligence purposes. Responsibilities: Scalability/Performance: Lead and provide expertise in scraping at scale e-commerce marketplaces. Data Source Identification: Identify relevant websites and online sources from which data needs to be scraped. Collaborate with the team to understand data require...

Position: Lead Data Crawling Engineer Location: Sector 23 Dwarka, Delhi This is a Delhi-based position and work from office only!! Job Summary: We are looking for an experienced Lead Data Crawling Engineer with 4–6 years of hands-on expertise in designing, optimizing, and scaling data extraction systems. The ideal candidate should be highly skilled in Python, Selenium, Scrapy, API integrations, SQL, and handling complex, large-scale crawling architectures. You will be responsible for building advanced automated scraping solutions, leading optimization initiatives, and ensuring the highest standards of data quality, efficiency, and compliance. Key Responsibilities: Design, develop, and mainta...

Role Overview: As a Node.js Engineer at DataWeave, you will be responsible for designing and developing web scraping and automation projects using Node.js and Python. Your main focus will be on optimizing processes for performance and scalability, troubleshooting and debugging issues, and collaborating with team members to enhance project architecture and performance. You will also be expected to stay updated with new technologies and advancements in web scraping and automation. Key Responsibilities: - Design and develop web scraping and automation projects using Node.js and Python - Optimize web scraping and automation processes for performance and scalability - Troubleshoot and debug web s...

Analyze performance data from various marketplaces to uncover trends, optimize listings, and improve conversion rates. Track and report KPIs such as sales velocity, click-through rates, return rates, and competitor pricing. Develop and maintain interactive dashboards and reports using tools such as Power BI, Tableau, or Looker. Analyze business performance, identify trends, and provide recommendations to improve key metrics. Product Listing Management Audit and optimize product listings including titles, descriptions, keywords, pricing, and images for maximum visibility and conversions. Conduct A/B testing on product attributes and monitor performance improvements. Ensure listings meet the c...

Job Title: Python Trainer (with Web Scraping Experience) Experience Required: 5+ Years (in Python training & mentoring) Employment Type: Full-time Key Responsibilities Conduct comprehensive Python training sessions for interns and entry-level engineers. Design, plan, and deliver training modules, assignments, and practical projects. Teach core Python concepts including: Data types, loops, conditionals, and functions OOP (Object-Oriented Programming) File handling and exception handling Modules, packages, and virtual environments Cover advanced topics such as: Web scraping using requests, BeautifulSoup, Scrapy, Selenium, and Playwright Database integration (MySQL, MongoDB) REST API integratio...

Role Overview: As a Python Developer with web scraping expertise, your main responsibility will be to design, develop, and maintain web scraping scripts using Python. You will be using web scraping libraries like Beautiful Soup, Scrapy, Selenium, and other tools to extract data from various websites. Your role will involve writing reusable, testable, and efficient code to extract both structured and unstructured data. Additionally, you will need to collaborate with software developers, data scientists, and other stakeholders to plan, design, develop, and launch new web scraping projects. Troubleshooting, debugging, and optimizing web scraping scripts will also be part of your responsibilitie...

As a Web Scraping Engineer at noon, your primary responsibility will be to design and implement robust scrapers to collect, clean, and normalize product data from e-commerce platforms. You will play a crucial role in building, maintaining, and scaling data extraction systems to support our analytics and competitive pricing systems. Key Responsibilities: - Develop and maintain scalable scraping pipelines to extract structured data from complex, dynamic e-commerce sites. - Design and manage distributed scraping infrastructure, including rotating proxies and request throttling. - Handle dynamic rendering, anti-bot measures, and API integrations when necessary. - Implement data quality validatio...

We are seeking a skilled Web Scraping Developer with strong expertise in building scrapers, crawlers, and data extraction pipelines for e-commerce and quick-commerce platforms. The ideal candidate will have hands-on experience in large-scale product data extraction, including pricing, rankings, product descriptions, and catalog details. Key Responsibilities Design, develop, and maintain scalable scrapers/crawlers for e-commerce websites. Extract structured data such as product listings, pricing, descriptions, reviews, and rankings. Handle dynamic content, cookies, pagination, AJAX, and anti-bot challenges efficiently. Ensure data accuracy, integrity, and scalability of scraping pipelines. Op...

Role & responsibilities Application Development: Design and build web-scale applications using Python (Django/Flask/FastAPI) and modern frontend frameworks (React / Angular / Vue). API Development: Develop and integrate RESTful and GraphQL APIs. Implement responsive, intuitive, and interactive user interfaces. Build scalable and secure backend systems with optimized database queries and caching strategies. Create and manage relational (PostgreSQL/MySQL) and NoSQL (MongoDB/Redis) databases. Deploy applications on AWS / Azure / GCP, including containerization (Docker, Kubernetes). Write clean, reusable, and maintainable code following industry best practices. Ensure system performance, securit...

Build, automate, and optimize business processes across multiple systems using scripts, APIs, and automation tools. Develop and maintain ETL pipelines to collect, clean, and integrate data from websites, APIs, and internal systems. Take full ownership of automation projects end-to-end — from understanding the problem to designing the solution, building it, monitoring it, and improving it. Create web automation and data extraction workflows using Python (Selenium, BeautifulSoup, Scrapy) or similar tools. Integrate different platforms and tools to improve operational efficiency and reduce manual work. Document processes, maintain system reliability, and proactively identify opportunities to au...

Python Developer – Web Scraping & Automation (WFH) Experience: 3-5 Years Location: Remote Mode of Engagement: Full-time No of Positions: 3 Educational Qualifications: Bachelor’s degree in Computer Science / Information Technology Industry: IT / Software Development Notice Period: Immediate Joiners Preferred What We Are Looking For: Strong hands-on experience in Python for large-scale web scraping and automation. Expert in web scraping using Scrapy, BeautifulSoup, Selenium, Playwright, and Requests. Excellent understanding of HTML, CSS, DOM structure , and browser-rendered content. Experience in backend/API development using FastAPI, Django, or Flask . Ability to handle proxies, user-agent ro...

Location: Remote Type: Full-time Experience : 3+ Years Salary: up to 60K/Month Role Summary We are looking for a hands-on AI Data Engineer who can independently manage end-to-end data workflows, including data collection, document processing, dataset preparation, retrieval pipelines, model fine-tuning, and data visualization. This role requires strong technical skills across Python, automation, ML tooling, and analytical reporting. Key Responsibilities (Technical) 1. Data Acquisition & Automation Build automated data collection workflows using tools such as Firecrawl , Playwright , Scrapy , or similar frameworks Extract multi-format documents (PDFs, HTML, text, images) Handle large-scale cra...

Job Title: Founding Data Engineer (ETL & Web Integration)- 6 Months Contract About Us At GoDiverse, we're on a mission to build a fairer and more efficient economy for small businesses. We are an early-stage tech company building an AI-powered platform to help enterprises discover and engage with innovative Small and Medium-sized Enterprises (SMEs). Our goal is to level the playing field, ensuring that smaller, dynamic companies get fair access to large-scale commercial opportunities. The Role We are looking for our Founding Data Engineer to build and own the entire data acquisition and preparation lifecycle for our platform. This is a hardcore, hands-on engineering role with no project mana...

You will be working at Rysysth Technologies, a growing IT company specializing in Microsoft technologies and digital solutions. They focus on SharePoint SPFx development, Power Automate solutions, Azure cloud services, Copilot Studio implementations, and custom web development. The company is striving to become the preferred technology partner for digital transformation. As a Python Developer AI & Web Scraper, your main responsibilities will include: - Developing reliable web scraping scripts and automation tools using Python and related frameworks. - Building scalable data extraction systems for structured and unstructured data. - Integrating AI/ML models for advanced text/data parsing. - H...