83 Scrapy Jobs - Page 4

Position Overview: We are seeking a motivated and experienced Web Scraping Developer with 2+ years of hands-on experience to join our team. The ideal candidate will possess strong technical skills in web scraping, data processing, and performance optimization, with expertise in Python, popular scraping frameworks, and large dataset management. You will play a critical role in extracting and processing data from a variety of websites while ensuring compliance with legal and ethical guidelines. Key Responsibilities: Develop and maintain scalable and optimized Python-based web scraping scripts. Use web scraping frameworks (Scrapy, Beautiful Soup, Selenium, etc.) to extract data from static and dynamic websites. Implement solutions for handling dynamic content using headless browsers like Playwright or Puppeteer. Extract and process data efficiently from complex HTML, CSS, JavaScript, and XPath structures. Work with large datasets, using tools such as Pandas for data manipulation, cleaning, and processing. Ensure proper handling of data export in formats like CSV, JSON, and direct database integration. Consume and integrate REST APIs, manage API rate limits, and handle HTTP protocols, cookies, and sessions. Manage data storage using relational databases like MySQL, optimising queries and indexing for large datasets. Troubleshoot and bypass common anti-scraping techniques such as CAPTCHAs, IP blocking, and user-agent tracking. Use tools and techniques like rotating proxies, headless browsers, and CAPTCHA-solving services to mitigate blocking. Collaborate with teams using version control systems like Git, perform code reviews, and contribute to collaborative workflows. Optimize scraping scripts for performance, including parallel processing or distributed scraping with tools like Celery, Redis, or AWS Lambda. Deploy scraping solutions using tools like Docker, AWS, or Google Cloud, and automate scraping tasks with schedulers (e.g., Cron). Implement robust error-handling mechanisms and monitor scraping jobs with logging frameworks. Stay updated with web scraping trends and ensure that projects comply with web scraping ethics, copyright laws, and website terms of service. Required Qualifications: 2+ years of hands-on experience in web scraping using Python. Strong proficiency in Scrapy, Beautiful Soup, Selenium, or similar scraping frameworks. Experience with headless browsers like Playwright or Puppeteer for handling complex websites. In-depth understanding of HTML, CSS, XPath, and JavaScript for dynamic content interaction. Proficiency in data handling with Pandas and experience in exporting data in multiple formats (CSV, JSON, databases). Strong knowledge of REST APIs and working with web protocols like HTTP, cookies, and session management. Experience in managing MySQL databases, query optimization, and indexing for large-scale data. Familiarity with anti-scraping techniques and proficiency in bypassing measures like CAPTCHAs and IP blocks. Experience with version control tools like Git and familiarity with collaborative workflows and code review processes. Hands-on experience in performance optimization, parallel processing, and distributed scraping. Knowledge of deploying scraping solutions using Docker, AWS, or Google Cloud. Strong problem-solving skills, including error-handling and debugging using logging frameworks. Awareness of web scraping ethics, copyright laws, and legal compliance with website terms of service. Preferred Qualifications: Familiarity with cloud-based solutions like AWS Lambda, Google Cloud, or Azure for distributed scraping. Experience with workflow automation tools and Cron jobs. Basic knowledge of frontend development (HTML, CSS, JavaScript) is a plus.

Dear Candidate, We are looking for a skilled Python Software Developer to design, develop, and optimize high-performance applications. You will work on building scalable back-end services, APIs, and data-driven applications while ensuring code quality and efficiency. If you are passionate about Python development, cloud technologies, and automation , wed love to hear from you! Key Responsibilities: Develop, test, and maintain backend applications and RESTful APIs using Python. Work with Django, Flask, or FastAPI frameworks to build scalable web applications. Optimize performance, security, and scalability of applications. Integrate third-party services, APIs, and data pipelines. Work with SQL and NoSQL databases (PostgreSQL, MySQL, MongoDB, Redis). Implement authentication and authorization mechanisms (JWT, OAuth, LDAP). Write clean, maintainable, and well-documented code following best practices. Conduct unit testing and test-driven development (TDD) using PyTest or Unittest. Implement CI/CD pipelines and automation for efficient deployments. Collaborate with DevOps teams to deploy applications on AWS, Azure, or GCP . Stay updated with Python libraries, frameworks, and industry trends . Required Skills & Qualifications: Strong proficiency in Python and object-oriented programming (OOP) . Experience with Django, Flask, or FastAPI for web application development. Familiarity with SQL and NoSQL databases (PostgreSQL, MySQL, MongoDB, Redis). Knowledge of RESTful APIs, GraphQL, and WebSockets . Experience with Docker and Kubernetes for containerized deployments. Hands-on experience with cloud platforms (AWS, GCP, or Azure). Strong understanding of data structures, algorithms, and system design . Experience with Celery, RabbitMQ, or Kafka for background task processing. Proficiency in Git and version control workflows (GitHub, GitLab, Bitbucket). Familiarity with CI/CD pipelines and automation (Jenkins, GitHub Actions, Azure DevOps). Strong problem-solving, debugging, and analytical skills. Soft Skills: Strong problem-solving and analytical skills. Excellent communication skills to work with cross-functional teams. Ability to work independently and as part of a team. Detail-oriented with a focus on delivering high-quality solutions Note: If you are interested, please share your updated resume and suggest the best number & time to connect with you. If your resume is shortlisted, one of the HR from my team will contact you as soon as possible. Srinivasa Reddy Kandi Delivery Manager Integra Technologies

Job Summary: We are seeking a detail-oriented and analytical Web Analyst who can play the role of Pricing & Content Coordinator with expertise in web scraping tools like Python and VBA to support our e-commerce and pricing initiatives. In this role, you will be responsible for collecting and analyzing online pricing data, managing key value indicator (KVI) items, and enhancing website content to improve customer experience. You will collaborate with cross-functional teams, including Pricing, Data Management, and e-commerce, to drive data-driven decisions and business growth . Essential Job Duties: Develop and implement web scraping scripts using Python (BeautifulSoup, Scrappy, and Selenium) to collect online pricing information from competitor websites and other relevant sources. Use the Spyder IDE for debugging, testing, and optimizing Python scripts for data extraction. Automate the data collection and reporting processes using Python/VBA to streamline workflows and improve efficiency. Collaborate with the Pricing team to establish structured frameworks for recording and analyzing online pricing data. Oversee the weekly management of key value indicator (KVI) items, ensuring accurate and up-to-date data for pricing decisions. Participate in category deep dives to collect regular price and promotional data, providing actionable insights to inform pricing strategies. Gather and maintain accurate item attributes (e.g., product title, descriptions, images, specifications) to enhance site merchandising on bumpertobumper.ca. Use the web scraping tools to extract and update product content from supplier websites or other online sources. Ensure website content is optimized to showcase our leading program of automotive aftermarket parts and tools. Support the integration of pricing and content data during the acquisition of new businesses. Clean, transform, and preprocess raw data using Python libraries like Pandas and NumPy to ensure accuracy and usability. Collaborate with the Data Management team to ensure seamless data integration and reporting. Provide actionable insights to cross-functional teams to improve pricing strategies and customer experience. Work closely with cross-functional teams, including Pricing, E-commerce, and Data Management, to align goals and deliverables. Document processes, workflows, and code for future reference and team collaboration. Required Qualifications & Skills: Basic Qualifications: (Minimum Required Experience, Education, Knowledge/Skills/Abilities, Essential Physical Demands) The candidate must hold a bachelors degree in BBA or B. Com, coupled with prior experience in marketing or pricing roles. Technical Skills: Proficiency in Python for web scraping (BeautifulSoup, Scrapy, Selenium, Requests). Experience with Spyder IDE for Python development. Strong knowledge of VBA for Excel automation and macro development. Familiarity with HTML, CSS, JavaScript, and XPath for web navigation and data extraction. Experience with data cleaning and manipulation using Python libraries like Pandas and NumPy. Education & Experience: Overall, 2 to 4 years of experience in pricing, data collection, or a related analytical role. Strong attention to detail, organizational skills, and the ability to collaborate effectively across teams. Proficiency in Microsoft Office, particularly Excel and Business Intelligence (BI) tools, including the use of MS Excel for calculations of large datasets; experience with Google Analytics is an asset. Excellent written and verbal communication skills in English; proficiency in French is an asset. An interest in the automotive industry is considered a plus. Experience in pricing analysis, competitor benchmarking, or e-commerce data management. Familiarity with web development concepts such as HTML, CSS, WordPress CMS and JavaScript for technical SEO purposes. Ability to work autonomously. High level of organizational skills with the ability to deal with conflicting priorities and deadlines. Hands-on person who can hit the ground running. Flexible with the ability to work well under pressure. Confident in problem-solving. Passionate about Data Analysis & eCommerce. Knowledge/Skills/Abilities: The person needs to fill in ideally following areas: Analytical skills enable the Marketing Assistant to compile data and present it in an easy-to-understand method. Being self-motivated and well-organized with a customer approach. Able to understand the requirements of the job. Ability and capacity to manage multiple tasks based on deadlines. Good understanding of marketing principles.

Implementation of Automation of various production processes using RPA frameworks like UIPath , Openbot, power automate, Python and Azure Design, develop and deploy automation solutions using different tools including LLM/GPT Required Candidate profile 4+ years of hands experience in UIPath and UIPath Document Understanding Experience with Openbot Hands on web scraping Experience in Azure Document Intelligence Proficiency in Python

Hello, Hope you are doing good. Urgent job openings for Python Lead - (Data Scraping Lead) @ GlobalData(Hyd). Job Description given below please go through to understand requirement. if requirement is matching to your profile share your updated resume @ mail id (m.salim@globaldata.com). Mention Subject Line :- Applying for Python Lead - (Data Scraping Lead) @ GlobalData(Hyd) Share your details in the mail :- Full Name : Mobile # : Qualification : Company Name : Designation : Total Work Experience Years : Current CTC : Expected CTC : Notice Period : Current Location/willing to relocate to Hyd? : Office Address : 3rd Floor, Jyoti Pinnacle Building, Opp to Prestige IVY League Appt, Kondapur Road, Hyderabad, Telangana-500081. Job Description :- We are looking for a Data Scraping Lead (Python Lead) to design and oversee large-scale web scraping solutions. The ideal candidate should have extensive experience in Python development, web scraping techniques, and managing data extraction pipelines. Key Responsibilities: Lead the architecture, design, and development of large-scale web scraping projects. Optimize scraping strategies for efficiency, accuracy, and scalability. Manage a team of developers, provide technical guidance, and review code. Handle dynamic content, CAPTCHA solving, and proxy management. Develop and maintain data validation and deduplication processes. Ensure adherence to best practices, including legal and ethical scraping guidelines. Work closely with data engineers and analysts to ensure seamless integration of extracted data. Requirements: 8+ years of experience in Python development with a focus on web scraping. Expertise in frameworks like Scrapy, Selenium, Playwright, or Puppeteer. Strong knowledge of HTML parsing, JavaScript execution, and browser automation. Experience with large-scale data processing Cloud platforms (AWS, GCP, Azure), and containerization (Docker, Kubernetes) is an advantage. Familiarity with database management systems (MSSQL, MySQL). Experience leading teams and mentoring junior developers. Excellent problem-solving and analytical skills. Thanks & Regards, Salim (Human Resources)

Job responsibilities • Innovate and Implement: Design, develop, and deploy AI solutions to optimize design options, streamline engineering workflows, and improve project planning. • Lead and Mentor: Guide AI product development, mentor junior team members, and foster a collaborative work environment. • Collaborate: Work closely with business leaders and end-users to identify AI-driven process improvements and support data-driven decision-making. • Prototype and Pilot: Rapidly prototype and test AI solutions in a fast-paced, agile environment. • Own and Maintain: Take ownership of AI products throughout their lifecycle, including deployment, testing, and ongoing maintenance. • Enhance and Update: Continuously enhance AI models and systems using the latest data, techniques, and advancements in technology. • Review and Recommend: Engage in architectural reviews, recommending scalable AI solutions that align with industry standards. • Stay Informed: Stay updated on the latest advancements in AI research and emerging technologies to bring fresh ideas to the team. Required Skills • Educational Background/Experience: A degree in Computer Science, AI, Machine Learning, or a related technical field, with more than 1yr of relevant corporate experience (Freshers who have recently completed courses in AI or internship experience gained as part of the curriculum shall not be considered). • Programming Skills: Strong proficiency in Python, with experience in AI and ML frameworks like PyTorch and Langchain. • AI Expertise: Proven experience in building and deploying AI/machine learning systems, including NLP and computer vision techniques. • LLM Knowledge: Familiarity with large language models (LLMs) such as OpenAI's GPT and experience with prompt engineering techniques. • Advanced Techniques: Knowledge of methodologies like Retrieve and Generate (RAG) for enhanced data retrieval and generation. • API and Automation: Experience with REST API architecture, database management (MySQL, Chrome DB/Pinecone), and automation tools like BeautifulSoup, Scrapy, and Selenium. • Cloud Platforms: Hands-on experience with cloud platforms like AWS, Azure, or GCP. • Leadership Ability: Proven track record in leading and mentoring technical teams to deliver high-quality solutions. • Communication Skills: Ability to clearly articulate technical concepts to non-technical stakeholders. • Passion for AI: A genuine enthusiasm for leveraging AI to innovate and transform traditional business processes.

knowledge of Python. Utilize Flask or Django for building web applications and RESTful APIs. Knowledge of web scraping libraries like BeautifulSoup or Scrapy. Implement and experiment with Lang Chain for building conversational AI systems.

Expertise in Python, JavaScript, Scrapy, Selenium, Puppeteer, and Beautiful Soup, handling dynamic content, and overcoming anti-crawling measures.