110 Web Scraping Jobs - Page 4

seeking Data Analyst / Data Engineer to support our data driven sales and marketing efforts and will play a crucial role in automating business data extraction , analyzing customer and lead data, and optimizing telemarketing campaigns.

Hello, Greetings from GlobalData!!! Hope you are doing good. Urgent job openings for Python - (Web scraping/crawling) for below roles @ GD-Hyderabad :- Senior Software Engineer - 5+ yrs experience Team Lead - 7+ yrs exp Assistant Manager - 10+ yrs exp Job Description given below please go through to understand requirement. If interested to apply please share your updated resume @ mail id (m.salim@globaldata.com). Mention Subject Line :-Applying for Python - (Web Scraping) @ GlobalData(Hyd) Share your details in the mail :- Full Name : Mobile # : Qualification : Company Name : Designation : Total Work Experience Years : Current CTC : Expected CTC : Notice Period : Current Location/willing to relocate to Hyd? : Job Description: We are seeking a highly skilled and motivated Python Web Crawling Developer with 3 to 8+ years of hands-on experience in web scraping and data extraction. The ideal candidate should have a solid background in Python-based scraping tools and libraries, and a proven track record of working on dynamic websites. Key Responsibilities: Design, develop, and maintain scalable web crawlers and scrapers. Extract and process data from various static and dynamic websites. Work with tools and libraries like Scrapy , Requests , BeautifulSoup , and Selenium . Maintain and optimize existing full-stack web scraping applications. Write and maintain automation scripts for data gathering and processing. Ensure data quality, performance, and reliability of scraping solutions. Collaborate with cross-functional teams for integration and analytics use cases. Required Skills: Strong proficiency in Python programming. In-depth experience with web scraping frameworks (Scrapy, Requests, BeautifulSoup, Selenium). Experience in handling dynamic content scraping (JavaScript-rendered pages). Working knowledge of SQL and database programming. Good analytical and problem-solving skills. Strong verbal and written communication skills. Experience in Python automation scripting. Good to Have: Exposure to Natural Language Processing (NLP) tools like NLTK . Basic understanding or hands-on experience with Machine Learning . Experience with Python web frameworks like Django , Flask , or FastAPI . Thanks & Regards, Salim (Human Resources)

Hi, With reference to your profile on job portal we would like to share an opportunity with you for one of our Gurgaon Based client for Gurgaon location. Please find below the details regarding same: Location: Remote/WFH Experience: 6-10 Years Title: Manager-Data Engineer (Web Scraping) Notice Period: Only Immediate Joiner - 30 Days Max Job Responsibilities Technical Skills Required: Proficiency in Python and SQL/Database skills is required. Must have strong expertise in using Pandas library (Python). Experience with web technologies (HTML/JS, APIs, etc.) is essential. Should have a good understanding of tools such as Scrapy, BeautifulSoup, and Selenium. Responsible for reviewing and approving pull requests to ensure clean, maintainable, and efficient code. Experience building scalable scraping solutions for large-scale data collection Familiarity with AWS technologies like S3, RDS, SNS, SQS, Lambda, and others is necessary. Qualifications Bachelors/masters degree in computer science or in any related field. Role Summary Leading and mentoring a team of seasoned Data engineers performing Web Scraping using various scraping techniques and then utilizing Pythons Pandas library for data cleaning and manipulation. Then ingesting the data into a Database/Warehouse, and scheduling the scrapers using Airflow or other tools Role Overview The Web Scraping Team is seeking a creative and detail-oriented Leaders to contribute to client projects and lead by examples. This team develops essential applications, datasets, and alerts that directly support client investment decisions. Our focus is to maintain operational excellence by providing high-quality proprietary datasets, timely notifications, and exceptional service. The ideal candidate will be self-motivated, self-sufficient, and possess a passion for tinkering and a love for automation. If in case you are interested to avail this opportunity then please revert with your updated profile asap to dbetal@vanassociates.com Note: Do not change the subject line while reverting. 1. Total Exp: 2. Relevant experience in Python, Pandas, Data Cleansing, Data Transformation, Team Management: 3. Current CTC: 4. Expected CTC: 5. Official Notice Period: 6. Ready to work in Gurgaon: 7. Availability for MS Teams Interviews in Weekdays: Thanks & Regards, Devanshu Betal Vanguard HR Associates Pvt. Ltd. Phone No- +918851081580 E-Mail:- dbetal@vanassociates.com

Write effective, scalable code Develop back-end components to improve responsiveness and overall performance Integrate user-facing elements into applications Test and debug programs Improve functionality of existing systems Required Candidate profile Expertise in at least one popular Python framework (like Django, Flask or Pyramid) Familiarity with front-end technologies (like JavaScript and HTML5) Team spirit Good problem-solving skills Perks and benefits Free meals and snacks. Bonus. Vision insurance.

Write effective, scalable code Develop back-end components to improve responsiveness and overall performance Integrate user-facing elements into applications Test and debug programs Improve functionality of existing systems Required Candidate profile Expertise in at least one popular Python framework (like Django, Flask or Pyramid) Familiarity with front-end technologies (like JavaScript and HTML5) Team spirit Good problem-solving skills Perks and benefits Free meals and snacks. Bonus. Vision insurance.

We are seeking a Market Research Analyst with 5-7 years of professional work experience to support our data & marketing teams in gathering, analyzing & visualizing business data. Required Candidate profile Strong proficiency in Adv. Excel, SQL, Python (BeautifulSoup/Selenium), Tableau, Data Visualization ,Statistical Analysis, Web scraping tools (ParseHub, Octoparse), SaaS Research and Analysis

Note: Please don't apply if you do not have at least 3 years of Scrapy experience. We are seeking a highly experienced Web Scraping Expert specialising in Scrapy-based web scraping and large-scale data extraction. This role is focused on building and optimizing web crawlers, handling anti-scraping measures, and ensuring efficient data pipelines for structured data collection. The ideal candidate will have 5+ years of hands-on experience developing Scrapy-based scraping solutions, implementing advanced evasion techniques, and managing high-volume web data extraction. You will collaborate with a cross-functional team to design, implement, and optimize scalable scraping systems that deliver high-quality, structured data for critical business needs. Key Responsibilities Scrapy-based Web Scraping Development Develop and maintain scalable web crawlers using Scrapy to extract structured data from diverse sources. Optimize Scrapy spiders for efficiency, reliability, and speed while minimizing detection risks. Handle dynamic content using middlewares, browser-based scraping (Playwright/Selenium), and API integrations. Implement proxy rotation, user-agent switching, and CAPTCHA solving techniques to bypass anti-bot measures. Advanced Anti-Scraping Evasion Techniques Utilize AI-driven approaches to adapt to bot detection and prevent blocks. Implement headless browser automation and request-mimicking strategies to mimic human behavior. Data Processing & Pipeline Management Extract, clean, and structure large-scale web data into structured formats like JSON, CSV, and databases. Optimize Scrapy pipelines for high-speed data processing and storage in MongoDB, PostgreSQL, or cloud storage (AWS S3). Code Quality & Performance Optimization Write clean, well-structured, and maintainable Python code for scraping solutions. Implement automated testing for data accuracy and scraper reliability. Continuously improve crawler efficiency by minimizing IP bans, request delays, and resource consumption. Required Skills and Experience Technical Expertise 5+ years of professional experience in Python development with a focus on web scraping. Proficiency in using Scrapy based scraping Strong understanding of HTML, CSS, JavaScript, and browser behavior. Experience with Docker will be a plus Expertise in handling APIs (RESTful and GraphQL) for data extraction. Proficiency in database systems like MongoDB, PostgreSQL Strong knowledge of version control systems like Git and collaboration platforms like GitHub. Key Attributes Strong problem-solving and analytical skills, with a focus on efficient solutions for complex scraping challenges. Excellent communication skills, both written and verbal. A passion for data and a keen eye for detail Why Join Us? Work on cutting-edge scraping technologies and AI-driven solutions. Collaborate with a team of talented professionals in a growth-driven environment. Opportunity to influence the development of data-driven business strategies through advanced scraping techniques. Competitive compensation and benefits.

Who We Are Applied Materials is the global leader in materials engineering solutions used to produce virtually every new chip and advanced display in the world. We design, build and service cutting-edge equipment that helps our customers manufacture display and semiconductor chips- the brains of devices we use every day. As the foundation of the global electronics industry, Applied enables the exciting technologies that literally connect our world- like AI and IoT. If you want to work beyond the cutting-edge, continuously pushing the boundaries of"science and engineering to make possible"the next generations of technology, join us to Make Possible® a Better Future. What We Offer Location: Bangalore,IND At Applied, we prioritize the well-being of you and your family and encourage you to bring your best self to work. Your happiness, health, and resiliency are at the core of our benefits and wellness programs. Our robust total rewards package makes it easier to take care of your whole self and your whole family. Were committed to providing programs and support that encourage personal and professional growth and care for you at work, at home, or wherever you may go. Learn more about our benefits . Youll also benefit from a supportive work culture that encourages you to learn, develop and grow your career as you take on challenges and drive innovative solutions for our customers."We empower our team to push the boundaries of what is possible"”while learning every day in a supportive leading global company. Visit our Careers website to learn more about careers at Applied. Job Expectations The candidate will be responsible for leading a team of data scientists who provide analytics services for Applied Materials installed base. The deliverables include developing new service capabilities, piloting them, and commercializing them in partnership with Engineering and Service Product Managers. The team will work with field engineers and product engineering teams to understand the requirements, bring forward creative ideas, develop proofs-of-concept, architect, design, develop, and modify algorithms into production code, provide production support, and train end-users. The skill sets in the team include descriptive statistical analysis, predictive statistical analysis using AI/ML, data visualization and analytics process automation, data cleansing, complex image processing, and text processing. Candidate should be willing to learn and adopt semiconductor industry as their career domain. Key Responsibilities Responsible for managing completion of assignments, projects and programs to support Applieds service business. Scope of algorithm development includes research, design, code development, implementation, and proliferation. Execute projects as needed to support the business. Review and monitor progress to milestones on development programs. Develop roadmaps for algorithmic development programs. Oversee algorithmic concept and feasibility for algorithmic modules, including problem statement definition, data gathering, literature review, concept selection, risks, and implementation constraints. Oversee documentation of algorithmic development and deployment, including integration into required systems, user testing, and user training. Oversee software and hardware implementation. Interact with internal and external customers to define gaps, identify opportunities, define program scope and deliverables, and proliferate solutions to the user base. Present to management for project reviews, interact with project stakeholders, run regular cadence meetings and work in alignment with team and organization goals. Responsible for technical development of team, objective setting, and performance management. Develop growth plan for the team, including identification of new areas of impact. Preferred programming and data science skills includePython, C++, Unix, Image Processing, Deep Learning, AI/ML, NLP, GenAI, Text Mining, Database Design and Management, Web Scraping, GPU Optimization. Proficient in business processes and software such as Microsoft Word/ Excel/ Powerpoint/ Teams, Atlassian JIRA and Confluence. Highly organized and detail-oriented. Ability to build and maintain positive and productive inter-departmental working relationships. Ability to work in a cross-functional organization and multitask on multiple projects. Drive team members to deliver programs on time and on budget. Excellent oral and written communication, organizational, analytical, and interpersonal skills. Interest in building a career in the semiconductor industry. Functional Knowledge Demonstrates comprehensive understanding of concepts and principles within own job family and knowledge of other related job families. Business Expertise Applies in-depth understanding of how own discipline integrates within the segment/function. Leadership Manages multiple related teams, sets organizational priorities and allocates resources. Problem Solving Identifies and resolves complex technical, operational and organizational problems. Impact Impacts the business results of a team or area by supporting and funding of projects, products, services and/or technologies and developing policies and plans. Guided by business unit, department or sub-functional business plans. Interpersonal Skills Influences others internally and externally, including senior management. Position requires understanding of Applied Materials global Standards of Business Conduct and compliance with these standards at all times. This includes demonstrating the highest level of ethical conduct reflecting Applied Materials core values. Education Bachelors, Masters, or Ph.D. Degree in Computer Science, Mathematics, or Engineering with a concentration in data science or AI/ML. Experience 15 years of experience Computer Science/ Mathematics/ Engineering background with 15 years of experience in performing statistical analysis, designing and developing Image Processing/ Computer Vision Algorithms, handling and analyzing large volumes of data. Semiconductor background is an added advantage. Prior team leadership experience is required. Additional Information Time Type: Full time Employee Type: Assignee / Regular Travel: Yes, 20% of the Time Relocation Eligible: Yes Applied Materials is an Equal Opportunity Employer. Qualified applicants will receive consideration for employment without regard to race, color, national origin, citizenship, ancestry, religion, creed, sex, sexual orientation, gender identity, age, disability, veteran or military status, or any other basis prohibited by law.

Summary To enhance user profiling and risk assessment, we are building web crawlers to collect relevant user data from third-party sources, forums, and the dark web. We are seeking a Senior Web Crawler & Data Extraction Engineer to design and implement these data collection solutions. Job Responsibilities Design, develop, and maintain web crawlers and scrapers to extract data from open web sources, forums, marketplaces, and the dark web. Implement data extraction pipelines that aggregate, clean, and structure data for fraud detection and risk profiling. Use Tor, VPNs, and other anonymization techniques to safely crawl the dark web while avoiding detection. Develop real-time monitoring solutions for tracking fraudulent activities, data breaches, and cybercrime discussions. Optimize crawling speed and ensure compliance with website terms of service, ethical standards, and legal frameworks. Integrate extracted data with fraud detection models, risk scoring algorithms, and cybersecurity intelligence tools. Work with data scientists and security analysts to develop threat intelligence dashboards from collected data. Implement anti-bot detection evasion techniques and handle CAPTCHAs using AI-driven solvers where necessary. Stay updated on OSINT (Open-Source Intelligence) techniques, web scraping best practices, and cybersecurity trends. Requirements 5+ years of experience in web crawling, data scraping, or cybersecurity data extraction. Strong proficiency in Python, Scrapy, Selenium, BeautifulSoup, Puppeteer, or similar frameworks. Experience working with Tor, proxies, and VPNs for anonymous web scraping. Deep understanding of HTTP protocols, web security, and bot detection mechanisms. Experience parsing structured and unstructured data from JSON, XML, and web pages. Strong knowledge of database management (SQL, NoSQL) for storing large-scale crawled data. Familiarity with AI/ML-based fraud detection techniques and data classification methods. Experience working with cybersecurity intelligence sources, dark web monitoring, and OSINT tools. Ability to implement scalable, distributed web crawling architectures. Knowledge of data privacy regulations (GDPR, CCPA) and ethical data collection practices. Nice to Have Experience in fintech, fraud detection, or threat intelligence. Knowledge of natural language processing (NLP) for analyzing cybercrime discussions. Familiarity with machine learning-driven anomaly detection for fraud prevention. Hands-on experience with cloud-based big data solutions (AWS, GCP, Azure, Elasticsearch, Kafka).

Position: Data Management Intern Location: Noida - Sector 126 Working days- 6 days, (10:30 am to 7:30 pm) Skills: Data entry, Basic excel, Data Correction, Web scraping. Company's Website- https://leverage.biz/ Experience: pursuing or freshers Role & responsibilities Web Scraping: You'll be responsible for extracting course information from university websites using the Octoparse Tool . Manual Data Correction/Management: You'll ensure the accuracy and quality of the extracted data through manual review and correction. Data Gathering: Collect structured and unstructured data through web scraping and other methods. Data Cleaning: Ensure data accuracy by identifying inconsistencies, duplication and errors in collected data-sets. Preferred candidate profile Education: Any field of study. Technical Skills: Basic understanding in Excel and data entry tools, Basic knowledge of web scraping tools (e.g., Octoparse) is a plus. Analytical Skills: Strong attention to detail and data accuracy. Communication Skills: Able to communicate professionally in office with cross functional teams Time Management: Ability to handle multiple tasks and meet deadlines in a fast-paced environment. Problem-Solving Skills: A proactive approach to addressing data collection challenges.

Skill Expectations Must-Have Skills: Strong hands-on experience in Python development Experience working with Fast API Data migration and data engineering experience (ETL, pipelines, transformations) Experience in web scraping and data extraction techniques Experience working with GCP Good-to-Have / Optional Skills: Exposure to FastAPI Experience or familiarity with GenAI / LLMs Knowledge of testing frameworks (e.g., PyTest) and CI/CD pipelines

Good knowledge and in Python, sql, Perl with 6+ years experience. Good problem solving skill. Ability to understand the data and its relations Capability to learn new technologies in short span of time. Should be able to work in Sprint and meet deadlines. Flexible Work time. Mandatory Skills: Python - Basics, Pandas, Web scrapping, File and XML Handling, Extracting/Manipulating -Excel/CSV/Any File Formats.Perl - Basics, CPAN modules, File and Web scrapping/Handling.** Work from option is available

We are looking for a highly skilled and self-motivated Python Developer with strong expertise in Flask, API integrations (like Amazon and stock market platforms), Selenium automation, and AI/LLM-based applications . The ideal candidate will support and enhance our data-driven platforms, build intelligent automation systems, and contribute to innovation in Business Intelligence and digital workflows. Key Requirements: Proficiency in Python with 2+ years of hands-on experience. Strong experience with Flask or similar Python web frameworks. Solid understanding of RESTful API development and consumption. Hands-on expertise in Selenium for browser automation tasks. Practical experience or projects using Large Language Models (LLMs), GPT, or AI toolkits . Familiarity with version control (Git), Docker, and deployment environments. Strong analytical and problem-solving skills. Excellent communication and collaboration abilities. Knowledge of Accounts will be added advantage. Send your resume to mandakranta.mahapatra@jaytea.com with the subject: "Application for Python Developer Position", including: - Current CTC - Expected CTC - Notice Period

Web scraping ( Imm joiner) Job Responsibilities: Develop and maintain web scraping scripts using Python and Selenium to extract data from websites and APIs Understanding requirement of customer needs, identify source and scrape/collect data and integrate the data crawled and scraped into our databases. Writing efficient, reusable, testable, and testing scalable code. Build code that is easily readable, properly documented, and follows key coding standards. Use Beautiful Soup and other scraping tools to clean and process data for analysis Research and learn new technologies to design code reviews and application enhancements/ upgrades Performance tuning and automation of application. Maintain communication with corporate clients, Coordination with team and management. Coordinating with development teams to determine application requirements. Prioritized daily workflows, including all projects and task Experience: Proficiency with Selenium and Beautiful Soup Depth knowledge of Data Structures, Operators, Datatypes, Casting, Methods, Conditions, Loops, Functions, Arrays, OOPS concept, iterators, Modules and MySQL integration with Python Strong understanding of web scraping techniques and best practices HTML, CSS, XPATH, Regex and HTML DOM. Knowledge of Libraries like Selenium, Requests, Scrappy, BeautifulSoup, NumPy, Pandas, Matplotlib, SciPy, TensorFlow, Keras, PyTorch, Plotly and Tableau. Good hands-on experience in Python languages and debugging with GDB Database & File Format: SQL, MySQL, Mongo dB, JSON, XML, Excel, CSV Development Tools: Microsoft Visual Studio, Visual Code, PyCharm, Power bi Knowledge of Rest API will be added as plus point

Company Overview BOLD is an established and fast-growing product company that transforms work lives. Since 2005,weve helped more than 10,000,000 folks from all over America(and beyond!) reach higher and do better. A career at BOLD promises great challenges, opportunities, culture and the environment. With our headquarters in Puerto Rico and offices in San Francisco and India, were a global organization on a path to change the career industry. Position Overview BOLD is seeking a highly skilled professional to spearhead the development of cutting-edge browser automation technology for our Expert Apply product. You will play a key role in designing scalable automation frameworks, tackling challenges in bot detection, and optimizing system performance. You'll also be responsible for building and monitoring metrics to ensure system reliability and robustness. If you are passionate about large-scale automation and system reliability, we want to hear from you. Role & responsibilities Design and architect scalable and robust enterprise-level automation applications using Python. Develop applications that run on PODs (Kubernetes), ensuring high availability and reliability. Debug complex issues in applications and devise solutions that enhance stability and performance. Identify performance bottlenecks within applications through profiling and metrics analysis. Optimize existing code to improve performance and efficiency, ensuring the system can handle high traffic loads. Utilize automation frameworks and tools such as Playwright, Chromium, and stealth browser for web automation tasks. Implement message handling to facilitate communication between different services. Develop web scraping solutions to gather and process data from various online sources. Analyze and troubleshoot software issues, providing timely resolutions to ensure system reliability Collaborate with cross-functional teams to understand user requirements and translate them into technical specifications. Review and enhance code quality through thorough testing and code reviews. Stay updated with industry trends and emerging technologies, integrating best practices into the development process Document architecture, design choices, and implementation details for future reference and knowledge sharing. Ensure compliance with security and data privacy standards throughout the application lifecycle. Preferred candidate profile Strong programming skills in Python like expertise in string manipulation and regular expression to effectively handle and process the text data during web scrapping and automation tasks. Deep understanding of OOP principles, including encapsulation, inheritance, and polymorphism, to design robust and maintainable software systems. Knowledge of common design patterns (e.g., Singleton, Factory, Observer) to enhance system design, improve code reusability, and implement best practices in software architecture. Solid foundation in algorithms (sorting, searching, parsing) and data structures (lists, dictionaries, trees) to solve complex problems efficiently and effectively during software development. Good understanding of how modern browsers function, including rendering engines, Java Script engines, HTTP protocols, and browser APIs. Experience optimizing scraping strategies based on browser behavior and performance. Experience with caching technologies (e.g. Redis, in-memory caching) Experience with messaging protocols (e.g. Azure service bus, Kafka, RabbitMQ) working knowledge and proven experience in containerization using Docker. Understanding of DevOps practices and CI/CD pipelines. Excellent communication skills and the ability to collaborate across time zones Excellent analytical and problem-solving skills. Knowledge of cloud computing, Amazon Web Services or Microsoft Azure

We are looking for a Python Developer who has expertise in web scraping and backend development. The ideal candidate should be proficient in Python frameworks, data extraction techniques, and API integration.

Job Information Job Opening ID ZR_2129_JOB Date Opened 05/03/2024 Industry Technology Job Type Work Experience 3-5 years Job Title DB developer -Python City Chennai Province Tamil Nadu Country India Postal Code 600001 Number of Positions 5 Good knowledge and in Python, sql, Perl with 6+ years experience. Good problem solving skill. Ability to understand the data and its relations Capability to learn new technologies in short span of time. Should be able to work in Sprint and meet deadlines. Flexible Work time. Mandatory Skills: Python - Basics, Pandas, Web scrapping, File and XML Handling, Extracting/Manipulating -Excel/CSV/Any File Formats.Perl - Basics, CPAN modules, File and Web scrapping/Handling.** Work from option is available check(event) ; career-website-detail-template-2 => apply(record.id,meta)" mousedown="lyte-button => check(event)" final-style="background-color:#2B39C2;border-color:#2B39C2;color:white;" final-class="lyte-button lyteBackgroundColorBtn lyteSuccess" lyte-rendered=""> I'm interested

Role & responsibilities Develop and implement Python scripts for web scraping using Selenium WebDriver to extract relevant data from client websites. Clean, transform, and manipulate extracted data using Python libraries (e.g., Pandas, BeautifulSoup) for schema (Structured data) markup implementation Write well-documented, maintainable, and efficient Python code adhering to best practices. Collaborate with SEOs and the Director of SEO to understand client requirements and translate them into technical solutions. Stay up-to-date on the latest trends and developments in web scraping, schema (Structured data) markup, and SEO best practices. Assist with testing and debugging developed scripts to ensure accuracy of schema (Structured data) implementation without any error. Experience working in Automation through AI agents Experience working with machine learning and AI (Artificial Intelligence) integration using Python. Preferred candidate profile Having 4-5 years of working experience in Python programming. Strong understanding of Python syntax, data structures, Iterator, Generators, Exception Handling, File handling, OOPs, Data Structures, ORM and object-oriented programming concepts. Proficiency in using web scraping libraries like Selenium WebDriver and Beautiful Soup. Must be familiar with Web Frameworks like HTML, CSS, JavaScript, Django or Flasks. Good knowledge of machine learning & ML frameworks like NumPy, Pandas, Kera's, scikit-learn, PyTorch, TensorFlow or Microsoft Azure Machine Learning will be added advantage. Must be familiar with development tools like Jupyter Notebook, IDLE, PyCharm or VS Code. Must be familiar with Scrum methodology, CI/CD, Git, Branching/Merging and test-driven software development. Candidates worked in product-based companies will be preferred. Excellent analytical and problem-solving skills. Ability to work independently and as part of a team. Strong communication and collaboration skills. A passion for SEO and a desire to learn about schema (Structured data) markup. Familiarity with cloud platforms (AWS, GCP, Azure DevOps, Azure Blob Storage Explorer) Experience with API integration. Experience working with AI (Artificial Intelligence) integration with Python to automate SEO tasks with Google Gemini, GenAI (Generative AI) & ChatGPT 4. Experience working in Automation through AI agents Good verbal and written communication skills.

Contract Duration – 3 Months Minimum 5yrs in Python Must Have: Python Web Scrapping (Selenium/Beautifulsoup) Pycharm Html, CSS, Oops AI Automation Good to have: Schema/ Structure Data Cloud Storage - Azure(blob storage)/AWS

About Us: LdotR is an online brand protection service company, offering businesses the right solution and services to protect, manage and benefit from their digital assets in the online space. We work across all digital platforms - Domains, Website, Social Media, Online Marketplaces, and App Stores to identify, assess and nullify brand infringements. About the Role: We are looking for an experienced Data Scraping Specialist to help us extract and structure data from leading social media platforms at scale. The ideal candidate will have hands-on expertise with scraping tools, APIs, and large-scale data processing. Key Responsibilities: Design and develop custom scraping solutions to extract public data from platforms like Instagram, Facebook, X (Twitter), LinkedIn, YouTube, etc. Handle large-scale scraping tasks with efficiency and resilience against rate-limiting and platform-specific restrictions. Clean, normalize, and structure the scraped data for analysis or downstream applications. Maintain scraping scripts to adapt to frequent platform changes. Ensure compliance with data protection policies and terms of service. Required Skills: Proficiency in Python and scraping libraries (e.g., Scrapy, BeautifulSoup, Selenium, Playwright). Experience with API integration (official or unofficial social media APIs). Familiarity with rotating proxies, headless browsers, and CAPTCHA-solving techniques. Strong understanding of data structuring formats like JSON, CSV, and databases (MongoDB, PostgreSQL, etc.). Experience with cloud-based scraping and storage solutions (AWS/GCP preferred). Good to Have: Knowledge of NLP or data analytics for social media sentiment or trend analysis. Understanding of GDPR and CCPA compliance. Prior work with third-party scraping platforms or browser automation tools. What We Offer: Opportunity to work on impactful, large-scale data projects. Flexible work arrangements. Competitive compensation based on experience and delivery.

Responsibilities:As a Data Engineer focused on web crawling and platform data acquisition, you will design, develop, and maintain large-scale web scraping pipelines to extract valuable platform data. Annual bonus Health insurance Provident fund

Key Responsibilities: Develop and maintain automated web scraping scripts using Python libraries such as Beautiful Soup, Scrapy, and Selenium. Optimize scraping pipelines for performance, scalability, and resource efficiency. Handle dynamic websites, CAPTCHA-solving, and implement IP rotation techniques for uninterrupted scraping. Process and clean raw data, ensuring accuracy and integrity in extracted datasets. Collaborate with cross-functional teams to understand data requirements and deliver actionable insights. Leverage APIs when web scraping is not feasible, managing authentication and request optimization. Document processes, pipelines, and troubleshooting steps for maintainable and reusable scraping solutions. Ensure compliance with legal and ethical web scraping practices, implementing security safeguards. Requirements: Education : Bachelors degree in Computer Science, Engineering, or a related field. Experience : 2+ years of Python development experience, with at least 1 year focused on web scraping. Technical Skills : Proficiency in Python and libraries like Beautiful Soup, Scrapy, and Selenium. Experience with regular expressions (Regex) for data parsing. Strong knowledge of HTTP protocols, cookies, headers, and user-agent rotation. Familiarity with databases (SQL and NoSQL) for storing scraped data. Hands-on experience with data manipulation libraries such as pandas and NumPy. Experience working with APIs and managing third-party integrations. Familiarity with version control systems like Git. Bonus Skills : Knowledge of containerization tools like Docker. Experience with distributed scraping solutions and task queues (e.g., Celery, RabbitMQ). Basic understanding of data visualization tools. Non-Technical Skills : Strong analytical and problem-solving skills. Excellent communication and documentation skills. Ability to work independently and collaboratively in a team environmen

About Us : We are a dynamic and innovative team specializing in building scalable and efficient web scraping solutions for complex e-commerce platforms and diverse web environments. Our team is proficient in a wide range of technologies and advanced web scraping techniques, ensuring high-quality data extraction and delivery Job Summary: We are seeking a talented Web Scraping Engineer / Data Extraction Specialist to join our growing team. The ideal candidate will have a strong background in web scraping, data extraction, and backend technologies. You will be responsible for designing, developing, and maintaining robust web scraping solutions, handling dynamic content, and overcoming anti-crawling measures. Responsibilities: Develop and maintain scalable web scraping scripts using Python, JavaScript, and related frameworks (e.g., Scrapy, Selenium, Puppeteer, Beautiful Soup, Cheerio.js). Implement advanced web scraping techniques, including API interception, sitemap parsing, and handling dynamic content. Design and build data pipelines for efficient data extraction, processing, and storage. Manage and optimize data extraction workflows, ensuring high speed and accuracy. Implement anti-crawling solutions, including IP rotation, proxy management, and CAPTCHA bypassing. Collaborate with cross-functional teams to gather requirements and deliver complex data solutions. Utilize backend technologies such as Flask, FastAPI, Django, Node.js, Spring Boot, and relational databases (PostgreSQL, MySQL) for data storage and API development. Work with cloud platforms like Azure and leverage services such as AzureML and ADLS GEN2. Employ data processing techniques using libraries like NumPy and Pandas. Use tools like Postman, MITM, and DevTools for API testing and network traffic analysis. Apply machine learning and NLP techniques for data analysis and processing (e.g., sentiment analysis, content classification). Set up and manage server-side scraping environments. Monitor and troubleshoot scraping scripts to ensure optimal performance. Required Skills: Strong proficiency in Python and/or JavaScript. Experience with web scraping libraries and frameworks (Scrapy, Selenium, Puppeteer, Beautiful Soup, Cheerio.js). Knowledge of backend technologies (Flask, FastAPI, Django, Node.js, Spring Boot). Experience with relational databases (PostgreSQL, MySQL). Understanding of HTTP/HTTPS protocols and API communication. Familiarity with cloud platforms (Azure). Ability to handle dynamic content and JavaScript-heavy websites. Experience with anti-crawling techniques (IP rotation, proxies, CAPTCHA bypassing). Data processing and analysis skills (NumPy, Pandas). Experience with API testing tools (Postman). Knowledge of machine learning and NLP concepts is a plus. Strong problem-solving and debugging skills. Excellent communication and collaboration skills. Experience: 5+ years of professional experience in web scraping or related fields. Education: Bachelor's degree in Computer Science, Engineering, or a related field

Develop and maintain scalable Python-based applications, microservices, and automated tools using Django. Design, implement, and maintain RESTful APIs and backend logic for various modules (college search, applications, data processing, etc.).

Overview The Data Technology team at MSCI is responsible for meeting the data requirements across various business areas, including Index, Analytics, and Sustainability. Our team collates data from multiple sources such as vendors (e.g., Bloomberg, Reuters), website acquisitions, and web scraping (e.g., financial news sites, company websites, exchange websites, filings). This data can be in structured or semi-structured formats. We normalize the data, perform quality checks, assign internal identifiers, and release it to downstream applications. Responsibilities As data engineers, we build scalable systems to process data in various formats and volumes, ranging from megabytes to terabytes. Our systems perform quality checks, match data across various sources, and release it in multiple formats. We leverage the latest technologies, sources, and tools to process the data. Some of the exciting technologies we work with include Snowflake, Databricks, and Apache Spark. Qualifications Core Java, Spring Boot, Apache Spark, Spring Batch, Python. Exposure to sql databases like Oracle, Mysql, Microsoft Sql is a must. Any experience/knowledge/certification on Cloud technology preferrably Microsoft Azure or Google cloud platform is good to have. Exposures to non sql databases like Neo4j or Document database is again good to have. What we offer you Transparent compensation schemes and comprehensive employee benefits, tailored to your location, ensuring your financial security, health, and overall wellbeing. Flexible working arrangements, advanced technology, and collaborative workspaces. A culture of high performance and innovation where we experiment with new ideas and take responsibility for achieving results. A global network of talented colleagues, who inspire, support, and share their expertise to innovate and deliver for our clients. Global Orientation program to kickstart your journey, followed by access to our Learning@MSCI platform, LinkedIn Learning Pro and tailored learning opportunities for ongoing skills development. Multi-directional career paths that offer professional growth and development through new challenges, internal mobility and expanded roles. We actively nurture an environment that builds a sense of inclusion belonging and connection, including eight Employee Resource Groups. All Abilities, Asian Support Network, Black Leadership Network, Climate Action Network, Hola! MSCI, Pride & Allies, Women in Tech, and Women’s Leadership Forum. At MSCI we are passionate about what we do, and we are inspired by our purpose – to power better investment decisions. You’ll be part of an industry-leading network of creative, curious, and entrepreneurial pioneers. This is a space where you can challenge yourself, set new standards and perform beyond expectations for yourself, our clients, and our industry. MSCI is a leading provider of critical decision support tools and services for the global investment community. With over 50 years of expertise in research, data, and technology, we power better investment decisions by enabling clients to understand and analyze key drivers of risk and return and confidently build more effective portfolios. We create industry-leading research-enhanced solutions that clients use to gain insight into and improve transparency across the investment process. MSCI Inc. is an equal opportunity employer. It is the policy of the firm to ensure equal employment opportunity without discrimination or harassment on the basis of race, color, religion, creed, age, sex, gender, gender identity, sexual orientation, national origin, citizenship, disability, marital and civil partnership/union status, pregnancy (including unlawful discrimination on the basis of a legally protected parental leave), veteran status, or any other characteristic protected by law. MSCI is also committed to working with and providing reasonable accommodations to individuals with disabilities. If you are an individual with a disability and would like to request a reasonable accommodation for any part of the application process, please email Disability.Assistance@msci.com and indicate the specifics of the assistance needed. Please note, this e-mail is intended only for individuals who are requesting a reasonable workplace accommodation; it is not intended for other inquiries. To all recruitment agencies MSCI does not accept unsolicited CVs/Resumes. Please do not forward CVs/Resumes to any MSCI employee, location, or website. MSCI is not responsible for any fees related to unsolicited CVs/Resumes. Note on recruitment scams We are aware of recruitment scams where fraudsters impersonating MSCI personnel may try and elicit personal information from job seekers. Read our full note on careers.msci.com