Junior Web Scraping Engineer

Role Overview: As a Web Scraper, your main responsibility is to apply your expertise to extract data from various online sources. You will be tasked with developing reliable web scrapers and parsers for different websites. Your role involves extracting structured/unstructured data and storing it in SQL/NoSQL data stores. Additionally, you will collaborate closely with Project/Business/Research teams to provide the scraped data for analysis. It will be your duty to maintain the scraping projects that are delivered to production and to create frameworks for automating and ensuring a continuous flow of data from multiple sources. You should be able to work independently with minimal supervision and develop a deep understanding of web data sources to determine how, when, and which data to scrape, parse, and store. Key Responsibilities: - Develop highly reliable web scrapers and parsers across various websites - Extract structured/unstructured data and store it in SQL/NoSQL data stores - Collaborate with Project/Business/Research teams to provide scraped data for analysis - Maintain scraping projects delivered to production - Develop frameworks for automating and maintaining a constant flow of data from multiple sources - Work independently with minimum supervision - Develop a deep understanding of web data sources and determine the data to scrape, parse, and store Qualifications Required: - 1 to 3 years of experience as a Web Scraper - Proficient knowledge in Python language and working knowledge of Web Crawling/Web scraping in Python Requests, Beautifulsoup or URLlib, and Selenium, Playwright - Strong knowledge of basic Linux commands for system navigation, management, and troubleshooting - Expertise in proxy usage for secure and efficient network operations - Experience with captcha-solving techniques for seamless automation and data extraction - Experience with data parsing - Strong knowledge of Regular expression, HTML, CSS, DOM, XPATH - Knowledge of Javascript would be a plus - Ability to access, manipulate, and transform data from various database and flat file sources - Essential skills in MongoDB & MYSQL - Ability to develop reusable code-based scraping products for others" use - Mandatory GIT knowledge for version control and collaborative development workflows - Experience handling cloud servers on platforms like AWS, GCP, and LEAPSWITCH for scalable and reliable infrastructure management - Ability to ask the right questions and deliver understandable and usable results to clients - Track record of problem-solving, innovative approaches, and tackling tough problems from different angles Company Details: Additional details about the company were not provided in the job description.,