Data Engineer - Web Scrapping & Enrichment

As an entrepreneurial, passionate, and driven Data Engineer at Startup Gala Intelligence backed by Navneet Tech Venture, you will play a crucial role in shaping the technology vision, architecture, and engineering culture of the company right from the beginning. Your contributions will be foundational in developing best practices and establishing the engineering team. **Key Responsibilities:** - **Web Scraping & Crawling:** Build and maintain automated scrapers to extract structured and unstructured data from websites, APIs, and public datasets. - **Scalable Scraping Systems:** Develop multi-threaded, distributed crawlers capable of handling high-volume data collection without interruptions. - **Data Parsing & Cleaning:** Normalize scraped data, remove noise, and ensure consistency before passing to data pipelines. - **Anti-bot & Evasion Tactics:** Implement proxy rotation, captcha solving, and request throttling techniques to handle scraping restrictions. - **Integration with Pipelines:** Deliver clean, structured datasets into NoSQL stores and ETL pipelines for further enrichment and graph-based storage. - **Data Quality & Validation:** Ensure data accuracy, deduplicate records, and maintain a trust scoring system for data confidence. - **Documentation & Maintenance:** Keep scrapers updated when websites change, and document scraping logic for reproducibility. **Qualifications Required:** - 2+ years of experience in web scraping, crawling, or data collection. - Strong proficiency in Python (libraries like BeautifulSoup, Scrapy, Selenium, Playwright, Requests). - Familiarity with NoSQL databases (MongoDB, DynamoDB) and data serialization formats (JSON, CSV, Parquet). - Experience in handling large-scale scraping with proxy management and rate-limiting. - Basic knowledge of ETL processes and integration with data pipelines. - Exposure to graph databases (Neo4j) is a plus. As part of Gala Intelligence, you will be working in a tech-driven startup dedicated to solving fraud detection and prevention challenges. The company values transparency, collaboration, and individual ownership, creating an environment where talented individuals can thrive and contribute to impactful solutions. If you are someone who enjoys early-stage challenges, thrives on owning the entire tech stack, and is passionate about building innovative, scalable solutions, we encourage you to apply. Join us in leveraging technology to combat fraud and make a meaningful impact from day one.,