156 Scrapy Jobs - Page 7

Job Title: AI/ML & Automation Intern Location: Remote Duration: 6 Months About us Astraeus Next Gen Pvt. Ltd. is building the next generation of intelligent automation and analytics tools powered by AI/ML. Our mission is to transform how businesses access, analyze, and act on data with zero-click intelligence. Role Overview We are seeking highly motivated interns from Tier 1 institutes (IITs, NITs, IIITs, BITS, etc.) with strong foundations in AI/ML, automation, full-stack development, deployment, and ETL pipelines. As an intern, you will work on live projects that involve developing intelligent automation tools, deploying scalable solutions, and building production-grade web apps and APIs. Key Responsibilities • Design, train, and evaluate ML models for real-world applications • Build scalable ETL pipelines to ingest and process large datasets • Automate web scraping, data collection, and report generation tasks • Develop and deploy backend services (Flask/FastAPI/Django) and frontend components (React.js/Next.js) • Deploy models and APIs on cloud platforms (AWS/GCP/Azure) using Docker, CI/CD pipelines • Collaborate with senior developers and product teams on full lifecycle development Requirements • Currently enrolled in a Tier 1 engineering institute (IIT, NIT, IIIT, BITS, etc.) • Strong programming skills in Python, JavaScript (Node/React), SQL • Hands-on experience with machine learning frameworks (Scikit-learn, TensorFlow, PyTorch) • Knowledge of full-stack development and modern web frameworks • Familiarity with cloud deployment, Docker, and REST APIs • Experience with automation/scraping tools (Selenium, Scrapy, BeautifulSoup) is a plus • Knowledge of MongoDB, Postgres, or other databases • Ability to work independently and in fast-paced, agile teams Good to Have • Exposure to tools like Apache Airflow, n8n, Kafka • Experience in report generation using Pandas, ReportLab, or similar tools • Contributions to open-source or personal projects in relevant domains What You’ll Get • Real-world experience building products. • Opportunity to convert to a full-time role • Work on high-impact projects in AI, automation, and intelligence systems To Apply: Submit your resume, portfolio (if any), and GitHub/LinkedIn profile to [shreejitsen@astraeusnextgen.com] Subject Line: AI/ML & Automation Intern Application - [Your Name] Note: This is an unpaid internship Show more Show less

Job Title: Sr. Python Developer – Web Scraper Location: Ahmedabad, Gujarat Job Type: Full Time Experience: 3+ Years Department: Python About Simform: Simform is a premier digital engineering company specializing in Cloud, Data, AI/ML, and Experience Engineering to create seamless digital experiences and scalable products. Simform is a strong partner for Microsoft, AWS, Google Cloud, and Databricks. With a presence in 5+ countries, Simform primarily serves North America, the UK, and the Northern European market. Simform takes pride in being one of the most reputed employers in the region, having created a thriving work culture with a high work-life balance that gives a sense of freedom and opportunity to grow. Role Overview: The Sr. Python Developer – Web Scraper will be responsible for building scalable, resilient, and intelligent web scraping solutions to extract structured and unstructured data from websites, APIs, and digital platforms. This role involves handling anti-bot mechanisms, dynamic content rendering, and real-time data extraction pipelines, while ensuring high data quality and performance. The ideal candidate will have strong Python skills, deep understanding of scraping techniques, and experience working with headless browsers and cloud platforms. Key Responsibilities: Design and implement scalable web scraping frameworks to collect data from complex and dynamic websites. Develop custom spiders/crawlers using Python libraries like Playwright, Puppeteer, Selenium, Scrapy, or BeautifulSoup. Apply advanced anti-bot evasion strategies such as CAPTCHA solving, IP rotation, user-agent spoofing, browser fingerprinting, and session/cookie management. Automate scraping tasks across distributed systems using tools like Celery, Airflow, cron, and ETL orchestration platforms. Optimize scraper performance for speed, accuracy, and resilience to website structure changes. Implement network interception, DOM traversal, WebSocket handling, and headless browser control. Store and manage scraped data in cloud or local storage using PostgreSQL, MongoDB, or S3. Integrate scraping systems with APIs or microservices for data consumption and downstream workflows. Monitor scraper reliability and handle retry logic, error logging, and dynamic throttling. Write modular, well-documented, and testable Python code with proper unit testing and version control. Collaborate with engineers, data scientists, and stakeholders to understand scraping goals and align them with business needs. Required Skills & Qualifications: Bachelor’s/Master’s degree in Computer Science, Engineering, or a related field. 3+ years of experience in Python development with specialization in web scraping. Deep understanding of modern anti-scraping defenses and bypass techniques (e.g., CAPTCHA, IP bans, dynamic rendering). Proficiency with headless browser tools like Playwright, Puppeteer, or Selenium. Strong grasp of DOM manipulation, JavaScript execution, network inspection, and asynchronous scraping using asyncio, aiohttp, etc. Experience in handling large-scale data extraction and storage using SQL and NoSQL databases. Hands-on experience deploying scrapers and automation workflows on AWS, GCP, or Azure. Familiarity with containerization using Docker and optional experience with Kubernetes. Comfortable with REST API integration, job scheduling, and microservices-based environments. Strong debugging, optimization, and testing skills. Clear understanding of legal and ethical scraping boundaries. Why Join Us: Young Team, Thriving Culture Flat-hierarchical, friendly, engineering-oriented, and growth-focused culture Well-balanced learning and growth opportunities Free health insurance Office facilities with a game zone, in-office kitchen with affordable lunch service, and free snacks Sponsorship for certifications/events and library service Flexible work timing, leaves for life events, WFH, and hybrid options

Key Responsibilities: Build and maintain backend services using Python frameworks like Django , Flask , or FastAPI . Design and consume RESTful APIs to integrate with frontend and external systems. Perform web scraping using: Selenium BeautifulSoup Scrapy or equivalent libraries Contribute to AI/ML-based applications , including model integration, API wrapping, and data handling. Work on data preprocessing, analysis , and pipeline creation for ML applications. Collaborate with cross-functional teams including frontend developers, designers, and PMs. Participate in agile practices: daily standups, sprint planning, and code reviews. Learn and assist in projects involving no-code/low-code platforms . Requirements: Bachelor’s degree in Computer Science, Engineering, or related field. 1-2 years of Python development experience. Strong experience in web scraping using tools like: Selenium , BeautifulSoup , and Scrapy Proficient in backend frameworks: Django , Flask , or FastAPI . Good understanding of REST APIs and ORMs . Experience working with relational databases (PostgreSQL, MySQL). Familiarity with Git and collaborative development workflows. Exposure to AI/ML concepts , model integration, or use of tools like scikit-learn , Pandas , NumPy . Good problem-solving and debugging skills. Nice to Have: Experience deploying ML models or building inference APIs. Familiarity with TensorFlow , PyTorch , or other ML frameworks. Knowledge of Docker , CI/CD , and cloud platforms (AWS, GCP, Azure). Experience with No-Code/Low-Code tools (training will be provided if not experienced). Understanding of frontend basics (React.js or similar) is a plus. Working LOcation : Mohali (Punjab) Experience required - 1-2 Years Job Type: Full-time Pay: Up to ₹30,000.00 per month Benefits: Health insurance Work Location: In person

Data Science and AI Developer **Job Description:** We are seeking a highly skilled and motivated Data Science and AI Developer to join our dynamic team. As a Data Science and AI Developer, you will be responsible for leveraging cutting-edge technologies to develop innovative solutions that drive business insights and enhance decision-making processes. No salary will be provided during the 3-month probation period. **Key Responsibilities:** 1. Develop and deploy machine learning models for predictive analytics, classification, clustering, and anomaly detection. 2. Design and implement algorithms for data mining, pattern recognition, and natural language processing. 3. Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions. 4. Utilize advanced statistical techniques to analyze complex datasets and extract actionable insights. 5. Implement scalable data pipelines for data ingestion, preprocessing, feature engineering, and model training. 6. Stay updated with the latest advancements in data science, machine learning, and artificial intelligence research. 7. Optimize model performance and scalability through experimentation and iteration. 8. Communicate findings and results to stakeholders through reports, presentations, and visualizations. 9. Ensure compliance with data privacy regulations and best practices in data handling and security. 10. Mentor junior team members and provide technical guidance and support. **Requirements:** 1. Bachelor’s or Master’s degree in Computer Science, Data Science, Statistics, or a related field. 2. Proven experience in developing and deploying machine learning models in production environments. 3. Proficiency in programming languages such as Python, R, or Scala, with strong software engineering skills. 4. Hands-on experience with machine learning libraries/frameworks such as TensorFlow, PyTorch, Scikit-learn, or Spark MLlib. 5. Solid understanding of data structures, algorithms, and computer science fundamentals. 6. Excellent problem-solving skills and the ability to think creatively to overcome challenges. 7. Strong communication and interpersonal skills, with the ability to work effectively in a collaborative team environment. 8. Certification in Data Science, Machine Learning, or Artificial Intelligence (e.g., Coursera, edX, Udacity, etc.). 9. Experience with cloud platforms such as AWS, Azure, or Google Cloud is a plus. 10. Familiarity with big data technologies (e.g., Hadoop, Spark, Kafka) is an advantage. Data Manipulation and Analysis : NumPy, Pandas Data Visualization : Matplotlib, Seaborn, Power BI Machine Learning Libraries : Scikit-learn, TensorFlow, Keras Statistical Analysis : SciPy Web Scrapping : Scrapy IDE : PyCharm, Google Colab HTML/CSS/JavaScript/React JS Proficiency in these core web development technologies is a must. Python Django Expertise: In-depth knowledge of e-commerce functionalities or deep Python Django knowledge. Theming: Proven experience in designing and implementing custom themes for Python websites. Responsive Design: Strong understanding of responsive design principles and the ability to create visually appealing and user-friendly interfaces for various devices. Problem Solving: Excellent problem-solving skills with the ability to troubleshoot and resolve issues independently. Collaboration: Ability to work closely with cross-functional teams, including marketing and design, to bring creative visions to life. interns must know about how to connect front end with datascience Also must Know to connect datascience to frontend **Benefits:** - Competitive salary package - Flexible working hours - Opportunities for career growth and professional development - Dynamic and innovative work environment Job Type: Full-time Pay: ₹8,000.00 - ₹12,000.00 per month Schedule: Day shift Ability to commute/relocate: Thiruvananthapuram, Kerala: Reliably commute or planning to relocate before starting work (Preferred) Work Location: In person

Proficient in Python, web scraping (BeautifulSoup, Scrapy, Selenium), HTML/CSS/JS, HTTP, DevTools, IP rotation, proxies, automation, bypassing protections (e.g. Cloudflare), GCP, GraphQL, NoSQL, and Selenium Wire. Skilled in scalable data handling. Provident fund Health insurance

COMPANY PROFILE:Company Name: Gmware pvt. ltd. Gmware is an IT company that specializes in serving overseas clients in different verticals. We are looking for a candidate who can join our engineering team and help us develop and maintain various software products. Requirements :Proven 6month+ of relevant experience as a Web Crawling & Scraping EngineerStay updated on the latest trends and technologies in web scraping, crawling, and data processing.Strong problem-solving skillsExcellent communication and collaboration skills.Handle dynamic and complex websites, including those with anti-scraping mechanisms.Troubleshoot and resolve issues related to data extraction, including data integrity and performance challenges.Proficient in popular web scraping technologies and frameworks, such as Selenium, Beautiful soup, Scrapy Contact No. : 80555-03000