Jobs

Interviews
Job Alerts
Tools

Upskill and Grow with AI

Mock Interview Practice interviews in realistic simulations

Coding Practice Improve your coding skills with challenges

Certification Earn certifications to validate your skills

AI Learning Get trained with AI expert sessions

Career Path AI insights for smarter career decisions

AI Job Match Score AI-Powered Job Match Against Your Resume and Optimize Your Resume

Career Tools and Resources

Resume Builder Build Professional Resume with Ease

ATS Friendliness Check Check Resume Friendliness for Applicant Tracking Systems

Auto Apply Apply to hundreds of jobs on any platform effortlessly

Co-Pilot (Chrome Extension) Your AI Assistant for Seamless Browsing Efficiency

Interview Questions Streamline interviews with ready-to-use questions

Salaries Discover market-driven salary insights across skillsets and geographies

Companies Explore leading companies actively hiring talent
For Employers

Jobs

Interviews

Home
>
Jobs in hyderabad
>
OSI Digital
>
Data Engineer

Data Engineer

OSI Digital

3 years

0 Lacs

hyderabad telangana india

Posted:18 hours ago| Platform:

Apply

Skills Required

data python management design scrapy ip extraction reliability engineering programming selenium fingerprinting http sql nosql debugging aws gcp etl technology

Work Mode

On-site

Job Type

Full Time

Job Description

We are looking for a skilled

Web Crawling Engineer

with strong expertise in Python and hands-on experience building scalable crawlers and scrapers. The ideal candidate should have deep knowledge of crawling frameworks, anti-blocking techniques, and proxy management, along with experience solving challenges like captchas and rate-limiting.

Key Responsibilities

Design, develop, and maintain scalable and efficient web crawlers for data extraction.
Work with Scrapy, Requests, and other Python libraries to implement crawling solutions.
Handle anti-scraping measures, including IP blocks, rate limits, and captchas.
Implement and manage proxy rotation and session management strategies.
Ensure high-quality and structured data extraction, cleaning, and storage.
Monitor crawler performance, troubleshoot issues, and optimize for reliability and efficiency.
Collaborate with the data engineering team to integrate crawled data into pipelines.
Stay updated with the latest tools, techniques, and best practices in web crawling.

Required Skills & Qualifications

Strong programming skills in Python.
Proven experience with Scrapy, Requests, BeautifulSoup, Selenium (if required).
Hands-on experience solving blocking issues, handling
Good understanding of proxy usage, rotation, and fingerprinting avoidance techniques.
Knowledge of HTTP, cookies, headers, sessions, and request/response cycles.
Experience with databases (SQL/NoSQL) for storing crawled data.
Strong debugging, problem-solving, and analytical skills.

Good to Have

Experience with cloud environments (AWS, GCP, Azure).
Knowledge of data pipelines and ETL processes.

Education & Experience

Bachelor’s degree in Computer Science, Information Technology, or equivalent practical experience.
3+ years of experience in web crawling or related fields

More Jobs at OSI Digital

Oracle Ebs Finance Functional Consultant

Chennai, Bengaluru, Hyderabad

5 - 10 yrs

INR 0 - 0 Lacs

Oracle Fusion/Cloud SCM Functional

Chennai, Bengaluru, Hyderabad

5 - 10 yrs

INR 0 - 0 Lacs

Data Architect_Remote

Hyderabad, Chennai, Bengaluru

8 - 13 yrs

INR 0 - 0 Lacs

Oracle Fusion/Cloud SCM Functional

Chennai, Tamil Nadu, India

5.0 - 10.0 yrs

INR 4 - 10 Lacs

Oracle Fusion/Cloud SCM Functional

Bengaluru / Bangalore, Karnataka, India

5.0 - 10.0 yrs

INR 4 - 10 Lacs

Mock Interview

Practice Video Interview with JobPe AI

Start Python Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.