Software Development Engineer II -Bot Detection

Role:

A key component of our Data Acquisition platform is the fleet of crawlers that we use to gather millions of web pages from hundreds of retailers every day. In order to gather data in a predictable and reliable manner, it s vital that each of our crawlers is able to navigate a retailer s website without being detected as being a bot.

The mission of our Bot Detection / evasion platform team is to provide tools and services that allow our crawlers to avoid detection and to gather data with high-levels of success. To achieve this, the team provides two key systems, Firstly, a service to provide good browsers, tailored to the needs and challenges of each retailer. Secondly, a library of captcha-solvers that help handle scenarios where detection is harder to avoid. We are looking for a highly skilled and technically curious software engineer to lead the team as we embark on the development of these vital systems.

What You will do:

• Lead a team of specialist backend software engineers in the development of new services to help avoid bot detection systems and to solve captchas


• Analyse bot-detection systems, establishing how they work and how to avoid them


• Reverse engineer highly-obfuscated bot-detection JavaScript libraries, and determine exactly what actions they perform.


• Collaborate with other senior engineers in the Data Acquisition tribe to determine an architectural approach to incorporate detection-avoidance capabilities into our wider platform.


• Take responsibility for the health of the team; fostering a positive and collaborative culture


• Be responsible for all aspects of the team s delivery and development; leading and championing agile processes, and driving continuous improvement.


• Balance hands-on development alongside team leadership responsibilities.


• Drive the growth and development of the team and its processes; exploring and championing new techniques, technologies and tools.


• Participate in engineering and architecture forums to help build a culture of excellence within the engineering organisation.

What Youll Bring:

• 7-9 years of experience in software engineering.


• 4+ years of experience in leading small to medium-sized teams.


• Expertise in scalable, distributed systems with strong problem-solving skills.


• Very strong understanding of web technologies: HTML, DOM, HTTP, and CSS


• Extensive experience and a deep understanding of JavaScript


• Significant experience working with Linux/Unix


• Strong understanding and hands-on experience with observability, system networking, performance optimizations.


• Experience working with cloud-based technologies and development processes


• Familiarity with SDLC and agile methodologies


• Strong understanding of modern coding practices and standards


• A passion for delivering quality software


• Experience with web-scraping and its challenges


• Strong ability to navigate ambiguity, break down complex problems, and drive solutions.

Nice to Haves:

• Experience with TypeScript


• Experience of building services in the cloud.


• Experience of containerisation using Docker and Kubernetes.


• Experience of building infrastructure as code, especially with Terraform.