19 Nov
|
TinEye
|
Toronto
Build large-scale web crawling infrastructure in Toronto with Python, Go, Rust, or C++ and asynchronous network programming expertise. This is an in-office position.
TinEye seeks a Web Crawling and Scraping Engineer for its small Toronto development team. You will design crawling strategies and operate high-throughput ingestion systems. The work connects crawled data with indexing and matching for TinEye’s image search technologies.
Key Responsibilities:
- Architect crawling strategies for client-requested page prioritization
- Maintain frontier scheduling, rate limiting, deduplication, parsing, and proxies
- Improve monitoring and logging for crawler performance and health
- Eliminate crawling and extraction bottlenecks for scalable resource usage
- Document debugging procedures and support data flow into indexing
Requirements:
- 4+ years designing large-scale crawling and indexing infrastructure
- Extensive distributed systems and real-time URL prioritization experience
- Expertise with crawling frameworks, scraping libraries, and HTTP protocols
- Experience with Puppeteer, Selenium, asynchronous I/O, queues, and storage
- Knowledge of SQL and NoSQL databases with solid problem-solving skills
Apply web crawling, distributed systems, and asynchronous programming expertise to TinEye’s Toronto image search infrastructure.
📌 Web Crawling Infrastructure Engineer (Toronto)
🏢 TinEye
📍 Toronto