19 Nov
|
TinEye
|
Toronto
Build large-scale web crawling infrastructure in Toronto with Python, Go, Rust, or C++ and asynchronous network programming expertise. This is an in-office position.TinEye seeks a Web Crawling and Scraping Engineer for its small Toronto development team. You will design crawling strategies and operate high-throughput ingestion systems. The work connects crawled data with indexing and matching for TinEye's image search technologies.Key Responsibilities:Architect crawling strategies for client-requested page prioritizationMaintain frontier scheduling, rate limiting, deduplication, parsing,
and proxiesImprove monitoring and logging for crawler performance and healthEliminate crawling and extraction bottlenecks for scalable resource usageDocument debugging procedures and support data flow into indexingRequirements:4+ years designing large-scale crawling and indexing infrastructureExtensive distributed systems and real-time URL prioritization experienceExpertise with crawling frameworks, scraping libraries, and HTTP protocolsExperience with Puppeteer, Selenium, asynchronous I/O, queues, and storageKnowledge of SQL and NoSQL databases with solid problem-solving skillsApply web crawling, distributed systems, and asynchronous programming expertise to TinEye's Toronto image search infrastructure.
📌 Web Crawling Infrastructure Engineer (Toronto)
🏢 TinEye
📍 Toronto