Bytespider

About this crawler

Bytespider is ByteDance’s training crawler. ByteDance's crawler — used to train Doubao and other ByteDance AI products. Heavily active on Chinese-language sites; less so on English-language.

How this page works

1. Every request to findloc.ai passes through a Next.js middleware.
2. If the User-Agent matches a known AI bot pattern, one row is inserted into the ai_crawler_visits Postgres table. Fire-and-forget so the bot’s response isn’t slowed.
3. This page queries that table via three SECURITY DEFINER RPCs (ai_bot_hit_counts, ai_bot_daily_hits, ai_bot_top_paths) on every render.
4. Page response is cached for 1 hour via Next.js ISR — so DB load is bounded even under load, and the freshest number is always at most 60 minutes old.