Reverse proxy all pages including standard request pages
Classify origin by CIDR IP range
Update regularly via cron
https://raw.githubusercontent.com/searchtheoryio/ips/refs/heads/main/dist/ips.json
Yes they do change
https://www.linkedin.com/posts/ryansiddle_woah-openai-just-added-512-new-ip-addresses-activity-7323590222904995843-UIs3/
Must log to PH directly to pass the data properly
https://simon-thompson.me/p/tracking-published-crawler-ip-ranges
https://help.oncrawl.com/en/articles/11872799-how-to-detect-and-analyze-ai-and-llms-bots-hits-using-log-monitoring
Last updated 11 months ago
