I Crawled 65,000 Pages of My Own Site Without Parsing a Single Line of HTML
Somewhere between talks on day one, I think it was during a hallway conversation about how badly LLM crawlers struggle with rendered DOMs. A question I had been chewing on for months finally crystalized...