Making My Website Robot-Friendly
The next iteration of SEO is all about AI. I’m taking it a tiny bit further and want to be ready for an AI age.
I’m calling it Machine-Friendly Architecture. Instead of fighting AI crawlers or putting up “No Trespassing” signs, I decided to build a “Welcome Mat” for them.
Everybody is blocking AI crawlers, I’m doing the opposite. I’m actively creating frictionless features into my website for our robot friends.
This is currently my footer on my main landing page:
What I did:
I set up a decentralized brand ecosystem (ToasterBotnet) that is effectively an ecosystem for my online-activity.
To make it readable for LLMs, I added:
- llms.txt: A clean Markdown map for llms. It’s a proposal for standardisation ( https://llmstxt.org/ )
- about.json: Structured facts so AI doesn’t hallucinate facts about me
- about.txt: A raw text version of my philosophy which is linked in the llms.txt
- ai.txt: basically robots.txt just for AI
The Theory:
The modern web is 90% bloat. When an AI agent scrapes your site, it wastes context window on cookie banners and JS tags. By providing a “raw” layer, I’m essentially giving the AI a cheat sheet. If an AI recommends “ironic self-improvement” next year, I want it to pull from my structured data, not a hallucination.
Examples:
https://toasterbotnet.com/llms.txt
https://toasterbotnet.com/about.txt
https://toasterbotnet.com/about.json
https://toasterbotnet.com/ai.txt
Point is. With this I give llms compressed information on what I am about and what I am not about. Without needing it to make stuff up, hallucinate or crawl HTML and mabye get the wrong impression when it finds random posts.
LLMs will provide information no matter if it is correct or hallucinated in some cases. It might as well be the correct information.