Token
A token is the basic unit of text an LLM processes — roughly ¾ of a word for common English, though punctuation, numbers, and rare or technical words can each become their own token. Every API request is priced and limited by token count, not word or character count.
Where This Term Is Used
Related Terms
Context Window
The context window is the maximum number of tokens a model can process in a single request — the system prompt, conversation history, retrieved documents, the current message, and the model's own output all draw from this same shared budget. Content beyond the limit is truncated or the request is rejected.
System Prompt
A system prompt is a set of instructions sent with every request that defines how a model should behave — its role, tone, constraints, and output format — separate from the user's actual message. It counts against the same token budget as everything else in the request, so a longer system prompt leaves less room for conversation and output.
llms.txt
llms.txt is a plain markdown file placed at a website's root that gives AI systems a structured summary of the site's pages, similar to how a sitemap helps search engines. It was proposed by Jeremy Howard of Answer.AI as a lightweight convention AI crawlers can read instead of parsing full HTML.