What are Tokens in AI? | AI Glossary

Intermediate

TLDR

In AI language models, a token is the basic unit of text the model reads and generates, roughly equivalent to three to four characters or about three-quarters of a word in English.

AI language models do not process text the way humans read it, word by word or letter by letter. Instead, they break text into tokens: chunks of characters that the model has learned to recognize as meaningful units. A single token might be a whole common word ("cat", "the", "and"), part of a longer word ("un-" from "unbelievable"), a punctuation mark, or even a space. The exact tokenization depends on the model.

Tokens matter for two practical reasons: pricing and context window limits. AI API pricing is almost always calculated per token, both for the input you send and the output you receive. A call that sends 1,000 words and receives a 500-word reply is consuming roughly 2,000 tokens. Context window sizes are also measured in tokens, not words or pages, which is why the same token count translates to different amounts of text in different languages.

For most users of consumer AI tools like ChatGPT or Claude, token counts are invisible, managed automatically behind the scenes. For developers building on AI APIs, token awareness is essential for cost management, rate limit planning, and designing prompts that stay within context window limits.

In practice

API cost estimation

A developer building a customer support bot estimates that each support conversation averages 800 input tokens and 200 output tokens. At $3 per million tokens, each conversation costs less than half a cent, allowing them to project costs at scale.

Prompt length planning

A team using Claude's 200,000-token context window wants to process legal contracts. Knowing that one token is roughly 4 characters, they calculate that a 200-page contract (~100,000 words) uses around 130,000 tokens, comfortably within the limit.

Rate limit management

An app hitting OpenAI's token-per-minute rate limits restructures its requests to batch smaller queries, reducing the number of API calls without changing the total volume of work.

Frequently asked questions

How many tokens is 1,000 words?+

Roughly 1,300 to 1,500 tokens for standard English prose. As a rule of thumb, one word in English is approximately 1.3 tokens. This varies by vocabulary: technical text and code tend to use more tokens per word than conversational language.

How much do tokens cost?+

Token pricing varies by model and provider. In 2026, prices range from under $0.10 per million tokens for lightweight models to $15 or more per million tokens for frontier models like GPT-4o and Claude Opus. Input tokens (what you send) are usually cheaper than output tokens (what the model generates). Consumer products like ChatGPT Plus charge a flat monthly fee rather than per-token pricing.

Is a token the same as a word?+

Not exactly. Common short words are often one token ("the", "is", "a"). Longer or less common words may be split into multiple tokens ("unbelievable" might be 3-4 tokens). Punctuation, spaces, and code symbols each count as tokens too. As a rough guide, 100 tokens is approximately 75 English words.

Do different AI models count tokens the same way?+

No. Each model uses its own tokenizer, and the same text may produce a different token count in GPT-4 vs Claude vs Gemini. The differences are usually small for English text but can be larger for other languages, code, or specialized vocabulary. Each provider offers tokenizer tools so developers can calculate exact counts for their specific model.

Bottom line

In AI language models, a token is the basic unit of text the model reads and generates, roughly equivalent to three to four characters or about three-quarters of a word in English.