Question 1

What is an LLM token, and why does token count matter for API pricing and context limits?

Accepted Answer

An LLM token is a unit of text after the model tokenizer splits input into subwords or symbols. APIs bill and enforce limits in tokens rather than raw characters, and each model tokenizer can split the same Unicode differently. This tool counts tokens locally so you can estimate prompt size, remaining context window, and rough cost before sending a request.

Question 2

How does this token counter compare to provider usage dashboards?

Accepted Answer

When the UI shows tiktoken-compatible counting, the WASM build uses the same CoreBPE encodings as tiktoken for the named model. Official dashboards may still differ slightly due to hidden system prefixes, formatting, or tool calls. Rows labeled proxy BPE or heuristic are approximations; use the provider for authoritative billing.

Question 3

What do tiktoken, proxy BPE, and heuristic mean in the token note?

Accepted Answer

Tiktoken means the count uses CoreBPE for the catalog model name. Proxy BPE loads a documented base vocabulary such as cl100k_base or o200k_base when the catalog maps a provider model to that encoding. Heuristic divides character count by four and rounds up; it is the coarsest mode and appears only when no BPE can be loaded.

Question 4

Why can the same pasted text get different token counts per model?

Accepted Answer

Each model uses a tokenizer vocabulary and merge table. GPT-4-class and GPT-5-class encodings may differ; multilingual and code-heavy text changes average bytes per token. Changing the model dropdown switches the tokenizer path so you can compare counts side by side.

Question 5

What does token highlighting show?

Accepted Answer

The result card colors each tokenizer piece so adjacent tokens alternate. tiktoken and proxy BPE use the same CoreBPE split as the count; heuristic mode chunks about four characters. Hover a span for its index. Very long prompts show the first 400 tokens.

Question 6

How does context window percentage work with the reserve field?

Accepted Answer

The context window is the maximum input tokens the model accepts for the entry in the catalog. The tool adds your counted tokens to the integer reserve (typically headroom for model output such as max_tokens). The percentage equals (tokens plus reserve) divided by the published context window. If the context is zero or unknown, the percentage is not meaningful.

Question 7

How are estimated input, cached-input, and output costs computed?

Accepted Answer

Estimated input multiplies your token count by the catalog input per million tokens rate. The cached estimate multiplies the same token count by the optional cached-input rate and assumes all tokens qualify for that tier; real billing may split cached versus uncached prefixes. Output estimate multiplies the reserve by the output per million token rate. Missing rates display as an em dash.

Question 8

Is the reference table a guarantee of charges?

Accepted Answer

No. Figures are bundled reference data with an as-of date and a pricing note. Providers change tiers, discounts, and regional pricing. External links point to official pages for verification. The table supports comparison and planning, not a binding quote.

Question 9

Is prompt text uploaded to a server?

Accepted Answer

No. Tokenization runs in WebAssembly in your browser. The implementation only stores UI preferences in localStorage. Your pasted prompt text does not leave the page for this tool in the way the client is built.

Question 10

How does split mode combine system, user, and assistant fields?

Accepted Answer

Non-empty sections are trimmed and joined with blank lines between sections. The combined string is tokenized. Assistant is optional for simulating prior turns; the tool does not call any API.

Question 11

What does Add Output do, and how many blocks can I add?

Accepted Answer

Add Output captures the model and pricing options from Model & options at that moment and appends a new result card for the same combined prompt. Use it to compare token counts, usage percentage, and indicative costs across different catalog rows or custom rates. You can add up to 5 blocks; remove a block with Remove if you need a slot for another configuration.

Question 12

What does including or excluding newlines in total characters change?

Accepted Answer

It only affects the character statistics cards, not the token count. Newlines remain in the combined text used for tokenization. Use it when you want editor-style character totals that include or exclude line breaks.

Question 13

How often is the reference catalog updated?

Accepted Answer

The catalog JSON is shipped as /data/llm_models.json and loaded in the browser; it includes a catalog_as_of field in the reference section. Regenerate that file when upstream pricing changes (see scripts/generate_llm_models.py). Always verify production budgets against the provider.

Question 14

Can I export the reference table?

Accepted Answer

Yes. Use Copy table (TSV) to copy tab-separated values for spreadsheets. Sorting and filters apply to the copied rows.

Question 15

Why might billed tokens differ from this counter in production?

Accepted Answer

Providers may add system or tool prefixes, format JSON or XML, re-tokenize after tool results, or classify cached tokens differently. Streaming endpoints and batch jobs may round differently. Treat this tool as planning; use API metering for invoices.

Question 16

Which OpenAI-style encodings does tiktoken-rs resolve in this WASM build?

Accepted Answer

The catalog maps model names and fallback keys such as o200k_base or cl100k_base to CoreBPE instances that match OpenAI tiktoken selections for compatible models.

GPT Token Counter – Count Prompt Tokens for GPT-4o, Claude, Gemini Before API Calls

GPT Token Counter – Count Prompt Tokens for GPT-4o, Claude, Gemini Before API Calls