guides/advanced-usage.md +17 −16
37 37
38- How much your API call costs, as you pay per token38- How much your API call costs, as you pay per token
39- How long your API call takes, as writing more tokens takes more time39- How long your API call takes, as writing more tokens takes more time
4040- Whether your API call works at all, as total tokens must be below the model's maximum limit (4097 tokens for `gpt-3.5-turbo`)- Whether your API call works at all, as input and output tokens must fit within the model's context window (see the [model pages](https://developers.openai.com/api/docs/models) for limits)
41 41
42Both input and output tokens count toward these quantities. For example, if your API call used 10 tokens in the message input and you received 20 tokens in the message output, you would be billed for 30 tokens. Note however that for some models the price per token is different for tokens in the input vs. the output (see the [pricing](https://openai.com/api/pricing) page for more information).42Both input and output tokens count toward these quantities. For example, if your API call used 10 tokens in the message input and you received 20 tokens in the message output, you would be billed for 30 tokens. Note however that for some models the price per token is different for tokens in the input vs. the output (see the [pricing](https://openai.com/api/pricing) page for more information).
43 43
44To see how many tokens are used by an API call, check the `usage` field in the API response (for example, `response['usage']['total_tokens']`).44To see how many tokens are used by an API call, check the `usage` field in the API response (for example, `response['usage']['total_tokens']`).
45 45
4646Chat models like `gpt-3.5-turbo` and `gpt-4-turbo-preview` use tokens in the same way as the models available in the completions API, but because of their message-based formatting, it's more difficult to count how many tokens will be used by a conversation.Chat models use additional tokens for message formatting, so counting the text alone doesn't give the full input token count. For Responses API requests, use the [input token counting API](https://developers.openai.com/api/docs/guides/token-counting) to count tokens before creating a response.
47 47
48 48
49 49
5050Below is an example function for counting tokens for messages passed to `gpt-3.5-turbo-0613`.The examples below use `gpt-3.5-turbo-0613`, which was [retired on September 13, 2024](https://developers.openai.com/api/docs/deprecations#2023-11-06-chat-model-updates).
51 51
5252The exact way that messages are converted into tokens may change from model to model. So when future model versions are released, the answers returned by this function may be only approximate.This local counter applies only to text messages for that historical snapshot. Message formatting varies by model.
53 53
54```python54```python
55import tiktoken
56
57
55def num_tokens_from_messages(messages, model="gpt-3.5-turbo-0613"):58def num_tokens_from_messages(messages, model="gpt-3.5-turbo-0613"):
56 """Returns the number of tokens used by a list of messages."""59 """Returns the number of tokens used by a list of messages."""
57 try:60 try:
58 encoding = tiktoken.encoding_for_model(model)61 encoding = tiktoken.encoding_for_model(model)
59 except KeyError:62 except KeyError:
60 encoding = tiktoken.get_encoding("cl100k_base")63 encoding = tiktoken.get_encoding("cl100k_base")
6164 if model == "gpt-3.5-turbo-0613": # note: future models may deviate from this if model == "gpt-3.5-turbo-0613":
62 num_tokens = 065 num_tokens = 0
63 for message in messages:66 for message in messages:
6467 num_tokens += ( num_tokens += 3 # Message formatting overhead.
65 4 # every message follows <im_start>{role/name}\n{content}<im_end>\n
66 )
67 for key, value in message.items():68 for key, value in message.items():
68 num_tokens += len(encoding.encode(value))69 num_tokens += len(encoding.encode(value))
6970 if key == "name": # if there's a name, the role is omitted if key == "name":
7071 num_tokens += -1 # role is always required and always 1 token num_tokens += 1 # Extra formatting for a named message.
7172 num_tokens += 2 # every reply is primed with <im_start>assistant num_tokens += 3 # Assistant reply priming.
72 return num_tokens73 return num_tokens
73 raise ValueError(74 raise ValueError(
74 f"num_tokens_from_messages() only supports gpt-3.5-turbo-0613, not {model}."75 f"num_tokens_from_messages() only supports gpt-3.5-turbo-0613, not {model}."
76```77```
77 78
78 79
7980Next, create a message and pass it to the function defined above to see the token count, this should match the value returned by the API usage parameter:Count the example messages locally:
80 81
81```python82```python
82messages = [83messages = [
113model = "gpt-3.5-turbo-0613"114model = "gpt-3.5-turbo-0613"
114 115
115print(f"{num_tokens_from_messages(messages, model)} prompt tokens counted.")116print(f"{num_tokens_from_messages(messages, model)} prompt tokens counted.")
116117# Should show ~126 total_tokens# 129 prompt tokens counted.
117```118```
118 119
119 120
120121To confirm the number generated by our function above is the same as what the API returns, create a new Chat Completion:The following historical example shows how to read the API's prompt token count. It cannot run against the retired model:
121 122
122```javascript123```javascript
123import OpenAI from "openai";124import OpenAI from "openai";
172 173
173Each message passed to the API consumes the number of tokens in the content, role, and other fields, plus a few extra for behind-the-scenes formatting. This may change slightly in the future.174Each message passed to the API consumes the number of tokens in the content, role, and other fields, plus a few extra for behind-the-scenes formatting. This may change slightly in the future.
174 175
175176If a conversation has too many tokens to fit within a model’s maximum limit (for example, more than 4097 tokens for `gpt-3.5-turbo` or more than 128k tokens for `gpt-4o`), you will have to truncate, omit, or otherwise shrink your text until it fits. Beware that if a message is removed from the messages input, the model will lose all knowledge of it.If a conversation has too many tokens to fit within a model’s context window, you will have to truncate, omit, or otherwise shrink your text until it fits. Beware that if a message is removed from the messages input, the model will lose all knowledge of it.
176 177
177178Note that long conversations are more likely to receive incomplete replies. For example, a `gpt-3.5-turbo` conversation that is 4090 tokens long will have its reply cut off after just 6 tokens.Leave room in the context window for output tokens, and check the model's maximum output token limit when setting an output budget.
178 179
179## Parameter details180## Parameter details
180 181