Skip to main content
POST
/
api
/
v2
/
tokenize
curl --request POST \
  --url https://paradigm.lighton.ai/api/v2/tokenize \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{
  "model": "alfred-40b-1123"
}'
{
  "model": "alfred-40b-1123",
  "prompt": "Hello, how are you today?"
}

Authorizations

​
Authorization
string
header
required

Bearer token authentication

Body

Request serializer for tokenize endpoint.

​
model
string
required

Model to use for tokenization, must exist and be configured from the admin

​
prompt
string

The text to tokenize

​
messages
object[]

List of messages to tokenize (alternative to prompt)

Response

Successful response

Response serializer for tokenize endpoint results.

​
total_tokens
integer
required

Total number of tokens in the input text

​
model
string
required

The model used for tokenization

​
tokens
integer[]

List of token IDs from the tokenization

​
prompt_tokens
integer

Number of tokens in the prompt (alias for total_tokens)

​
object
string
default:tokenization

Object type, always 'tokenization'

⌘I