Create text tokens

This endpoint can be used to convert strings into tokens. It is a simple proxy forwarding your requests to the desired model. Any LightOn model is deployed on a vLLM-based image. Supported Input:

• prompt: Simple text string to tokenize
• messages: Array of chat messages to tokenize (alternative to prompt)

POST /api/v2/tokenize

Try it

cURL

TextTokenizationExample

curl --request POST \
  --url https://paradigm.lighton.ai/api/v2/tokenize \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "model": "alfred-40b-1123",
  "prompt": "Hello, how are you today?"
}
'

200

TextTokenizationExample

{
  "model": "alfred-40b-1123",
  "prompt": "Hello, how are you today?"
}

Authorizations

Authorization · string · header · required

Bearer token authentication

Body

Request serializer for tokenize endpoint.

model · string · required

Model to use for tokenization, must exist and be configured from the admin

prompt · string

The text to tokenize

messages · object[]

List of messages to tokenize (alternative to prompt)

Show child attributes

messages. {key} · any

Response

200

application/json

Successful response

Response serializer for tokenize endpoint results.

total_tokens · integer · required

Total number of tokens in the input text

model · string · required

The model used for tokenization

tokens · integer[]

List of token IDs from the tokenization

prompt_tokens · integer

Number of tokens in the prompt (alias for total_tokens)

object · string · default: tokenization

Object type, always 'tokenization'