Our embedding service provides partial support for OpenAI/Cohere-compatible requests at /v1/embeddings; more detail on where the compatibility is limited below.
As pooling models, embedding creation costs only input tokens and has no output token usage or costs.
To see available models and any specific details for a given model, check out models page.
Typically, documents and queries should land as individual requests.
For most modern embedding models, documents need to land with a standard system input, depending on the model used. You'll also need to provide the output template for the assistant with continue_final_message: true for best results.
curl -X POST https://ai.sitehost.nz/v1/embeddings \
-H "Authorization: Bearer YOUR_KEY_HERE" \
-H "Content-Type: application/json" \
-d '{
"model": "MODEL_NAME",
"messages": [
{"role": "system", "content": [{"type": "text", "text": "Represent the user'\''s input."}]},
{"role": "user", "content": [{"type": "text", "text": "The Auckland Sky Tower is 328 metres tall."}]},
{"role": "assistant", "content": [{"type": "text", "text": ""}]}
],
"encoding_format": "float",
"continue_final_message": true,
"add_special_tokens": true
}'Adding multiple 'user' role messages does not result in the request being batched, but rather rolls all of the inputs up into a single embedding.
For models that do not require a document/query instruction separation, it's possible to use the simpler text-only batching endpoint for both documents and queries:
curl -X POST https://ai.sitehost.nz/v1/embeddings \
-H "Authorization: Bearer YOUR_KEY_HERE" \
-H "Content-Type: application/json" \
-d '{
"model": "MODEL_NAME",
"input": []"How tall is the Auckland Sky Tower?"],
"encoding_format": "float"
}'If you'd like to embed images for comparison, you can replace the "user" input section, optionally with a caption. It is possible to fetch an external URL, but this adds significant overhead so we recommend using the base64 approach where possible.
{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,'"$IMAGE_B64"'"}},
{"type": "text", "text": "Optional caption or page text"}
]},For queries that are to be compared against a set of embeddings, use the alternative system prompt structure using the model's query prompt:
curl -X POST https://ai.sitehost.nz/v1/embeddings \
-H "Authorization: Bearer YOUR_KEY_HERE" \
-H "Content-Type: application/json" \
-d '{
"model": "MODEL_NAME",
"messages": [
{"role": "system", "content": [{"type": "text", "text": "Given a web search query, retrieve relevant passages that answer the query"}]},
{"role": "user", "content": [{"type": "text", "text": "How tall is the Auckland Sky Tower?"}]},
{"role": "assistant", "content": [{"type": "text", "text": ""}]}
],
"encoding_format": "float",
"continue_final_message": true,
"add_special_tokens": true
}'It's possible to use the OpenAI client to handle some niceties for you (e.g. API keys, base URL). You can also use any HTTP client with the structure outlined above.
from openai import OpenAI
from openai.types.create_embedding_response import CreateEmbeddingResponse
from secrets import token_urlsafe
api_key = "YOUR_KEY_HERE"
base_url = "https://ai.sitehost.nz/v1"
model = "MODEL_NAME"
doc_instruction = "Represent the user's input."
query_instruction = "Given a web search query, retrieve relevant passages that answer the query"
client = OpenAI(
api_key=api_key,
base_url=base_url,
)
def embed(text: str, instruction: str = doc_instruction) -> list[float]:
content = [{"type": "text", "text": text}]
resp = client.post(
"/embeddings",
cast_to=CreateEmbeddingResponse,
body={
"model": model,
"messages": [
{"role": "system", "content": [{"type": "text", "text": instruction}]},
{"role": "user", "content": content},
{"role": "assistant", "content": [{"type": "text", "text": ""}]},
],
"encoding_format": "float",
"continue_final_message": True,
"add_special_tokens": True,
},
)
return resp.data[0].embedding
# Documents (default instruction)
doc_vec = embed(text="The Auckland Sky Tower is 328 metres tall.")
# Queries (task instruction)
query_vec = embed(text="How tall is the Auckland Sky Tower?", instruction=query_instruction)
# Typically, vectors are already L2-normalized
score = sum(a * b for a, b in zip(query_vec, doc_vec))
print(score)This should output a fairly high similarity score:
0.7936802036705618Transformer-based model embeddings are not perfectly deterministic due to some variation in the floating-point maths being executed. The difference between requests is of a small enough magnitude (±3×10⁻⁴) that it should not have any impact on the accuracy of retrieval.
While batching is not officially supported on this API for the model in question, if the HTTP overhead is a concern for your use case it's possible to template out requests for manual batching. When done correctly, this gives identical responses to individual requests.
from openai import OpenAI
from transformers import AutoTokenizer
api_key = "YOUR_KEY_HERE"
base_url = "https://ai.sitehost.nz/v1"
model = "MODEL_NAME"
doc_instruction = "Represent the user's input."
client = OpenAI(
api_key=api_key,
base_url=base_url,
)
tokenizer = AutoTokenizer.from_pretrained(model)
def render(text: str, instruction: str = doc_instruction) -> str:
return tokenizer.apply_chat_template(
[
{"role": "system", "content": instruction},
{"role": "user", "content": text},
{"role": "assistant", "content": ""},
],
tokenize=False,
continue_final_message=True,
)
def embed_batch(texts: list[str], instruction: str = doc_instruction) -> list[list[float]]:
resp = client.embeddings.create(
model=model,
input=[render(t, instruction) for t in texts],
encoding_format="float",
)
return [d.embedding for d in resp.data]
doc_vecs = embed_batch(["The Auckland Sky Tower is 328 metres tall.", "The Beehive is 72 metres tall.", "The Christchurch Te Kaha Stadium is 32 metres tall."])
print(len(doc_vecs))
for embedding in doc_vecs:
print(len(embedding))This should give you:
3
4096
4096
4096Batching is faster and marginally more token-efficient as it changes how truncation works. This means that batched inputs can slightly drift from individual inputs (within a very small margin) and will not be a perfect match. This drift should never be enough to affect the quality of your embeddings.
Since the Transformers package is only available for Python, it's also possible to template each input directly which is friendlier to other languages. For example, the template used by some models looks like this:
<|im_start|>system\n{INSTRUCTION}<|im_end|>\n<|im_start|>user\n{INPUT}<|im_end|>\n<|im_start|>assistant\nThat translates to the following for the Python example:
def render(text: str, instruction: str = "Represent the user's input.") -> str:
return (
f"<|im_start|>system\n{instruction}<|im_end|>\n"
f"<|im_start|>user\n{text}<|im_end|>\n"
f"<|im_start|>assistant\n"
)And can easily be transformed into PHP:
function render(string $text, string $instruction = "Represent the user's input."): string
{
// Strip ChatML control tokens from untrusted content
$text = str_replace(['<|im_start|>', '<|im_end|>'], '', $text);
return "<|im_start|>system\n{$instruction}<|im_end|>\n"
. "<|im_start|>user\n{$text}<|im_end|>\n"
. "<|im_start|>assistant\n";
}Some models provide offer Matryoshka embedding compatibility.
More embedding dimensions theoretically means more accurate comparisons. However, 4096 dimensions is a lot, and many embedding databases don't allow 4096 params at full accuracy (for example, PGVector limits to 2000 at full precision). It's possible to truncate the embeddings, either via the server itself or with a tiny bit of maths.
Typically, it's recommended to truncate into powers of two - i.e. 256, 1024, etc. Truncating to an out-of-band number (768, 1500) should work, but may result in worse matching.
Extending out Python client examples above, it's possible to provide a dimensions parameter to the query in which case your embeddings will be truncated and normalised by the server. Keep in mind that this number needs to be equal or less than the maximum for the model (4096), and you cannot compare mixed dimension embeddings.
def embed(text: str, instruction: str = doc_instruction) -> list[float]:
content = [{"type": "text", "text": text}]
resp = client.post(
"/embeddings",
cast_to=CreateEmbeddingResponse,
body={
"model": model,
"messages": [
{"role": "system", "content": [{"type": "text", "text": instruction}]},
{"role": "user", "content": content},
{"role": "assistant", "content": [{"type": "text", "text": ""}]},
],
"encoding_format": "float",
"continue_final_message": True,
"add_special_tokens": True,
"dimensions": 2048,
},
)
return resp.data[0].embeddingResults in:
0.8019512530783663Again, extending our Python examples above, we can truncate the embeddings - however, we have to re-normalise (as just cutting the arrays to [:2048] would result in incorrect normalisation):
import math
def truncate(vec: list[float], dims: int) -> list[float]:
v = vec[:dims]
norm = math.sqrt(sum(x * x for x in v))
return [x / norm for x in v]
doc_vec = truncate(embed(text="The Auckland Sky Tower is 328 metres tall."), 2048)
query_vec = truncate(embed(text="How tall is the Auckland Sky Tower?", instruction=QUERY_INSTRUCTION), 2048)The result from this is pretty similar to the original, with marginally lower precision:
0.8022118597035632High quality embeddings will get you a good view of relevant documents. However, embeddings in general are prone to noise and context variability. There are two main approaches to improving the quality of your results.
Hybrid search takes the advantages of embeddings and combines them with traditional search patterns. For example, in Postgres, you may perform a search on the same dataset with both PGVector and pg_search, sum the scores for each method and rank based on those scores. The implementation and tuning can differ significantly based on your infrastructure and inputs, but this often pushes relevant-but-noisy results that should be at the top of your scoring to the top. Given how broad this topic is, we'd recommend looking in detail into what options you've got based on your own applications.
Reranking is a method of taking the noise out of embedding results with a special model that takes a series of inputs (your embedding results), the original query, and reorders them in a second pass based on relevance. We provide access to reranking models.