Embeddings and choosing a vector database

What embeddings do

An embedding model turns text into a vector, a fixed-length list of numbers. The model is trained so that texts with related meanings end up near each other. A question about "cancelling a subscription" can then match a passage about "ending your plan", even though the two share few words.

In a typical semantic search or RAG pipeline, you embed each stored passage ahead of time and save the vectors. When a query arrives, you embed it with the same model and look for the nearest stored vectors. The results are then returned directly or passed to an LLM as context.

Two limits are worth remembering. First, similarity is not truth or relevance: a close match can be out of date or simply about a neighbouring topic. Second, embeddings can blur exact details. Product codes, names and precise phrases are often better found by keyword search, which is why hybrid search and reranking are so widely used.

Choosing an embedding model

Published comparisons are a starting point at most, since your documents and your users' language may look very different from any general test set. These are the criteria to consider:

• Retrieval quality on your data. This outweighs everything else, and it is covered in the next section.
• Domain and language. Legal text, code, medical notes and mixed-language content can each behave differently. If users search in more than one language, test each one.
• Dimensions. More dimensions can hold more information, but they also use more storage and memory and can slow search. Some models let you choose a shorter output. Measure whether a smaller size loses anything that matters to you.
• Input length. Models have a maximum input size, and text beyond it may be cut off or rejected. Your chunking strategy has to fit within that limit.
• Hosted or self-hosted. A hosted API is simple to adopt but sends your text to a third party and depends on that provider's terms and availability. Running a model yourself gives more control, but you take on the hardware and operations.
• Cost at both stages. Include the initial embedding of everything, ongoing updates and query-time calls, plus query latency.
• Model stability. Find out how the provider handles model retirement, since a retired model can force you to re-embed everything.

Some models expect queries and documents to be treated differently, for example through a prefix or a separate setting. Read the model's documentation, as getting this wrong can quietly reduce quality.

Testing models on your own data

A small experiment tells you more than a general comparison:

1. Collect realistic queries. Use real user questions if you have them, including vague and badly worded ones.
2. Mark the right passages. For each query, record which passages would genuinely help.
3. Embed the same chunks with each candidate. Change only one thing at a time, or you will not know the cause of a difference.
4. Compare retrieval metrics. Recall and rank-aware measures at the number of results you actually use are good choices. How to evaluate a RAG system explains how to set these up.
5. Read the failures. Averages hide patterns, such as one model consistently failing on acronyms.

Here is an illustrative sketch of the core comparison. It uses exact search over normalised vectors, which is simple and fine for a small experiment:

# Illustrative only. embed() is a placeholder returning normalised vectors.
def recall_at_k(queries, passages, relevant, embed, k):
    doc_vectors = [embed(text) for text in passages]
    hits = 0
    for query_id, query in queries.items():
        query_vector = embed(query)
        scores = [
            (sum(a * b for a, b in zip(query_vector, vector)), index)
            for index, vector in enumerate(doc_vectors)
        ]
        top = {index for _, index in sorted(scores, reverse=True)[:k]}
        if top & relevant[query_id]:
            hits += 1
    return hits / len(queries)

Practical rules for storing embeddings

• Never mix models. Vectors from different models are not comparable. Queries and documents must use the same model, or a correctly paired one.
• Record the model with each vector. Store the model name and version as metadata, along with the source document and chunk ID.
• Keep the original text. You will need it to display results, and to re-embed when you change model or chunking.
• Match the distance metric. Cosine similarity, dot product and Euclidean distance are the usual options. Use what the model's documentation recommends. For normalised vectors, cosine and dot product produce the same ordering.
• Plan for re-embedding. Build a process that can write to a new index, compare it with the old one and switch over, so a model change does not become an emergency.

Do you need a dedicated vector database?

You have three broad choices, and the first may be enough for longer than you expect:

For a small collection, even exact search in memory can work. Start with what your team already operates unless a specific requirement rules it out, and keep retrieval behind your own small interface so that moving later is possible.

Criteria for choosing a vector store

These questions matter more than any ranking, and the answers will differ between projects:

Metadata filtering

Most real queries have conditions such as tenant, language, date or permission level. Ask how filtering interacts with vector search. Filtering after the search can leave too few results when the filter is selective, so test selective filters with realistic data.

Search quality and index type

Large collections generally use approximate nearest neighbour indexes, which trade a little recall for speed. Graph-based and cluster-based indexes are common examples. Look at what can be tuned, then measure recall against exact search on your data. Check also for built-in keyword or hybrid search if your project needs it.

Updates and deletes

How quickly does a new document become searchable? How are deletes handled, and when is data actually removed? This affects both freshness and privacy obligations. Some indexes need periodic rebuilds or compaction as data changes.

Scale and resource use

Estimate your vector count, dimensions, query rate and growth. Some indexes are memory-hungry, so check whether storage options such as disk-based indexes or compression suit your needs, and how they affect recall. Test with a realistic volume, since behaviour on a small sample may not carry over.

Security and isolation

For multi-tenant data, decide between separate collections and a tenant filter, then confirm that one customer can never see another's results. Check encryption, access controls and where data is stored. Vectors derived from sensitive text should be treated as sensitive too.

Operations

Consider backups and restore, monitoring, upgrades, how the system handles failure, and whether you want a managed service or to run it yourself. Also check how easily you can export your data, including the original text and metadata.

Common mistakes

• Tuning the database before checking retrieval. Poor results can come from chunking, the model or missing keyword search, so rule those out before blaming the store.
• Benchmarking without filters. A test without your real filters may not reflect production behaviour.
• Treating a similarity score as confidence. Scores are not comparable across models and do not directly mean "correct". Any threshold needs testing on your own data.
• Forgetting to sync. When a source document changes or is deleted, its old chunks must go too, or the system will keep retrieving them.

For help designing retrieval around your own content, see RAG development.

Frequently asked questions

Can I change embedding model later?

Yes, but you need to re-embed all your content, because vectors from different models are not comparable. Keeping the original text and building a new index alongside the old one makes this a manageable change.

Do I need a dedicated vector database for RAG?

Not necessarily. Vector support in a database you already run can be enough, particularly for smaller collections. A dedicated system makes more sense when a specific requirement cannot be met otherwise.

Are more embedding dimensions always better?

No. More dimensions can carry more information but use more storage and memory and may slow search. Compare options on your own retrieval tests before accepting the extra cost.

What is the best vector database?

There is no single best option. The right choice depends on your filtering, update, scale, security and operational needs, and on what your team already runs. Test candidates with your own data and real filters.