Vector Databases You can choose From

Vector databases offer efficiency and scalability, and are transforming the way we harness the potential of embedding data in the digital age. There are quite a few open source vector databases that come with their own benefits. We take a quick look at them.

- Mohit Raj

In natural language processing (NLP), an embedding is a representation of text in the form of vectors. The goal of an embedding is to capture the semantic meaning of words or documents in a way that can be understood by a machine learning model.

A vector database (or an embedding database) in NLP is a specialised database designed to efficiently store, retrieve, and perform operations on high-dimensional vector data (such as the embeddings mentioned above). Vector databases are optimised to perform nearest neighbour search operations efficiently, which is a common requirement in NLP applications. They provide a way of organising and searching through large amounts of embedding data, which can be beneficial in various tasks like information retrieval, document similarity, clustering, and others.

As an example, let’s say you’ve embedded a large number of documents using a Doc2Vec model. Now, given a new document, you want to find the most similar documents in your database. To do this, you would:

1. First, embed the new document into the same high-dimensional space.

2. Next, search the vector database for the vectors closest to the new document’s vector. This is the nearest neighbour search.

Due to the high-dimensional nature of the data, this search can be computationally intensive. However, vector databases use specialised indexing and querying algorithms (like k-d trees, ball trees, or hashing techniques) to speed up these operations. Examples of such databases include FAISS developed by Facebook AI and Annoy developed by Spotify.

Open source vector databases 

Weaviate: