Vector Databases for Beginners

In the realm of Artificial Intelligence (AI), the small token limit often emerges as a tricky terrain to navigate. Every text string is a token, and when tokens band together, they hit a boundary. Here’s where Vector Databases come to the rescue.

Transforming verbose text into compact vectors, they ease the passage through the token limitations, offering a smoother sail for AI developers. This blog post unfolds the magic of Vector Databases in transcending token hurdles, paving the way for robust and efficient AI applications.

Join me as we delve into how Vector Databases are becoming the compass for AI developers in the stormy seas of token limitations.

What are Vector Databases?

Dive into the digital realm where data takes on a new dimension with vector databases. Unlike typical databases that deal with text or numbers, vector databases are the shining knights when it comes to handling geometric data. They store, manage, and conjure up data in the form of points, lines, and polygons, making them a treasure for applications like geographic information systems (GIS) and computer-aided design (CAD).

Now, imagine a city planner crafting the blueprint of a future urban jungle. With a vector database, each road, building, and park is plotted with geometric precision, creating a digital mirror of the real world. It’s not just about static images but dynamic data that shifts and evolves, offering a playground for professionals to analyze spatial relationships and patterns.

Each entry in a vector database is like a brushstroke in a larger picture, defined by coordinates and connected by lines. This is where the magic happens – each point, line, and polygon is a vessel of information, ready to be queried, analyzed, and displayed. The digital landscapes they create are rich with details, serving as the backbone for many modern-day applications.

And guess what? Vector databases are not just smart but also efficient. They compress the spatial data, ensuring a lean and fast system, ready to deliver at the speed of thought. When it comes to harnessing the power of geometric data, vector databases are the unsung heroes, making the invisible, visible and the complex, simple. They are the silent engines driving forward a world full of spatial possibilities, one vector at a time. So, whether you’re navigating through the concrete jungles or designing the next architectural marvel, vector databases are your go-to companion.

Why Vector Databases Matter for AI so much?

The dance between vector databases and Artificial Intelligence (AI) is a sight to behold. In a world where data is king, vector databases offer a royal court for AI algorithms to thrive. Here’s a glimpse into their camaraderie:

Feast of Features: Vector databases are like a banquet of information where each geometric shape holds a treasure trove of features. AI thrives on such rich data, finding patterns and insights like a master detective on a hot trail.
Token Economy: When dealing with text, the limitation of token size often becomes a bottleneck. However, vector databases elegantly sidestep this hurdle. By storing data in a geometric format, they encapsulate more information in a compact manner, making it a smoother sail for AI algorithms.
Spatial Splendor: AI has a knack for understanding spatial relationships, and vector databases are its playground. They provide a spatial context that’s invaluable in many AI applications, be it autonomous vehicles navigating the streets or drones mapping out terrains.
Speed and Scale: In the grand theatre of data, speed and scale are the lead actors. Vector databases deliver on both fronts, ensuring AI algorithms get a fast and uninterrupted flow of data, fueling their quest for learning and evolving.
Precision Playground: Precision is the name of the game in AI, and vector databases are the perfect playing field. They ensure that the geometric data is accurate and high-quality, which is crucial for the success of AI applications.
Search and Similarity: Vector databases excel in similarity search, a prized skill in the AI arena. They enable swift and accurate searches even in high-dimensional spaces, making them a darling of machine learning and AI practitioners.
Real-time Revelry: The real world doesn’t wait and nor do vector databases. They empower AI systems to interact with geometric data in real time, unlocking a universe of dynamic applications.

In the enthralling narrative of AI, vector databases are the unsung heroes, offering a realm where algorithms can dance to the rhythm of geometric data, uncovering patterns and insights that fuel the innovation of tomorrow. The alliance of vector databases and AI is like a melody that’s orchestrated to explore the uncharted, solve the unsolvable, and imagine the unimaginable. So, when it comes to AI meeting vector databases, it’s not just a collaboration, it’s a celebration of what’s possible.

A Real-world Use-case for Vector Databases

Let’s venture into the realm of Smart Urban Planning using a fictional city called Technoville as our canvas.

Use-Case: Smart Urban Planning in Technoville

In the heart of Technoville lies an ambition to evolve into a smart city, a nexus of technology and urban living. The city planners have a vision to optimize traffic flow, reduce energy consumption, and enhance emergency services. To achieve this, they decide to deploy an AI-powered Urban Intelligence System. This system is envisioned to analyze real-time traffic data, monitor energy usage, and respond to emergency situations swiftly.

Now, let’s weave in ChatGPT and vector databases into this narrative:

Role of ChatGPT: The planners employ ChatGPT to interact with the Urban Intelligence System. ChatGPT, with its linguistic prowess, translates the planners’ queries into actionable insights. For instance, it helps in analyzing traffic patterns, suggesting optimal routes, and even drafting reports on energy consumption trends.

Token Limit Hurdle: However, the road to smart urban living isn’t without bumps. ChatGPT faces a challenge – the token limit. When trying to process vast amounts of textual data related to traffic and energy patterns, it hits the token limit which hinders its ability to analyze and generate comprehensive insights.

Vector Database to the Rescue: Enter vector databases, the unsung hero in this tale. The geometric data from various city sensors is stored in a vector database, encapsulating a wealth of information in a compact, geometric format. Instead of juggling cumbersome textual data, ChatGPT now interacts with the vector database to fetch and analyze geometric data.

Efficient Data Management: The vector database efficiently manages spatial data, reducing the load on ChatGPT. It can now fetch precise data points instead of wading through a sea of text, making the process leaner.
Enhanced Analysis: Vector databases enable ChatGPT to analyze spatial relationships effortlessly. For instance, it can now easily understand and analyze the traffic flow between different city zones, aiding in real-time decision-making.
Real-Time Insights: The speed at which vector databases deliver data ensures that ChatGPT can provide real-time insights to the planners. Be it rerouting traffic during peak hours or optimizing energy distribution, the decisions are now data-driven and timely.
Scalability: As Technoville grows, so does the data. Vector databases ensure that the system scales smoothly, always keeping pace with the evolving urban landscape.

The marriage of ChatGPT and vector databases in Technoville is nothing short of a technological romance, driving the city closer to its smart urban living dream. Together, they navigate the complex urban data, ensuring that Technoville is not just a smart city, but a wise one.

Converting a Database to Vector Database

Storing your data into a vector database is done via embeddings.
Text embeddings are the magical mirrors that reflect the kinship between text strings. They are usually summoned for:

Search Quests: Here, they rank the results based on how relevant they are to the curious query string.
Clustering Conclave: In this gathering, text strings unite under the banner of similarity.
Recommendation Revelry: This is where items sharing text string relations come recommended.
Anomaly Alert: In this scrutiny, the odd ones with little relatedness are spotted.
Diversity Delve: Here, we dive into the sea of similarity distributions to explore the landscape.
Classification Caravan: In this expedition, text strings find their tribe, getting classified by their closest label.

Imagine an embedding as a vector, a scroll of floating point numbers. The tale of relatedness between two vectors is told by the distance between them. A short distance whispers a tale of high relatedness while a long distance narrates a saga of low relatedness. Through the lens of embeddings, every text string embarks on a journey, finding its kin and understanding its stance in the vast kingdom of text.

Here is an example from OpenAI’s embeddings documentation:

Request
curl https://api.openai.com/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "input": "Your text string goes here",
    "model": "text-embedding-ada-002"
  }'

Response
{
  "data": [
    {
      "embedding": [
        -0.006929283495992422,
        -0.005336422007530928,
        ...
        -4.547132266452536e-05,
        -0.024047505110502243
      ],
      "index": 0,
      "object": "embedding"
    }
  ],
  "model": "text-embedding-ada-002",
  "object": "list",
  "usage": {
    "prompt_tokens": 5,
    "total_tokens": 5
  }
}

Using Embeddings in Prompts

Imagine you’re developing a smart recipe generator named “Culinario”. You’ve trained it on a plethora of recipes. Now, you want to find recipes similar to a specific recipe, say, a classic “Spaghetti Carbonara.” Instead of sending a text prompt, you decide to utilize the power of embeddings.

Firstly, you would convert the text of “Spaghetti Carbonara” recipe into a vector using a text embedding model. Now, your vector (embedding) may look something like this (simplified for illustration):

[0.12, -0.08, 0.23, -0.19, 0.45, ...]

Now, you craft a prompt to send to Culinario:

{
    "action": "find_similar_recipes",
    "recipe_embedding": [0.12, -0.08, 0.23, -0.19, 0.45, ...]
}

In this prompt:

The action field instructs Culinario on what to do – find similar recipes.
The recipe_embedding field carries the essence of Spaghetti Carbonara in a condensed vector form.

Upon receiving this prompt, Culinario dives into its treasure trove of recipe embeddings, seeking those that resonate with the essence of Spaghetti Carbonara. It then presents a bounty of recipes that share the soul of the classic dish, transporting you into a realm of culinary adventures that echo the heart of Spaghetti Carbonara!

In this whimsical interaction, the embedding empowers the prompt to convey the spirit of Spaghetti Carbonara, setting the stage for Culinario to unveil a world of similar recipes, thus making the quest for culinary exploration a delightful saga!

List of Some Vector Databases

Below is a list of vector databases, some of which are open-source (free) while others are paid:

Pinecone:
- Open Source: No
- Description: A managed, cloud-native vector database with a straightforward API. It facilitates speedy and accurate data processing, offering features like duplicate detection, rank tracking, and data search¹.
Milvus:
- Open Source: Yes
- Description: Designed to ease embedding similarity search and AI applications, Milvus simplifies unstructured data search, offering a consistent user experience across various deployment environments¹²³.
Chroma:
- Open Source: Yes
- Description: An AI-native embedding database aimed at simplifying the creation of Large Language Model (LLM) applications by making knowledge, facts, and skills pluggable for LLMs¹.
Pinecone, Milvus, KX:
- These databases are listed among the best free vector database software options, with Milvus and Pinecone specifically mentioned across different sources⁴.
DeepsetAI, Faiss by Facebook, pgvector, Pinecone, Supabase:
- These databases are listed as some of the best vector databases for building AI apps with embeddings⁵.

These databases cater to a variety of needs, from facilitating embedding similarity search to easing the creation of AI applications. Open-source options like Milvus and Chroma provide a cost-effective solution while offering a robust set of features to navigate the small token limit challenge efficiently. On the other hand, Pinecone, a paid solution, offers a managed environment, relieving users from infrastructure maintenance burdens.

These databases are a treasure trove for AI developers, offering a pathway to transcend the token limitations and venture into a realm where text data transforms into geometric vectors, unlocking new horizons of possibilities.