I’d like to take a moment to talk about a technology that I’m particularly looking forward to utilizing in my capstone project this quarter: a vector database. In the current technology landscape where big data reigns supreme, vector databases are pivotal when it comes to powering advanced search and recommendation systems – especially in artificial intelligence and machine learning-powered applications. My project intends to serve such a recommendation system, and the vector magic that happens behind the scenes in a vector database will make it all possible.
Like relational databases, vector databases store, manage, and retrieve data. Relational databases organize this data into interrelated entities of rows and columns, which is perfect for handling structured data and supporting precise, conditional queries. Vector databases on the other hand store data as vectors. Vectors – more or less points in multi-dimensional space – can be thought to be similar based on their proximity to one another. Thus utilizing vectors enables matches based on similarity rather than precision when making queries against the data, which makes something like a recommendation engine possible with unstructured data like text, images and audio.
For our vector database, we’ve decided to use Weaviate. In our project, we first collect text data using a web crawler and store it into a PostgreSQL database after some cleanup and processing. The data then goes through a process called “embedding”, where the data is converted into vector representation, and ultimately stored into a vector database for the recommendation engine. Weaviate has built-in machine learning models and can automatically handle embedding for us to greatly streamline this process. This, along with real-time indexing, allows newly crawled and processed data to be searchable almost instantly.
Anyway, we haven’t quite made it that far in our project just yet, but it’s hard to contain my excitement about seeing something like Weaviate in action. I’ve really enjoyed diving into the workings of vector databases, not only for our project but also because of how important vector data has become across the industry. More and more companies are trying to leverage artificial intelligence and machine learning in their products and relying on vector databases like Weaviate to do so!
For more information about Weaviate, check out the official documentation.
Leave a Reply