🥞
Store and load FAISS VectorDB
Here you will learn how to store and load a FAISS VectorDB.
First, we are going to assume that you have prepared the chunks you would like to store and created a faiss db through something like
api_key = db.secrets.get("OPENAI_API_KEY")
VStore = FAISS.from_texts(
texts, OpenAIEmbeddings(openai_api_key=api_key), metadatas=metadatas
)
Then, to persist the faiss database, you can use the Databutton binary storage to store the index as follows
def store_index_in_db(index, name):
faiss.write_index(index.index, "docs.index")
with open("docs.index", "rb") as file:
db.storage.binary.put(f"{name}.faiss.index", file.read())
index.index = None
db.storage.binary.put(f"{name}.faiss.pkl", pickle.dumps(index))
store_index_in_db(VStore, "myvectordb")
To load it from storage again, you can use the following function
def load_index_from_db(index_name):
findex = db.storage.binary.get(f"{index_name}.faiss.index")
with open("docs.index", "wb") as file:
file.write(findex)
index = faiss.read_index("docs.index")
VectorDB = pickle.loads(db.storage.binary.get(f"{index_name}.faiss.pkl"))
VectorDB.index = index
return VectorDB
VStore = load_index_from_db("myvectordb")
To list all the faiss indexes you have saved with the above methods, you can use
files = db.storage.binary.list()
s = pd.Series([r.name for r in l])
faiss_list = list(s[s.str.contains("faiss")].str.split(".", expand=True)[0].unique())
To delete an index, you can use the following function
def delete_index(index_name):
file1 = f"{index_name}.faiss.index"
file2 = f"{index_name}.faiss.pkl"
db.storage.binary.delete(file1)
db.storage.binary.delete(file2)
Last modified 15d ago