Build a Local LLM Knowledge Graph of Your Notes: Ollama, FAISS, Neo4j

If you’ve kept notes for a few years, in Markdown files, Obsidian, exported docs or a folder of text files, you’ve probably had the experience of knowing you wrote something down and not being able to find it. Search helps if you remember the exact words. It doesn’t help with questions like “which projects did I discuss with Ali?” or “what have I written about caching?”

A small knowledge graph, built and queried with a local language model, can answer those questions without sending your notes to anyone. This guide walks through a working setup that runs on a normal laptop, using free tools: Ollama for the models, FAISS for semantic search and Neo4j for the graph.

What we’re building

The system has two complementary indexes over the same notes:

A vector index for semantic search. Each chunk of a note is turned into an embedding, a list of numbers representing its meaning, so a search for “caching” can find a note about “Redis TTLs” even if the word “caching” never appears.
A graph of entities and relationships. A local LLM reads each note and extracts people, projects, tools and topics, and how they relate. That lets you ask structured questions: every note that mentions both a person and a project, or every tool used in a given project.

At query time, you retrieve relevant chunks from the vector index, pull in related entities from the graph, and pass both to the local LLM to write an answer that cites the source notes.

What you need

Hardware: a laptop or desktop with at least 16 GB of RAM. A GPU helps but isn’t required for a personal note collection.
Ollama to run models locally.
Python 3.10+ with a few packages.
Neo4j Community Edition, easiest to run with Docker.

# Models: one for embeddings, one small chat model
ollama pull nomic-embed-text
ollama pull llama3.1:8b

# Python packages
pip install ollama faiss-cpu numpy neo4j

# Neo4j in Docker (change the password)
docker run -d --name neo4j -p 7474:7474 -p 7687:7687 \
  -e NEO4J_AUTH=neo4j/change-this-password neo4j:5

Any capable instruction-following model that fits your hardware will work. Smaller models run faster but extract entities less reliably, so try a couple on your own notes.

Step 1: Load and chunk your notes

Long notes should be split into chunks so each embedding represents one idea. Splitting on blank lines (paragraphs) and grouping them up to a size limit is simple and works well for personal notes.

from pathlib import Path

def load_chunks(folder, max_chars=1200):
    chunks = []
    for path in Path(folder).rglob("*.md"):
        text = path.read_text(encoding="utf-8", errors="ignore")
        buf = ""
        for para in text.split("\n\n"):
            if len(buf) + len(para) > max_chars and buf:
                chunks.append({"note": str(path), "text": buf.strip()})
                buf = ""
            buf += para + "\n\n"
        if buf.strip():
            chunks.append({"note": str(path), "text": buf.strip()})
    return chunks

chunks = load_chunks("notes")

Step 2: Build the vector index

We embed each chunk with a dedicated embedding model. Note that embedding models and chat models are different things: nomic-embed-text is designed to produce embeddings, while the chat model generates text. Normalising the vectors and using an inner-product index gives cosine similarity.

import numpy as np, faiss, ollama

def embed(texts):
    res = ollama.embed(model="nomic-embed-text", input=texts)
    vecs = np.array(res["embeddings"], dtype="float32")
    faiss.normalize_L2(vecs)
    return vecs

vectors = np.vstack([embed([c["text"] for c in chunks[i:i+32]])
                     for i in range(0, len(chunks), 32)])

index = faiss.IndexFlatIP(vectors.shape[1])
index.add(vectors)
faiss.write_index(index, "notes.faiss")

A flat index compares the query with every vector. That sounds slow, but for a few thousand or even tens of thousands of chunks it’s fast and exact. Approximate indexes such as IVF only become worthwhile at much larger sizes.

Step 3: Extract entities and relationships

Now we ask the local model to read each chunk and return structured JSON. Asking for a fixed, small set of entity types keeps the graph tidy.

import json

PROMPT = """Extract entities and relationships from the note below.
Return JSON only, in this shape:
{"entities":[{"name":"...","type":"Person|Project|Tool|Topic"}],
 "relations":[{"source":"...","relation":"WORKS_ON|USES|ABOUT|KNOWS","target":"..."}]}
Only include things clearly stated in the note.

Note:
"""

def extract(text):
    resp = ollama.chat(
        model="llama3.1:8b",
        messages=[{"role": "user", "content": PROMPT + text}],
        format="json",
        options={"temperature": 0},
    )
    try:
        return json.loads(resp["message"]["content"])
    except json.JSONDecodeError:
        return {"entities": [], "relations": []}

Setting format="json" tells Ollama to constrain the output to valid JSON, and temperature 0 makes results more consistent. Expect some mistakes: a small model will occasionally miss an entity or invent a relationship. For personal notes that’s usually acceptable, and you can spot-check a sample.

Step 4: Load the graph into Neo4j

We use MERGE so the same entity mentioned in many notes becomes a single node. Each entity is also linked to the chunk it came from, so answers can cite sources.

from neo4j import GraphDatabase

driver = GraphDatabase.driver("bolt://localhost:7687",
                              auth=("neo4j", "change-this-password"))

for i, c in enumerate(chunks):
    data = extract(c["text"])
    driver.execute_query(
        "MERGE (ch:Chunk {id:$id}) SET ch.note=$note, ch.text=$text",
        id=i, note=c["note"], text=c["text"])
    for e in data.get("entities", []):
        driver.execute_query(
            """MERGE (n:Entity {name:$name}) SET n.type=$type
               WITH n MATCH (ch:Chunk {id:$id}) MERGE (ch)-[:MENTIONS]->(n)""",
            name=e["name"].strip(), type=e.get("type", "Topic"), id=i)
    for r in data.get("relations", []):
        driver.execute_query(
            """MERGE (a:Entity {name:$s}) MERGE (b:Entity {name:$t})
               MERGE (a)-[:RELATED {type:$rel}]->(b)""",
            s=r["source"].strip(), t=r["target"].strip(), rel=r["relation"])

Extraction is the slow part, because it runs the chat model once per chunk. On a laptop without a GPU, a few thousand chunks can take hours, so run it overnight the first time and afterwards only process new or changed notes.

Step 5: Ask questions

The query function combines both indexes: find the most similar chunks, find the entities they mention and what those entities connect to, then ask the model to answer using only that context.

def ask(question, k=5):
    qv = embed([question])
    _, ids = index.search(qv, k)
    hits = [chunks[i] for i in ids[0] if i != -1]

    records, _, _ = driver.execute_query(
        """MATCH (ch:Chunk)-[:MENTIONS]->(e)-[r:RELATED]-(other)
           WHERE ch.id IN $ids
           RETURN e.name AS a, r.type AS rel, other.name AS b LIMIT 30""",
        ids=[int(i) for i in ids[0]])
    facts = "\n".join(f"{r['a']} {r['rel']} {r['b']}" for r in records)

    context = "\n\n".join(f"[{h['note']}]\n{h['text']}" for h in hits)
    prompt = (f"Answer using only the notes and facts below. "
              f"Cite note file names.\n\nFacts:\n{facts}\n\nNotes:\n{context}"
              f"\n\nQuestion: {question}")
    resp = ollama.chat(model="llama3.1:8b",
                       messages=[{"role": "user", "content": prompt}])
    return resp["message"]["content"]

print(ask("What have I written about caching?"))

You can also query the graph directly in the Neo4j Browser at http://localhost:7474. For example, MATCH (p:Entity {type:'Person'})-[:RELATED]-(x) RETURN p, x shows people and everything connected to them, which is often more revealing than any single answer.

Improving results

Normalise entity names. “Postgres”, “PostgreSQL” and “postgres db” become three nodes unless you merge them. A small alias dictionary, or lower-casing names before MERGE, helps a lot.
Add keyword search. Embeddings are weak at exact matches like error codes or IDs. A simple full-text search, such as SQLite FTS5 or Neo4j’s full-text index, alongside FAISS covers that gap.
Keep chunk sizes moderate. Very large chunks blur several topics into one embedding. Very small ones lose context.
Re-index incrementally. Store a hash of each note and only re-embed and re-extract notes that changed.

Limits to keep in mind

This setup is private and costs nothing beyond electricity, but it’s not magic. Small local models make extraction errors, answers are only as good as the retrieved chunks, and the model can still misstate what a note says. Treat answers as pointers to your notes, and click through to the sources, rather than as final truth. For many people that’s exactly what they need: a way to find what they already wrote.