spiyweb

spiyweb v0.2.4spreading-activation retrieval for RAG

Not top-k.
The whole web.

spiyweb drops your query into a graph of passages as an energy seed and lets it spread. Close evidence lights up first; evidence that is only connected through other passages lights up a hop later. When the energy runs out, the web stops itself. No k anywhere.

pip install spiyweb
import spiyweb three steps, right below

Python 3.11+ on Windows, macOS and Linux; the core pulls in nothing. Type spiyweb in a terminal to watch your app's queries spread.

A garden spider at the centre of its orb web, every radial thread and spiral lit white against black. Wood engraving.
Jan Mankes, Kruisspin in zijn web, 1916. The web around it is live: click the field to drop a query in.

Start here

Three steps, then import spiyweb.

You do not need a retrieval library of your own. A folder of documents is enough.

  1. Install it

    The core is pure Python with no dependencies; the index-time extras bring the embedding model, spaCy and FAISS.

    terminal
    $ pip install "spiyweb[index]"
$ python -m spacy download xx_ent_wiki_sm

  2. Point it at your documents

    A folder of .txt or .md files is enough: no vector database, no schema, no code. It embeds, extracts entities and spins the graph. With a local LLM it also splits every passage into facts, and that is what lets the web beat plain top-k.

    terminal
    $ spiyweb index docs/ my-index

# sharper, with Ollama running: passages split into facts
$ spiyweb index docs/ my-index --propositions

import spiyweb and ask

One call returns the passages the web lit up, ranked by energy, with how many sources voted for each and how sure it is overall. Paste them into whatever model you already use: the retrieval is the library's job, the answer stays yours.

app.py
import spiyweb

with spiyweb.open_index("my-index") as index:
    answer = index.retrieve("who signed off on the release?")

# The web stopped itself; hand what it found to any LLM.
context = "\n\n".join(p.text for p in answer.passages)

for p in answer.passages:
    print(f"{p.energy:5.2f}  {p.votes} votes  {p.source_id}")
print(answer.confidence)  # how sure it is: energy, nodes, hop depth

No k to tune, no retriever to build. Every query it answers can also play live in the monitor just below.

Plate I

A query, as it spreads.

spiyweb — live monitor

File 01

Seed. Spread. Converge. Stop.

Top-k ends when it has k. The web ends when the energy does.

Where does the question land?

A round web with its spider at the hub and a fly caught near the rim. Wood engraving.

The only moment cosine similarity decides anything.

The query embedding goes into the graph as a seed of energy, 10.0 of it. It makes first contact with its nearest atoms by cosine, and the seed is split between them in proportion to how close each one is.

From there on, similarity steps aside. The energy travels along typed edges: shared entities are the main fuel, structure (same document, same section, the next chunk) and chunk-to-proposition links carry the rest. A layer can be switched off without touching the core.

edge layerssemanticentitystructuralderivationlearned

A garden spider letting itself down from a leafy twig on a single thread. Wood engraving.
Epeira diadema suspended by its thread, Popular Science Monthly, 1896. One thread down, and it has landed.
HOP 0
Q = 10.0 → A 5.625 · C 4.375split .9 : .7

How far does one thread carry?

An orb web framed by twigs and leaves, the spiral fine as hair. Wood engraving.

A weak thread fades faster than a strong one. That is the whole rule.

Each lit node keeps 40% of what it holds and forwards 60%. The decay is multiplicative, never a fixed subtraction, so a path of weak links dies quickly and a path of strong ones carries.

What it forwards is divided by edge weight: one strong neighbour takes most of it, five equal neighbours take about 12% each. When a neighbour turns out to be a near-copy, its edge is cut and its share goes to the others, and the idea it copied gets a vote instead of a second slot.

configdamping 0.60proportional splitdedup → vote

An armoured Minerva beside a huge spider web hung between trees, its spider at the centre. Woodcut.
After Hans Weiditz, Minerva beside a spider web, c. 1532. Every strand pulls on its neighbours.
HOP 1
A forwards 3.375 → A′ cut, idea A: 2 votesduplicate
B 2.250 · D 1.125 C forwards 2.625 → D 1.750 · E 0.875 ✕

What happens where two weak paths meet?

A perfect orb web drawn as geometry, its spider waiting at the edge. Wood engraving.

D was never the closest passage. It finishes third.

Energy that arrives by more than one path adds up. Neither A nor C alone would have lifted D far, but together they hand it 2.875, and a passage no similarity search ranked near the top ends up in the answer. That is converging evidence, for free.

A question with two parts goes in as two coloured seeds. Where the colours meet is a bridge: the passage a multi-hop answer has to cross, and the one plain top-k is least likely to return.

mechanismsadditive accumulationcoloured seedsbridge detection

A vaulted ceiling seen from below, a vast spider web spanning it from rib to rib. Woodcut.
After Hans Weiditz, Vault with a spider web, c. 1532. Every thread runs back to the hub.
HOP 2
D = 1.125 + 1.750 = 2.875 → F 1.725converging evidence

Who decides when it is enough?

A banded garden spider resting head-down at the centre of its web, legs paired along the zigzag silk. Line engraving.

Nobody asked for five results. The energy ran out.

The web stops on its own when the next hop would carry less than 15% of the energy injected. Relative, not absolute, so it behaves the same when a follow-up question starts on warm ground or a profile changes the seed.

There is no k parameter. max_nodes and max_hop exist only as overflow guards. The caller picks a profile, precise, explore or compare, and each carries its own damping, floor and seed width; a follow-up question keeps 20–30% of the last turn’s warmth.

profilespreciseexplore (default)compare

Two men in a landscape stop to study a spider web strung between the branches of a tree. Etching.
Jan Luyken, Two men study a spider web, c. 1700. Looking until there is nothing left to see.
HOP 3
F would forward 1.035 < 1.500the web stops

File 02

The canonical trace.

D is never the passage most like the question, and it ranks third. That gap is the whole idea.

The canonical trace: seed Q reaches A and C, then B, D and E, then F. D is reached by two paths and ranks third. 0.90.7✂0.80.40.60.30.5QACA′BDEF10.05.6254.3752.2501.1252.8750.8751.725idea A: 2 votes0.875 < 1.5, dies← A + C: converging1.035 < 1.5: stop

Seed. The question enters the graph as 10.0 units of energy. Nothing else is lit yet.

Hop 0. First contact, by cosine: the seed is split .9 : .7 between A and C. This is the only step similarity decides.

Hop 1. A and C forward 60% of what they hold. A′ is a near-copy of A: its edge is cut, its share goes to B and D, and idea A gets a second vote. E receives 0.875, under the 1.5 floor, and dies.

Hop 2. D is reached by two paths: 1.125 + 1.750 = 2.875. Converging evidence. D forwards 1.725 to F.

Hop 3. F would forward 1.035, under the floor, so the web stops itself. Nobody set a result count.

nodeenergy
A·5.625
C·4.375
D·1.1252.875
B·2.250
F·1.725
E·0.875 ✕
A′·cut → vote for A
  1. A5.6252 votes
  2. C4.375
  3. D2.875never the most similar
  4. B2.250
  5. F1.725

File 03

What top-k cannot see.

Keep plain top-k for lookups. This is for answers spread across passages.

Plain top-kthe k nearest chunksIterative retrievalan LLM rewrites, searches againspiyweba seed of energy, spreading
Starts fromthe chunks most similar to the questionthe same, then a rewritten question, then anotherits nearest atoms, then the threads between passages
Evidence that is only connectednever seen, unless it is also similarfound if a rewrite happens to name itreached through shared entities, a hop or two out
Ten near-copies of one passageten slots, one factten slots, one factone slot, and ten votes for it
When it stopsat k, whatever is leftafter a fixed number of roundswhen the energy runs out
Why this passage?a similarity scorethe rewritten query that found itthe activation path that carried energy to it
Saying “I don’t know”no signal: k results, alwaysno signala confidence score, gap warnings and a refusal report
LLM calls per questionnoneabout 4about 2.3, none of them inside the core

File 04

It can say “I don’t know.”

A retriever that cannot say how sure it is leaves the guessing to the model.

  • Votes per idea

    answer.votes()

    How many separate sources repeated each idea before their copies were cut. Corpus support, counted by document.

  • Activation paths

    answer.paths()

    How the energy reached every passage, hop by hop, and how many paths converged on it. Hand them to the model as explanations, not debug output.

  • Theme clusters

    answer.clusters()

    What lit up, grouped: these are separate themes, and this is where they intersect.

  • Confidence

    answer.confidence

    Total energy held, nodes lit, hop depth reached. The library reports; you decide what counts as “I don’t know”.

  • Corpus-gap warnings

    answer.gaps()

    Two dense clusters with no bridge between them: the corpus is missing the passage that would connect them.

  • Refusal report

    answer.refusal()

    When confidence is low, a template-built account of why: which clusters lit, where the bridge is missing, where the energy died. No model writes it.

  • Contradiction records

    answer.conflicts

    Pairs of sources that oppose each other, both kept, with build_conflict_question() turning each into a question for the user.

  • Minerva, helmeted, points at Arachne beside her loom as the weaver is turned into a spider. Etching.
    Antonio Tempesta, Minerva turns Arachne into a spider, 1606. The weaver who would not hide her work.

File 05

Six things the web will never do.

Settled before the first benchmark. A better score does not reopen them.

  1. Never puts a language model in the core.

    core/ takes arrays and a config and returns numbers: no model, no vector store, no network, no filesystem. LLM calls happen at index time, for propositions and ambiguous entities, or in your own code.

    Arrays in, numbers out. Every later phase leans on that.

  2. Never asks you for k.

    The web stops on a relative energy threshold. A k= argument would quietly bring back the cut-off the method argues against, so there is none; slice answer.passages if you want fewer.

    Take five if you like. It will not pretend five is a property of the corpus.

  3. Never picks a winner in silence.

    A contradiction is negative charge: opposing atoms damp each other instead of adding up. When one survives into the result, both sides enter the context, flagged as disputed, with a ready-made question for the user.

    build_conflict_question(answer.conflicts[0]).text

    Two sources make opposing claims:
      (A) changelog: v2 shipped 12 March
      (B) minutes: release moved to 19 March
    Which should the answer rely on?
    1 Rely on (A) · 2 Rely on (B) · 3 Keep both, marked as disputed

    no answer back? option 3 is what happens

    Disputed is an answer too.

  4. Never throws a duplicate away.

    A near-copy found while the energy spreads loses its edge, and the idea it repeats gains a vote, counted per source, never per chunk. Plain MMR would drop it and lose the signal.

    answer.passages, three sources saying one thing

    5.62  3 votes  v2 rollout approved Tuesday
    4.38  1 vote   Tuesday review: decisions

    two copies cut, two votes kept, one slot used

    Repetition is support, not content. Not truth either: a vote counts sources.

  5. Never lets what it learned rewrite the graph.

    Usage reinforcement lives in a separate layer you can switch off. The base graph is never mutated; edges nobody ever used are pruned offline, and merging nodes waits until it can be undone.

    Switch the learned layer off and you get back the graph you built.

  6. Never installs what you did not ask for.

    pip install spiyweb pulls in nothing else, and import spiyweb loads no numpy, torch or FAISS. Index-time tools come as extras, and CI checks the claim on the built wheel, not on the checkout.

    The wheel job measures the artifact, not the promise.

A decorative title band: a cross spider at the hub of a half-wheel web, threads radiating across a dark field. Woodcut.
Gerrit Willem Dijsselhof, Title head with a cross spider and web, 1893. Every thread accounted for.

File 06

Measured on multi-hop.

Against plain top-k and against iterative retrieval, the stronger baseline. 1000 questions a run.

  1. .5094S@5 on MuSiQue

    Ahead of both baselines

    MuSiQue, tuning set (seed 42), 1000 questionsiterative .4631 · top-k .3090 · +.046, CI [+.030, +.062]

  2. .5073S@5, fresh questions

    It holds on a second draw

    MuSiQue, confirmation set (seed 123), configuration frozeniterative .4420 · top-k .3046 · +.065, CI [+.048, +.082]

  3. .7130S@5 on 2WikiMultihopQA

    It transfers unchanged

    2WikiMultihopQA, 1000 questions, the MuSiQue configuration as it wasiterative .687 · top-k .468 · +.026, CI [+.016, +.037]

  4. ~2.3LLM calls per question

    Cheaper than the baseline it beats

    at query time, every run aboveiterative retrieval needs about 4

A sector of an orb web strung with dew, every thread a line of bright beads against black. Engraving.
Sector of a dew-laden orb web, Popular Science Monthly, 1890. Every bead where it was measured.

how it is scored

S@5 = 0.65 · support recall + 0.35 · Novelty@5

  • Novelty@5 counts relevant passages the web returns and plain top-k never does.
  • Paired bootstrap intervals over the same 1000 questions for every method.
  • The winning configuration is applied unchanged to every dataset after the first.

Every number on this band is in RESULTS.md, with its interval.

File 07

Where the web thins.

A retriever that explains itself also has to say where its threads give out.

  1. Spreading activation is not new.

    HippoRAG, GraphRAG, LightRAG and RAPTOR already walk graphs. The claim here is narrower: duplicates turned into votes, coloured bridges, and the honesty outputs. Prior art for the vote mechanism is welcome.

  2. Busy nodes get a thin share.

    A proportional split punishes a passage connected to everything: its energy is ground into many small pieces. Softening it, weighting by sim ** alpha, is an open question.

  3. Repetition is not truth.

    A vote measures corpus support. Counting by source rather than by chunk limits the damage of one document copied ten times, but a corpus that is wrong everywhere still votes for the wrong thing.

  4. Indexing costs model calls.

    Proposition extraction and the ambiguous half of entity extraction use an LLM at index time. On a fixed benchmark that is fine; on your corpus, measure it before you point it at a large one.

  5. Contradiction detection is early.

    The mechanism for disputed sources is in place and tested; the index-time detector that finds them misses most real pairs today. Treat no dispute flagged as “none found”, not “none exist”.

  6. The learned layer can drift.

    Reinforcement without forgetting pulls the graph toward whatever was asked most. That is why it lives in its own layer, off by default, and why consolidation prunes rather than merges.

File 08

The same web, behind a pipe.

No screen needed: the monitor's verbs work in scripts, and the library under them is three lines.

pip install "spiyweb[index]"# the index-time extras, once
python -m spacy download xx_ent_wiki_sm# the entity model
spiyweb index docs/ my-index# a directory of .txt/.md → an index
spiyweb index docs/ my-index --propositions# sharper: a local LLM splits passages into facts
spiyweb query my-index "what happened afterwards"# what lit up, as bars
spiyweb query my-index "…" --json# the same, machine-readable
spiyweb lint my-index# what is wrong with the corpus
spiyweb version# which extras are missing, and the line to fix it

bare spiyweb, depending on where it runs

a terminalthe live monitor takes the window over
a pipe or CIthe usage text and a non-zero exit, never a hung prompt
no braille, no box linesSPIYWEB_ASCII=1 draws the plain glyph set
Ctrl-Cexit 130, nothing half-written

lint is the diagnostic that needs no query: it reads the graph's shape and reports islands nothing bridges, hubs that grind arriving energy into dust, near-identical passages that will compete for one seed slot, and edge layers that came out empty, so a corpus problem is not mistaken for a retrieval one.

Every query an open index answers is recorded as a self-contained trace: the lit subgraph, the passages, the energy ledger and the settings it ran with. load_traces() reads it back on a machine that holds no index at all.

app.py
import spiyweb

with spiyweb.open_index("my-index") as index:
    answer = index.retrieve("who signed off on the release?")

    for passage in answer.passages:
        print(f"{passage.energy:5.2f}  {passage.votes} votes  {passage.text[:60]}")

    print(answer.confidence)   # total energy, node count, hop depth
    for path in answer.paths():  # how the energy reached each node
        print(path)
A cross spider seen from above, sitting in the middle of its web; the threads cut white out of black. Woodcut.
Julie de Graag, Spider in a web, 1918. The same web, cut in wood.