Skip to main content
This guide walks through reproducing EverOS’s LoCoMo retrieval scores locally using the hybrid and agentic search methods.

​
Prerequisites

  • Python 3.12, uv
  • A .env at the repo root with the LLM / embedding credentials EverOS needs:
    • EVEROS_LLM__MODEL, EVEROS_LLM__API_KEY, EVEROS_LLM__BASE_URL
    • EVEROS_EMBEDDING__*
    • EVEROS_RERANK__*
    • The benchmark driver also reads LLM_API_KEY / ANSWER_MODEL / JUDGE_MODEL for the answer + judge passes.
Install the project:

​
Steps

1

Prepare the dataset

Place the LoCoMo file at data/locomo10.json (the dataset is distributed by the LoCoMo authors, not this repo). Override the path with --data-path if you keep it elsewhere.
2

Start the server

EVEROS_ROOT isolates one benchmark’s corpus from another — change it (or rm -rf it) whenever you want a clean run.Leave the server running in one terminal and run the benchmark from another.
3

Run hybrid

Single conversation:
All 10 conversations, 2-way parallel:
The wrapper picks up EVEROS_ROOT from the environment so the cascade poll path matches the server’s data root. If you set them differently, pass --corpus-path explicitly.
4

Run agentic

Same wrapper, swap --methods:
You can benchmark multiple methods in one go — they share the same ingested corpus:

​
Results

Default output root is benchmark_results/run_<timestamp>/. Override with --output-root:
An aggregate accuracy table prints at the end of the wrapper run.

​
Notes

  • Re-running on the same corpus — add --skip-add to skip ingest and reuse what’s already in ~/.everos. Useful when comparing methods side by side.
  • Judge variance — --judge-runs 3 runs the judge three times per question and majority-votes; slower but reduces LLM-judge noise.