> ## Documentation Index
> Fetch the complete documentation index at: https://docs.evermind.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# LoCoMo Benchmark

> Reproducing EverOS's LoCoMo retrieval scores locally using hybrid and agentic search methods.

This guide walks through reproducing EverOS's LoCoMo retrieval scores locally using the `hybrid` and `agentic` search methods.

## Prerequisites

* Python **3.12**, [uv](https://docs.astral.sh/uv/)
* A `.env` at the repo root with the LLM / embedding credentials EverOS needs:
  * `EVEROS_LLM__MODEL`, `EVEROS_LLM__API_KEY`, `EVEROS_LLM__BASE_URL`
  * `EVEROS_EMBEDDING__*`
  * `EVEROS_RERANK__*`
  * The benchmark driver also reads `LLM_API_KEY` / `ANSWER_MODEL` / `JUDGE_MODEL` for the answer + judge passes.

Install the project:

```bash theme={null}
uv sync
```

## Steps

<Steps>
  <Step title="Prepare the dataset">
    Place the LoCoMo file at `data/locomo10.json` (the dataset is distributed by the LoCoMo authors, not this repo). Override the path with `--data-path` if you keep it elsewhere.
  </Step>

  <Step title="Start the server">
    ```bash theme={null}
    EVEROS_ROOT=~/.everos \
    uv run python -m everos.entrypoints.cli.main server start --port 8000
    ```

    `EVEROS_ROOT` isolates one benchmark's corpus from another — change it (or `rm -rf` it) whenever you want a clean run.

    Leave the server running in one terminal and run the benchmark from another.
  </Step>

  <Step title="Run hybrid">
    Single conversation:

    ```bash theme={null}
    bash tests/run_locomo_batch.sh \
      --conv-indices 0 \
      --methods hybrid \
      --base-url http://localhost:8000 \
      --top-k 10
    ```

    All 10 conversations, 2-way parallel:

    ```bash theme={null}
    bash tests/run_locomo_batch.sh \
      --conv-indices 0-9 \
      --methods hybrid \
      --base-url http://localhost:8000 \
      --top-k 10 \
      --concurrency 2
    ```

    The wrapper picks up `EVEROS_ROOT` from the environment so the cascade poll path matches the server's data root. If you set them differently, pass `--corpus-path` explicitly.
  </Step>

  <Step title="Run agentic">
    Same wrapper, swap `--methods`:

    ```bash theme={null}
    bash tests/run_locomo_batch.sh \
      --conv-indices 0-9 \
      --methods agentic \
      --base-url http://localhost:8000 \
      --top-k 10 \
      --concurrency 2
    ```

    You can benchmark multiple methods in one go — they share the same ingested corpus:

    ```bash theme={null}
    bash tests/run_locomo_batch.sh \
      --conv-indices 0-9 \
      --methods hybrid,agentic \
      --base-url http://localhost:8000 \
      --top-k 10 \
      --concurrency 2
    ```
  </Step>
</Steps>

## Results

Default output root is `benchmark_results/run_<timestamp>/`. Override with `--output-root`:

```
<output_root>/
├── conv0.json … conv9.json          # per-conv summary + per-question details
├── conv0.log  … conv9.log           # per-conv stdout (only in --concurrency >1 mode)
└── conv0_checkpoints/ …             # incremental search/answer/eval JSON
```

An aggregate accuracy table prints at the end of the wrapper run.

## Notes

* **Re-running on the same corpus** — add `--skip-add` to skip ingest and reuse what's already in `~/.everos`. Useful when comparing methods side by side.
* **Judge variance** — `--judge-runs 3` runs the judge three times per question and majority-votes; slower but reduces LLM-judge noise.