> ## Documentation Index
> Fetch the complete documentation index at: https://docs.evermind.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Multimodal Memory

> Extract memories from images, documents, audio, and video alongside text

EverOS goes beyond text. You can attach images, PDFs, audio recordings, and other file types to your messages, and EverOS will extract relevant information from them to enrich the memory.

## Supported Content Types

<CardGroup cols={3}>
  <Card title="Images" icon="image">
    JPG, PNG, GIF, WebP

    Max size: **10 MB**
  </Card>

  <Card title="Documents" icon="file-lines">
    PDF, DOC, HTML, TXT

    Max size: **100 MB**
  </Card>

  <Card title="Audio & Video" icon="film">
    MP3, WAV, MP4, WebM

    Max size: **500 MB**
  </Card>
</CardGroup>

## How It Works

When you include multimodal content in a message, EverOS processes the file to extract textual information and other relevant details, then stores them as part of the user's memory — just like text messages.

The key difference from text-only messages is that the `content` field becomes an **array of content items** instead of a plain string.

### Content Item Structure

Each item in the content array has a `type` field that determines how EverOS processes it:

| Type    | Description         | Required Fields |
| ------- | ------------------- | --------------- |
| `text`  | Plain text content  | `text`          |
| `image` | Image file          | `uri`           |
| `audio` | Audio recording     | `uri`           |
| `doc`   | Document (TXT, DOC) | `uri`           |
| `pdf`   | PDF document        | `uri`           |
| `html`  | HTML content        | `uri`           |
| `email` | Email message       | `uri`           |

<Note>
  The `uri` field contains an object key returned by the upload flow (see below), not a public URL.
</Note>

## Upload Flow (API)

Uploading multimodal data via the API is a **3-step process**:

<Steps>
  <Step title="Get a pre-signed upload URL">
    Call `POST /api/v1/object/sign` to get a pre-signed S3 upload URL and an `objectKey` for your file.

    ```bash theme={null}
    curl -X POST https://api.evermind.ai/api/v1/object/sign \
      -H "Authorization: Bearer <api_key>" \
      -H "Content-Type: application/json" \
      -d '{
        "objectList": [
          {
            "fileId": "my_image_001",
            "fileName": "meeting-whiteboard.png",
            "fileType": "image"
          }
        ]
      }'
    ```

    Response (key fields):

    ```json theme={null}
    {
      "result": {
        "data": {
          "objectList": [
            {
              "fileId": "my_image_001",
              "objectKey": "982a8db8c67be277/b3303282-2c49-4564-bb1a-6e415b781d0c",
              "objectSignedInfo": {
                "url": "https://s3.amazonaws.com/bucket-name",
                "fields": {
                  "key": "982a8db8c67be277/b3303282-...",
                  "policy": "eyJjb25k...",
                  "x-amz-algorithm": "AWS4-HMAC-SHA256",
                  "x-amz-credential": "...",
                  "x-amz-signature": "..."
                }
              }
            }
          ]
        }
      }
    }
    ```

    Save the `objectKey` — you will need it in Step 3.
  </Step>

  <Step title="Upload the file to S3">
    Use the returned `objectSignedInfo.url` and `fields` to upload your file via an S3 POST request.

    ```bash theme={null}
    curl -X POST "https://s3.amazonaws.com/bucket-name" \
      -F "key=982a8db8c67be277/b3303282-2c49-4564-bb1a-6e415b781d0c" \
      -F "policy=eyJjb25k..." \
      -F "x-amz-algorithm=AWS4-HMAC-SHA256" \
      -F "x-amz-credential=..." \
      -F "x-amz-signature=..." \
      -F "file=@meeting-whiteboard.png"
    ```

    A successful upload returns HTTP **204 No Content**.
  </Step>

  <Step title="Add memory with the objectKey">
    Now call any of the add memories endpoints with `content` as an array. Set the `uri` field to the `objectKey` from Step 1.

    ```bash theme={null}
    curl -X POST https://api.evermind.ai/api/v1/memories \
      -H "Authorization: Bearer <api_key>" \
      -H "Content-Type: application/json" \
      -d '{
        "user_id": "user_alice",
        "messages": [
          {
            "role": "user",
            "timestamp": 1711900000000,
            "content": [
              {
                "type": "text",
                "text": "Here is the whiteboard from today meeting"
              },
              {
                "type": "image",
                "uri": "982a8db8c67be277/b3303282-2c49-4564-bb1a-6e415b781d0c",
                "name": "meeting-whiteboard.png",
                "ext": "png"
              }
            ]
          }
        ]
      }'
    ```

    EverOS will process the image, extract relevant information, and store it as part of the user's memory.
  </Step>
</Steps>

### Batch Upload

You can sign multiple files in a single request by passing multiple items in the `objectList` array:

```bash theme={null}
curl -X POST https://api.evermind.ai/api/v1/object/sign \
  -H "Authorization: Bearer <api_key>" \
  -H "Content-Type: application/json" \
  -d '{
    "objectList": [
      {"fileId": "img_001", "fileName": "photo.jpg", "fileType": "image"},
      {"fileId": "doc_001", "fileName": "report.pdf", "fileType": "file"},
      {"fileId": "vid_001", "fileName": "recording.mp4", "fileType": "video"}
    ]
  }'
```

Then upload each file to its respective pre-signed URL and include all `objectKey` values in your message content array.

## Upload with Python SDK

<Tip>
  The Python SDK simplifies multimodal uploads to a **single step**. It automatically handles file signing, uploading, and memory creation behind the scenes.
</Tip>

```python theme={null}
import time
from everos_cloud import EverOS

client = EverOS()
memories = client.v1.memories

now_ms = int(time.time() * 1000)

response = memories.add(
    user_id="user_demo_001",
    session_id="session_gs_003",
    messages=[
        {
            "role": "user",
            "timestamp": now_ms,
            "content": [
                {"type": "text", "text": "Here is a photo from today's meeting."},
                {
                    "type": "image",
                    "uri": "./whiteboard.jpg",           # local path — auto-uploaded
                    "name": "whiteboard.jpg",
                    "ext": "jpg",
                    "text": "meeting whiteboard",
                },
            ],
        },
        {
            "role": "user",
            "timestamp": now_ms + 60_000,               # 1 minute later
            "content": [
                {"type": "text", "text": "Reference image from the web."},
                {
                    "type": "image",
                    "uri": "https://example.com/restaurant.jpg",  # HTTP URL — auto-downloaded + uploaded
                },
            ],
        },
    ],
)
print(f"status={response.data.status}  task_id={response.data.task_id}")
```

## Mixing Text and Files

A single message can contain both text and multiple file attachments. EverOS processes all content items together to build a complete memory:

```json theme={null}
{
  "role": "user",
  "timestamp": 1711900000000,
  "content": [
    {"type": "text", "text": "Summary of the Q3 planning session"},
    {"type": "image", "uri": "<objectKey_1>", "name": "whiteboard.png", "ext": "png"},
    {"type": "pdf", "uri": "<objectKey_2>", "name": "q3-plan.pdf", "ext": "pdf"},
    {"type": "audio", "uri": "<objectKey_3>", "name": "recording.mp3", "ext": "mp3"}
  ]
}
```

<Note>
  **Text items have no limit** per message. Non-text items (image, audio, doc, etc.) are limited to **10 per message**. The total request body must be under **300 KB** (file content is stored on S3, not in the request body).
</Note>

## Works with All Memory Endpoints

The multimodal content format works with all three add memories endpoints:

| Endpoint                      | Use Case                                                      |
| ----------------------------- | ------------------------------------------------------------- |
| `POST /api/v1/memories`       | Personal memories with attachments                            |
| `POST /api/v1/memories/group` | Group conversations with shared files                         |
| `POST /api/v1/memories/agent` | Agent trajectories with tool outputs (screenshots, documents) |

## Content Item Reference

| Field         | Type           | Description                                                                   |
| ------------- | -------------- | ----------------------------------------------------------------------------- |
| `type`        | string         | **Required.** One of: `text`, `image`, `audio`, `doc`, `pdf`, `html`, `email` |
| `text`        | string \| null | Content body. For `type: text`, this is the actual text                       |
| `uri`         | string \| null | Object key from the upload flow (returned by `POST /api/v1/object/sign`)      |
| `name`        | string \| null | File name                                                                     |
| `ext`         | string \| null | File extension (e.g., `png`, `mp3`, `pdf`)                                    |
| `source`      | string \| null | Content source (e.g., `google_doc`, `notion`, `confluence`, `zoom`)           |
| `source_info` | object \| null | Source-related traceability metadata                                          |
| `extras`      | object \| null | Type-specific extra fields                                                    |

## Next Steps

<CardGroup cols={2}>
  <Card title="Upload API Reference" icon="cloud-arrow-up" href="/api-reference/storage/upload-multimodal-data">
    See the full API specification for file signing
  </Card>

  <Card title="Add Memories API" icon="plus" href="/api-reference/memories/add-personal-memories">
    See the add memories endpoint with content array format
  </Card>
</CardGroup>