# Hexabot 3.4.0: rebuilding RAG around helpers and the database

Hexabot 3.4.0 is now available. It brings a more responsive admin experience, workflow and agent reliability fixes, improved graph layouts, and a substantial round of dependency updates.

At the center of this release is a deeper change: we rewrote Retrieval-Augmented Generation (RAG), removed LlamaIndex from the core, and made retrieval a helper-based capability.

This was not a cosmetic refactor. It changes how Hexabot thinks about retrieval, consistency, and customization.

## The problem was not one more missing hook

The previous RAG implementation maintained two bespoke indexes outside the canonical CMS content model: one for lexical retrieval and another for embeddings. Content lifecycle events were responsible for keeping those indexes synchronized with the database.

That approach looked reasonable at first. It also created a growing list of edge cases:

*   cascade deletion of a content type could bypass per-content deletion hooks;
    
*   a failed or silent `deleteRefDoc` operation could leave stale indexed data;
    
*   deactivated content could remain retrievable;
    
*   content written while RAG was disabled could create index drift;
    
*   reset and reindex operations could race with ongoing updates;
    
*   disabled or incomplete configuration could return empty results that looked like valid “no match” responses;
    
*   lexical and embedding modes reported configuration failures differently.
    

We added reconciliation and reindexing behavior over time, but those were repairs around the consistency gap. As long as correctness depended on event choreography across multiple indexes, every new lifecycle path was another opportunity for drift.

The conclusion was simple: fixing the individual symptoms again would preserve their shared cause.

## A smaller core and explicit retrieval choices

Hexabot 3.4.0 replaces the old RAG stack with a common helper contract. The active strategy is selected through `global_settings.default_rag_helper`.

| Helper | Retrieval | Database | Embedding provider | Installation |
| --- | --- | --- | --- | --- |
| `fulltext-search` | Lexical full-text search | SQLite or PostgreSQL | Not required | Built into `@hexabot-ai/api` |
| `sqlite-vector` | Semantic vector search with `sqlite-vec` | SQLite | Required | `hexabot-helper-sqlite-vector` |
| `pgvector` | Semantic vector search with `pgvector` | PostgreSQL | Required | `hexabot-helper-pgvector` |

### Full-text search is built in

`fulltext-search` is the default and needs no external embedding service.

On SQLite, it uses FTS5. On PostgreSQL, it uses native full-text search with a GIN expression index over the canonical content search text. The CMS database is the corpus, so there is no separate lexical document store waiting to drift out of sync.

This is the best starting point for installations that need keyword, phrase, identifier, or exact-term retrieval without the cost and operational requirements of embeddings.

### Vector search is installed as an extension

Semantic retrieval is provided by two database-specific packages:

*   [`hexabot-helper-sqlite-vector`](https://github.com/hexabot-ai/hexabot-helper-sqlite-vector) stores embeddings in SQLite and searches them through `sqlite-vec`.
    
*   [`hexabot-helper-pgvector`](https://github.com/hexabot-ai/hexabot-helper-pgvector) stores embeddings in PostgreSQL and searches them through the `vector` extension.
    

Both packages use the shared RAG helper contract exposed by `@hexabot-ai/api`, own their settings and storage, and are discovered automatically after installation. No manual module registration is required. Since they target different database engines, only the helper compatible with the active database is available at runtime.

Moving these implementations into extensions is intentional. There are many valid approaches to chunking, embedding, hybrid search, reranking, and index durability. An installable helper is a clearer customization boundary than growing one universal RAG service with an expanding set of modes and switches.

## Clearer behavior for workflows and agents

The previous `rag_settings.enabled` switch has been removed. Retrieval is always available through whichever helper is selected in `global_settings.default_rag_helper`.

This removes the ambiguous state where `retrieve_rag_content` could silently return:

```json
{
  "hits": [],
  "text": ""
}
```

even when retrieval was disabled or unavailable.

Helper configuration and availability failures now use typed errors. The `retrieve_rag_content` workflow action turns those failures into a consistent, visible response:

```json
{
  "hits": [],
  "text": "",
  "warning": "A description of the missing or unavailable configuration"
}
```

That shape is safer for both deterministic workflows and agent tool calls. A workflow author can distinguish “nothing matched” from “retrieval could not run” without having to infer which legacy mode or setting caused the result.

The old `mode: lexical | embedding` action setting is retained as a deprecated compatibility field, but helper selection now determines the retrieval strategy.

## Upgrade guide for 3.3.x projects

### 1\. Decide whether you need lexical or semantic retrieval

If lexical keyword search meets your needs, upgrade `@hexabot-ai/api` and keep:

```text
global_settings.default_rag_helper = fulltext-search
```

No embedding provider or vector helper is needed.

If you need semantic retrieval, add the package that matches your database. Installing both is also supported:

```json
{
  "dependencies": {
    "hexabot-helper-pgvector": "^3.4.2",
    "hexabot-helper-sqlite-vector": "^3.4.2"
  }
}
```

Install your project dependencies and restart the Hexabot API so extension discovery can load the helper.

### 2\. Configure embeddings in the admin UI

Create a credential for OpenAI or an OpenAI-compatible embedding endpoint. Then open the settings group for the compatible helper:

*   `sqlite-vector` for SQLite;
    
*   `pgvector` for PostgreSQL.
    

Configure the embedding provider, model, credential, optional base URL, dimensions, and chunking behavior. Finally, set `global_settings.default_rag_helper` to the helper name.

Selecting a vector helper triggers indexing of the content corpus. Until the helper and credential are configured, leave `fulltext-search` selected to keep retrieval available.

### 3\. Make `pgvector` available to PostgreSQL

The PostgreSQL helper requires the server-side `vector` extension. For Docker Compose deployments, update the PostgreSQL service in `docker-compose.postgres.yml`:

```yaml
services:
  postgres:
    image: pgvector/pgvector:pg16
```

The image makes the extension available to PostgreSQL. If you use a managed database or maintain PostgreSQL yourself, install or enable `pgvector` using the process supported by your provider.

### 4\. Understand the migration and rollback behavior

The core 3.4.0 migration provisions `fulltext-search` and defaults new selections to it.

On SQLite, the migration removes the obsolete LlamaIndex-era RAG tables, FTS mirror, and triggers. This cleanup is automatic and prevents old triggers from interfering with content writes. No manual SQLite cleanup is required.

On PostgreSQL, legacy LlamaIndex structures are intentionally preserved. This gives operators a rollback window. Verify the new retrieval helper and its index before removing those structures manually.

When the PostgreSQL vector helper is installed, its bootstrap logic can migrate legacy pre-3.4 embedding settings into the helper-owned settings group. The storage structures themselves remain untouched until you decide the rollback window is closed.

### 5\. New 3.4 projects are ready

The updated [Hexabot starter template](https://github.com/hexabot-ai/hexabot-template-starter) includes both semantic helper packages:

```json
{
  "hexabot-helper-pgvector": "^3.4.2",
  "hexabot-helper-sqlite-vector": "^3.4.2"
}
```

Its PostgreSQL Docker overlay already uses `pgvector/pgvector:pg16`. New projects created from the template therefore have the built-in full-text helper and the vector helper compatible with their active database without any manual extension wiring.

## A more usable admin interface

RAG is the architectural headline, but it is not the only visible change in 3.4.0.

Data grids and page headers across CRUD pages have been redesigned for clearer hierarchy and better responsive behavior. Filters collapse into an accessible popover when horizontal space is limited, and actions, pagination, timestamps, titles, and descriptions now fit together more consistently across entity lists.

The result is especially noticeable on settings, users, sources, workflow runs, and other data-heavy pages.

## Workflow, agent, CMS, and graph improvements

The release also includes a long list of focused improvements:

*   workflow runs have easier navigation, and execution state is restored after a refresh;
    
*   visual-editor definitions avoid tool-name collisions;
    
*   workflow actions and tool settings receive stronger schema validation;
    
*   graph layout is more reliable around start and end nodes, nested loops, attachments, sibling groups, placeholders, and terminal branches;
    
*   agent results correctly aggregate tool calls and tool results across all steps;
    
*   OpenAI-compatible providers load correctly and expose their base URL configuration;
    
*   required CMS content-type fields are enforced consistently;
    
*   content timestamps and several schema-form, drawer, autocomplete, and YAML-editor interactions have been corrected;
    
*   vulnerable transitive and direct dependencies received a broad security update.
    

## A foundation for more RAG approaches

The most important outcome of this rewrite is not that Hexabot now offers three retrieval choices. It is that retrieval has a stable extension boundary.

Teams can keep the zero-configuration database-native full-text helper, install the vector helper for their database, or build a helper around another approach without changing the core CMS and workflow architecture.

That is a better fit for the reality of RAG: different applications need different trade-offs, and those choices should be explicit.

Read the [Hexabot 3.4.0 release](https://github.com/hexabot-ai/Hexabot/releases/tag/v3.4.0), review the [RAG rewrite](https://github.com/hexabot-ai/Hexabot/pull/2160), or join the discussion in [issue #2141](https://github.com/hexabot-ai/Hexabot/issues/2141).