Most Agent Memory Systems are Unfinished Databases

The small CRM app I built is to help a medium business manage the customer relationships of their half-dozen travelling sales people. The goal was more insight on how customer's are being engaged, in a way that works for the sales folk who are constantly in their cars on the road.

The initial scope was simple: capture events, tasks, and reminders and extract entities (people and businesses) to build the data into a mini CRM. The first iteration of this heavily relied upon agent memory - with tools available to extract entities and action items from unstructured messages into a lightweight CRM (clients, contacts, tasks) while also storing contextual notes as searchable memory for the agent to recall later. The structured tables handled the "what" — who's a client, who's a contact, what needs doing. Memory handled the "everything else" — the context around interactions, relationship history, facts that didn't fit a column.

As I dived deeper into the solution and started using it, it became clear that the memory was holding onto data that was not reliably queriable - yes I built solutions for semantic and fuzzy and keyword matched searching, but I had to fall back to searching the raw interaction logs to ask questions like 'when did we go see Joe at location Y' or 'how many times did I mention the new sales program last week'. The information was technically in there, but it was buried in the daily memory summary that then needed to look at the interaction logs to get the answer that still required an LLM to parse the raw text data to respond. Contrast this with a structured interactions table — account, person, channel, date — that's a single filtered, repeatable query.

What started as a CRM with memory filling the gaps gradually revealed that the gaps were just tables we hadn't designed yet.

There is still a place for agentic memory - but I think it is exactly for use cases that are either so broad they can never fit into structured data or they could be placed into structured data in the future when a schema becomes more apparent.

I don't have a solid answer for whether solutions like mempalace are viable long-term alternatives to structured data or they just kick the can further down the line until you reach a point of needing structured data.

Ultimately my current position is to use unstructured memories when the information is valuable but its future use is unknown at write time - you can't design a schema for data whose query patterns don't exist yet.