Daniel Arcé

What this is: the working method behind Visual Search Workbench, a native macOS app that searches photo folders by text or by picture, named Semantic Image Search through its 0.2.0 build. That page shows what shipped. This one shows how it was designed and reviewed. What this is not: a general methodology claim. One product has been through this process end to end; the method is reported, not proven.

The map comes first

Before any interface code, the whole product was written down as a UX map: a plain-text inventory of screens, the zones inside each screen, the states each zone can be in, and the flows that connect them. The map is a version-controlled file whose review renderings come out as ASCII drawings, so it can be diffed and cited like any other source file.

Text-first sounds austere, but it forces a useful discipline. A drawing can leave a state vague. A map cannot: every zone must list what it shows when the data is missing, loading, partial, or wrong, and an unlisted state is visible as a gap in the file. The final delivery map for this product names flows with titles like partial census disclosure and what is not done to you, because those journeys had to exist as designed things before they could be built.

Each screen in the map carries one question. The cold-start screen asks how a person knows their photos will not be copied anywhere. The indexing screen asks what a half-finished index claims to be. Holding one question per screen kept reviews from sliding into general impressions.

The map is the single source of truth. It can be projected onto a spatial canvas for review, but the projection is one-way: nothing drawn on the canvas flows back. Two writable copies of the same design would drift apart silently, which is the same failure a database book warns about when one change is written to two stores.

Rules with names

The second instrument is the Heuristics Canon, a corpus of decision rules distilled from published sources. Each rule carries a stable ID and a link to the work it rests on, so a design critique can say exactly which rule it leans on and where that rule came from.

Three of the distilled sources did the most work here, and none of them is a design book. Designing Data-Intensive Applications is about how data systems keep derived copies honest. Release It! is about how software survives the failure of things it depends on. Latency is about why delay behaves the way it does. Distilling them into rules made their lessons portable: a principle written for a server farm can be held against a search box, because the rule states its trigger and the trigger either fires or it does not.

The review loop was mechanical. Update the map, run the critique pack against it, and resolve what fires before any interface work starts. Findings cite rule IDs, so a disagreement becomes an argument over whether a named rule applies to this screen.

What the rules said about search

Searching pictures by meaning rests on embeddings. An embedding model turns a picture, or a phrase, into a point in an abstract space, placed so that similar things land near each other. Search is then just nearness: embed the query, rank the library by distance. The canon's machine-learning rules take that simple picture and put boundaries on it, and each boundary became an interface decision.

Points are only comparable inside one space. Vectors from different models, or the same model with different preprocessing, cannot be ranked against each other. The product pins one embedding contract covering the model and everything done to a picture or phrase on its way to becoming a vector, and every gallery and query vector lives under that contract. During development the contract ID stayed literally unset until the model's license and conversion checks passed: the index refused to exist before its provenance did.

Nearest-k always answers. Ranking by nearness returns the k nearest photos even when all of them are far. Research on the shipped model found no trustworthy distance threshold to draw a line with, so the product does not pretend to have one. Results are captioned nearest, never matches; raw similarity scores are never shown as confidence; and a ranking that comes back nearly flat is treated as the designed unknown state, shown as "no close matches" rather than a weakly ordered grid. The canon calls this abstention: unknown is a valid result, and the below-threshold action is designed rather than left to chance.

The index is derived data. The embedding index sits beside the photo folders as disposable metadata: delete it and nothing about the files changes. That is the database book's discipline applied to pictures. A derived copy is rebuilt from its source, never treated as an authority, and when the embedding contract changes, the index is rebuilt rather than patched, because vectors from the new contract cannot be compared with the old ones.

Measure the plain path before approximating. Fast approximate indexes admit misses by design, and the canon requires that cost to be measured before it is paid. On this library scale, a measured exhaustive scan over a memory-mapped matrix met the budget on the target machine, so no approximation shipped at all.

Engineering rules that reached the screen

The most direct transfer came from rules that were never about interfaces.

Release It! holds that an undesigned state is a bug in itself. Applied to a search product, that rule turns every blank into a question: what does the app claim when no folder is chosen, when the model file is unreadable, when a disk is unplugged mid-index? Each condition got its own named state and its own wording, which the product page walks through screen by screen. The same book's insistence that a system assumes its dependencies will fail shaped what those states say: the missing model is named by path, the unplugged disk by name, so recovery is an action rather than a guess.

The no-dual-writes rule reached all the way into accessibility. The caveat a sighted person reads and the caveat VoiceOver speaks come from one string, because two copies of the same sentence would drift exactly the way two writable databases do.

The latency book contributed a habit of budgeting the slow path rather than the average one. On-device search has no network to blame, so the budget was set against the worst case that matters, a cold start over the full library on the smallest supported machine, and measured there.

The critique ran against the code, too

The map was written before the interface existed, which means the built interface could still betray it. So after the screens were running, the ux-map workflow from WorkBay, the task harness that carried the build, rendered the map again as ASCII screens, this time anchored line by line to the running code, and the critique pack was held against that render. One late pass caught three defects that pixel-level review had walked past. This is the screen the render exposed, drawn from the map artifact in the product repository:

┌─ Search ──────────────────────────────────────────────────────────────┐
│ [ query…                              ] (or drop an image)            │
│                                                                       │
│   No library yet — open a folder to search.                           │
│ ┌───────────────────────────────────────────────────────────────────┐ │
│ │ Preparing search…                                      [Cancel]   │ │
│ │ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  (indeterminate)                   │ │
│ └───────────────────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────────────────┘
    two competing "nothing here yet" messages, rendered at the same time
    this [Cancel] cancels nothing

Both messages describe the same fresh launch, and each came from its own independent condition, so nothing prevented both from being true together. The Cancel is worse than redundant: it could not stop the model load it appeared to govern, and its handler was borrowed from the indexing banner, so pressing it wrote a cancelled entry into the job history for an indexing run that had never started. The interface was fabricating a past. A third defect sat one state away: the layout meter for the similarity map could stack on top of this banner as a second, visually identical bar with its own Cancel, and only their spoken accessibility names told them apart, so a screen-reader user could tell which Cancel stopped which job while a sighted user could not.

The fix was structural rather than cosmetic. The status region became a single function that returns exactly one state, tested in a precedence order that puts what a person can act on ahead of what they can only wait for:

   indexing ───────►  live work first
   error ──────────►  then why it stopped
   empty library ──►  then what to do
   preparing ──────►  then just wait
   otherwise ──────►  ready

The rebuilt waiting state names its subject and offers no button it cannot honor:

┌────────────────────────────────────────────────────────────────┐
│ ▟▙▟▙▟▙▟▙▟▙▟▙  (indeterminate)                                  │
│ Preparing search   Loading the search model…       (no Cancel) │
└────────────────────────────────────────────────────────────────┘

A unit test now walks every combination of those inputs and asserts that exactly one state renders and only indexing offers Cancel. Each critique finding cites the file and lines it checked, states which claims were verified by reading the code, and flags what was asserted without measurement; all three were fixed before release. The lesson repeats the map's founding one: interface states written as independent conditionals will eventually all be true at once, and only an inventory of states catches that before a person does.

What the build remembered about itself

The build ran inside a task-memory harness, tooling that records the process the way the map records the design, in a database that travels with the repository. Decisions are stored with the rationale behind them, and review findings with how each was resolved and why. The commands that actually verified a change are stored too, so a later session can rerun proof instead of trusting a summary.

That record is blunt about how machine review behaves. Review findings outnumbered recorded decisions roughly two to one, and only about a third of them ended as integrated changes; more than half were superseded or withdrawn on the way. Keeping the written reason each finding died is what stops the next review from raising it again.

The work itself fanned out across short-lived isolated copies of the repository, with several different coding models doing the writing. None of their output merged on trust: every working copy ended in a report that a review had to accept. Even the blockers are part of the record. Most are budget exhaustions, a session stopped when it spent past its token allowance, and one remote reviewer that overran its time bound was excluded from the merge decision instead of being waited on. That is the engineering book's bounded-wait rule applied to the reviewers themselves.

The same discipline in the tools

The product was built with agent tooling, and one tool made the parallel unmissable. Codemap is a graph index over the app's own source code, mapping functions and their call relationships, with a co-occurrence signal that lets agents search code by meaning. It is, structurally, the same object as the photo index — a lossy, derived snapshot of a corpus, useful precisely because it is smaller than what it summarizes.

It fails the same ways, too. A truncated result list can read as a complete one. A malformed query can come back empty and read as "nothing exists." Its semantic signal is co-occurrence within one repository, not a trained model, so it returns weak nearest neighbours for terms it has never seen instead of saying no match. The tooling's guardrails answer each of those with the same move the product makes: only a surface that can actually check completeness may license a claim of absence, freshness is reported rather than assumed, and the underlying corpus stays the authority over its index.

The parallel runs deeper than codemap. The task memory embeds what it stores: each decision and finding is turned into a vector by a small local text-embedding model, and a fresh session recalls prior work by nearness to whatever it is working on now. The workbench ranks photographs by nearness in an embedding space; the memory that built the workbench ranks its own past the same way, under the same rules. One embedding space per comparison, and whatever retrieval surfaces is a candidate to verify against the source, never an authority.

That convergence is the finding these notes record. Rules written for databases and servers produced the product's caveat states, and the same rules governed how the build examined and remembered itself. A derived index, whatever it indexes, must say how much of its corpus it can see.