Visual Search Workbench
How do you make local AI search useful when the system cannot assume the library is complete?
Missing, partial, and unavailable data each get a named state. An incomplete view of the library never reads as a complete one.
A workbench for pictures that already live in folders, and stay there.
Final product
About the screens. The pixel screenshots come from the 0.2.0 build (27 August 2026). The ASCII screens beside them are drawn from the shipped code. A test in the product repository pins each quoted string: it must occur verbatim in the named Swift file. The drawings are the checkable contract. The screenshots show the same surfaces as pixels. The window title in that build still reads Semantic Image Search, the product's working name; the rename to Visual Search Workbench ships in the next build, and the screens here stay faithful to the build they came from.
At a glance
- Platform: Native macOS. The models run on the Mac, within 8 GB of memory.
- Library access: Reads only the folders the person has chosen. Gallery images stay on the device.
- Query: Text, or an image dropped from anywhere. Any photo can pivot to its visual neighbours. Searching by picture is the capability macOS Photos does not offer.
- Surfaces: Search, plus Similarity, Places, and Timeline for relationships a query did not express.
Problem
Photo libraries hold more meaning than filenames and folders expose. Cloud search creates a privacy boundary, and local models have their own limits. A useful local product must find meaningful relationships, and it must tell the truth about how much of the library it can see.
Core journey
Choose a folder, index it, and search by text or by image. Inspect a result. Then explore Similarity, Places, or Timeline for relationships the query did not express.
Cold start and indexing
Screen 1 — Cold start. Before a folder is chosen, Search says "Choose a folder to search. Your photos stay where they are." The Layers panel is empty, and no zero-result count is drawn.
The start state is kept apart from an empty result. A count of zero before any library exists would imply a search that found nothing. The first line also states what the app does with the files: it reads them where they sit, and it does not import or copy them.
Screen 2 — Indexing. A determinate meter reports "Embedding 812 of 1284…" above a two-row photo grid.
The meter shows how far indexing has got and how far it has to go. Cancelling leaves a partial index, and the interface labels it partial.
Ranked results
Screen 3 — Semantic results. This screen shows a semantic query for a red bicycle and a grid of 64 nearest results.
Results are ordered by nearness to the query. Position carries the ranking. Raw cosine scores are not shown as confidence. When nothing is near, the grid shows a "No close matches" state instead of a weakly ranked list.
Search with an image
A query does not have to be words. "Search with image…" embeds a picture with the same encoder that indexed the library. A file dropped on the window does the same. Every photo is then ranked by nearness to it. Find similar starts the same search from any selected photo. Both query kinds land in the one embedding space the index was built in, so their rankings mean the same thing.
The mode announces itself. The banner states "Results are photos visually similar to 56740009.JPG — not a text search" and offers a way back. An image-ranked grid is never mistaken for an answer to the text in the search field.
Inspection
Selecting a photo opens the facts behind the result. A machine ranking can be checked against what the file records.
Screen 4 — A photo selected, details open. IMG_4417.HEIC is selected; double-click opens it. The details panel shows size (4032 × 3024), camera (Apple iPhone 13 Pro), lens (26 mm f/1.5), capture time (2021:07:14 18:22:00), and GPS (64.1466°, -21.9426°) above an offline mini-map.
Each label sits with its value. The panel keeps its open or closed state as the selection changes. The mini-map draws an offline graticule and never fetches tiles.
Layers
Segmentation gives each photo a strip of cutout layers, named by what the detector saw: cat, plant, left plant, person, structure. Every name carries the experimental marker. A machine-drawn matte does not get the same authority as the file facts above it. A click opens a layer full-size on an alpha checkerboard with one action, Save this layer…. Save layers… exports the set for composition in other tools.
The app groups people without identifying them: chips read Person 1, Person 2, Person 3; a file with none says "No person names in this file"; and no chip resolves to a name.
Designing incomplete truth
"No result" is not the same state as "I cannot currently see the whole library." The product keeps those conditions separate because collapsing them would turn missing evidence into a product claim. Every condition below is worded for its cause and offers a way out; the rules behind that split, and the process that applied them, are the subject of A Map and a Canon.
No folder chosen is the start state, held apart from an empty result for the reason Screen 1 gives.
No matches means the query ran across the chosen folders and found nothing near. The interface still shows what was searched, so the person can widen the folders or change the query.
Model unavailable means the local model is absent or unreadable. Search names the missing package. A bare "no results" would blame the photos for a missing file.
Partial library covers an index interrupted before it finished and metadata that exists for only part of the collection. The affected views are scoped to what was read and say why the rest is missing.
Unavailable volume or permission covers an offline disk and a folder the app was denied. Either is reported as unreachable, which is different from searched and irrelevant.
Experimental marks machine-derived behaviour that is still preliminary, so it does not carry the same authority as the rest of the app.
Screen 6 — Partial-library caveat. A warning says 312 photos were never opened because their files are on an unplugged disk, which the caveat names, so dates and places describe the rest of the library.
The caveat qualifies every plot and names why the library is partial. The visible text and the VoiceOver text come from one string, so they cannot drift apart.
Screen 8 — Missing model and settings recovery. The model panel names /Volumes/Butter/cache/sis-coreml-export and says that detection still works when the folder is unreadable, but search does not. Settings lists ~/Pictures/2021-iceland, states that no photo is filtered or certified safe, and shows the off-by-default local Ollama option.
The model path and its readability are shown as separate facts. When search cannot run, the missing package is named. The one optional network hop goes to a local Ollama socket. It is off by default and named by address (127.0.0.1).
Similarity, Places, and Timeline
Similarity, Places, and Timeline share one selection. Moving between them keeps it: the frame changes while the anchor does not. Each view reports how much of the selection it can show.
Similarity
The Similarity badge starts at 47, matching the selected set. After the Places map selection narrows, it reads 38.
The view lays the whole indexed library out as a two-dimensional map, projected by UMAP from the same embeddings that answer queries. Photos near the current results appear as thumbnails; the rest stay points. Drag pans. Shift-drag selects a region, and the selection carries into the grid below.
The map is drawn for selection, and its geometry stays inside what a projection can say. A UMAP layout preserves neighbourhoods rather than distances (McInnes, Healy & Melville 2018). So the surface has no axes and no distance readout, and a gap between two clusters is not a fact about the library: claims from the layout stop at local contiguity.
Places
Screen 5 — Explore: Similarity / Places / Timeline. The first badges read Similarity 47, Places 41, and Timeline 47: only 41 of the selected photos have coordinates. The Places state then reads 38, 41, 41. It shows 41 of 1284 photos, with 1198 lacking coordinates and 45 unreadable. A 41-photo map selection outside the search results is listed last, in place order.
The badge counts differ on purpose: 47 photos are selected, and 41 of them have coordinates. Photos without coordinates, unreadable files, and never-opened files stay separate explanations. A map selection outside the search results is listed after the ranked results and labelled unranked.
Timeline
Screen 7 — Timeline: rest, zoom, truncation. The all-dates view spans 2019–2025 and holds 94 undated photos off the axis. A legend names each bar's date source: EXIF exposure date, file write time, or source unknown. The July 2021 zoom uses daily GMT periods. The truncated view says 3 photos dated before 1962 are off the left of the axis and offers a Before 1962 action.
The breadcrumb records the window that was asked for. The range line states what the chart draws. Undated photos sit beside the axis with their own count. When the axis is cut, the chart says how many photos lie beyond it and offers a button that goes there.
Next to Apple Photos
macOS Photos already searches by meaning: Apple's search guide covers natural-language queries, text inside images, dates, and detected people, pets, and objects. "Semantic search" alone would be a parity claim. What Photos does not offer is visual search: a picture as the query, and any result as the start of the next search. That is the differentiator, and the rest of the product is arranged around it.
- Image as query. Photos search starts from words and built-in categories. Here the query can be a picture from anywhere, and any photo pivots to its visual neighbours. The Similarity map extends the same idea to the whole library at once.
- Folders as the corpus. Photos can keep referenced originals outside its library at the cost of managed-library features, and separate libraries require switching. Here the library is the chosen folders searched together, and the index is disposable metadata beside them: deleting it changes nothing about the files.
- Inspectable retrieval. Ranking is cosine nearness in one pinned, openly licensed embedding space, recorded in the product's architecture decisions, so a wrong result can be reasoned about. Photos' pipeline cannot be examined from outside.
- Layers out. Photos lifts one subject as a sticker. Here segmentation writes typed, saveable alpha layers meant for composition elsewhere.
Together those make the product a visual-search workbench for folders. It is aimed at people whose pictures live outside a managed library: photographers' archives, scan sets, design assets, external volumes.
Under real use
Trust in the Explore surfaces comes down to how they behave in the middle of a task. Each behaviour below is held by a test.
- The frame you set up survives the move. Similarity, Places, and Timeline hold their zoom and pan across tab changes. Looking at the same photos another way costs one click and no reorientation. A regression check covers tab changes.
- A partial view says so. When files cannot be read, the plot names which disk is unreachable and how many photos that leaves out. VoiceOver speaks the caveat when it appears. Without it, a partial chart looks exactly like a complete one.
- A selection cannot outlive what it pointed at. Changing the timeline axis clears the selection. A count on screen always refers to a range the chart draws now. A mutation check covers it.
Keyboard operation and VoiceOver were designed in from the start. Every state above has spoken wording, and the partial-library caveat is announced when it appears.
Evaluation and evidence
Memory and runtime were measured on an 8 GB Apple Silicon machine. Network checks found no egress of gallery data in the tested flows. Model licences were reviewed. Automated tests cover the core behaviour, and adversarial review targeted the privacy claims and the failure states.
How the product was designed and reviewed, from the whole-product UX map to the rule corpus that vetted each screen, is its own article: A Map and a Canon.
The app works end to end on the machine it was designed for. It has not been measured on a library larger than the test corpus, and a task-based study with users outside the review has not yet run. That study is the next step: it will show whether people beyond the review can form useful queries and read the caveats as written.