Finding a Visual Direction with UMAP
Format: methodEvidence: in daily use
A useful picture can change what you are looking for. You may start with a subject in mind, then notice a colour relationship or composition worth following. That shift interests me because it turns search into part of developing an idea.
Visual Search Workbench gives that shift a path through a photo collection. A search produces candidates; a candidate can become the next query. Alongside the search, a UMAP (Uniform Manifold Approximation and Projection) map offers regions of related images to explore.
A selected map region opens its photographs in the linked image grid, where their composition, colour, and subject can be judged together.
Search can lead somewhere you did not expect
Search often starts with a fairly definite task: find a photograph you remember. A description can bring likely candidates into view even when you have forgotten the filename. Once you recognise the picture, the search has done its job.
Visual exploration has a less definite endpoint. A photograph can be useful because it suggests what to look for next. In a reference collection, for example, you might follow a visual resemblance toward a composition you had not yet formulated. The value of that next image depends on the idea you are developing, not only on its match to the original words.
VSW lets a result become the next query through Find similar. The chosen image becomes the reference for a new ranking, with its name visible above the results. Back to text results provides a return to the earlier search. These controls make it possible to follow an image without treating every query as an unrelated start.
The map makes related images browsable
A ranked list arranges candidates around one query. A map provides a different way to look through the material: inspect a region, see which pictures it contains, and choose where to go next. UMAP supplies the arrangement for VSW’s Similarity view.
How UMAP works, in plain terms
The app’s image model turns each photograph into an embedding, a list of numbers used to compare images. Those numbers work together as a representation; individual entries are not labels such as “red,” “flower,” or “portrait.” Comparing the representations gives the software a way to find nearby images without consulting manually assigned tags.
Those nearby images form neighbourhoods. UMAP represents neighbourhood relationships as a weighted graph: a network of connections whose strengths reflect the local relationships in the input. It then looks for a lower-dimensional arrangement that preserves those relationships as well as possible. This is the graph-and-layout process described in the UMAP authors’ explanation.
The arrangement becomes the map on screen. Its purpose is to make some of those relationships available to explore while accepting that a compact layout loses information. The image model supplies the representations; UMAP arranges them; the person decides which pictures are worth following.
What the arrangement means
A neighbourhood is a place to investigate, not a category the app has established. Its pictures may be useful together, or their resemblance may be unimportant to the current brief. The photographs themselves let you make that distinction.
The layout also changes how relationships look. A wide gap between groups is not an exact measure of their difference, and a dense patch does not establish the density of the original representations. Apparent splits can arise during the reduction, as the UMAP clustering guide explains.
Two settings help explain why maps can look different: neighbourhood size and minimum distance. Neighbourhood size determines how broadly the method looks around each item, shifting emphasis between fine local detail and a wider arrangement. Minimum distance affects how tightly points can pack in the layout. The parameter guide illustrates both effects. These are explanations of the method, not controls I am claiming the current app exposes.
The image grid makes the map usable
A region on the map becomes an image selection. That selection appears in the grid, where you can inspect the photographs at a useful size. A promising photograph can then supply the reference for another search.
The grid and map also need distinct interpretations. A search ranking compares the query with the image representations. The distance between marks on the displayed map is not that ranking. Keeping those roles clear lets the map support browsing without asking it to be a precise measuring instrument.
Keep the exploration on an everyday Mac
Local processing was a design constraint for the app. The photographs stay in the folders the person chooses, and the models ship with the application. I tested that arrangement on an 8 GB Apple Silicon Mac.
That constraint extends beyond running the model. My case study describes the latest-range decision, slider corrections, and metadata filling in after the plot and timeline appear. I associate those changes with the 0.2.6 release record, which was unavailable during this article’s source review; the release-specific claims remain my account. The 8 GB test identifies a working configuration; larger-library performance and peak memory still need measurement.
Test retrieval and exploration separately
The design now has an implementation to evaluate. I review the path from query to selection to next action using screen inventories and the running app. The interaction reference makes that path inspectable. WorkBay and the Heuristics Canon support my reviews; no external user study has run.
That evaluation needs to distinguish retrieval from exploration. Retrieval asks whether someone can find a remembered photograph. Exploration asks whether someone can assemble useful references for a creative brief, including pictures that change the direction of the brief.
For retrieval, I would examine completion, mistakes, and effort. For exploration, I would examine the images people keep, the paths they take, and their reasons for continuing from one picture to another. Comparing the map with the grid and image-to-image search would help establish what the map contributes.
An exploration study would need to allow for a useful selection that takes someone beyond the subject they first searched for.