Eight real held-out images
Examples
Eight real images the model did not train on, chosen to show different combinations of results. For each, compare the output probability (the probability the network gives its top answer) with its familiarity score at the early and late stages, then open the image in Try.
Eight held-out images. Each with output probability and two familiarity scores.
The three results are: the output probability; the familiarity score at the early stages; and the familiarity score at the late stage. Learn explains how familiarity is worked out.
The early-stage score combines four middle stages of the network (it takes their median). The late-stage score comes from the last stage. Each card places the image in one of five bands, from least to most familiar, with edges set on one held-out set of 2,000 images and rates measured on a second, separate held-out set of 2,000 (Learn explains how). The bands are not medical thresholds or absolute measures of familiarity.
Loading the example images…
Dis/agreement between output probability and familiarity
The examples come from 48 held-out images unseen by the model: the first 24 benign and the first 24 malignant images in one held-out set. Each image is free to reuse (CC0), and its source is credited under it. Every image was run through the same model this site uses, so the numbers match what you see in Try.
From those 48 we picked four images, one for each mix of high or low output probability with high or low late-stage familiarity. We added two where early and late familiarity disagree, one the model got wrong and one it got right. When we picked them, “high” meant above the middle of the 48. The cards now describe each image with the bands instead.
Because these were picked by hand, they show what can happen, not how often it happens.
A familiar-looking case
The model favours one label, and the image looks familiar. It sits close to training images the model labelled correctly, so the same-label part of the familiarity score is high. This describes what surrounds the image. It does not prove the answer is right.
Very sure, but near past mistakes
The model can be very sure of a label even when the image closely resembles training images it wrongly gave that same label. Its output probability cannot show this. The past-mistakes part of the familiarity score can, and it pulls the score down.
An entirely unfamiliar case
A phone photo, a drawing or a pattern of colours can still get a very sure answer. The familiarity score will not always catch this either. Because it is a ratio, it can stay high even when the image is far from every training image.
Choose an example image and see all five familiarity scores →More networks are being added
This site currently shows one network, EfficientNetB0. The same method is being run on additional networks, and their results will be added here.