Siddhesh Sali

← All work

ml

Out-of-Distribution Detection with Vision-Language Features

Turning Maximum Concept Matching from a zero-shot method into a few-shot one with prompt tuning, so a model can tell when an input is nothing it was trained on.

A diagram: a dense cluster of points inside three dashed rings, with sparse hollow points scattered well outside them.

A classifier will confidently name anything you show it, including things that belong to none of its classes. In deployment that is the failure that matters — not being wrong about a hard example, but being certain about an input that should have been refused.

Out-of-distribution detection is the job of noticing. Most methods do it inside a single modality: look at the image features, decide whether they are unusual. That throws away a signal, because “is this one of my classes?” is a question about meaning, and meaning is where vision-language models are strong.

Maximum Concept Matching scores an image against the text embeddings of the class names — an out-of-distribution input matches none of them well. It works zero-shot, which is its appeal and also its ceiling: it never gets to look at your actual data.

The contribution here was to add prompt tuning on top, which makes MCM a few-shot method instead. A handful of in-distribution examples is enough to learn the prompts, and the concepts stop being generic and start being yours.

The result, and why it is smaller than it looks

Measured on Oxford-IIIT Pets as the in-distribution set, against iNaturalist, SUN, Places365 and DTD as out-of-distribution: tuned prompts move the average AUROC from 99.62 to 99.64, and FPR95 from 1.70 to 1.62.

That is not much, and it is worth being straight about why. Zero-shot MCM on this pair of datasets is already at 99.6 — pets against textures and landscapes is not a hard problem, and there is no room above it. The honest reading is that the separation gets visibly cleaner (the density plot in the gallery is the same scores, before and after) while the metric everyone quotes has already saturated. A harder in-distribution set, with semantically near classes on the other side, is where this method would have something to prove.

Two smaller things fell out of it. How long you tune is not a monotone dial — the longer run helped, and the shorter one came out worse than the untuned prompts it was meant to improve on. And decoding the learned soft prompts back to their nearest tokens returns nothing readable: lindsey, sikhs, generates. Whatever the prompts have learned, it is not English, which is a useful thing to know before writing a paragraph about what the model “understands”.

Supervised by Prof. Biplab Banerjee at IIT Bombay, January – July 2023, and the direct continuation of the deep metric learning thesis — same idea, that language is an underused supervisory signal for vision, pointed at a different problem.

Gallery

0303
Two density plots side by side. In each, a narrow green out-of-distribution peak sits left of a broad blue in-distribution hump; in the right-hand plot the two overlap noticeably less.
Fig. 01Pets against DTD. Hard prompts on the left, tuned on the right — the overlap is what an OOD detector actually lives on.
A grouped bar chart of FPR95 across four out-of-distribution sets and their average, comparing hard prompts against two tuning runs.
Fig. 02False positives at 95% recall. The longer tuning run helps a little; the shorter one is worse than not tuning at all.
A grouped bar chart of AUROC across the same four sets, every bar between 98.5 and 99.8 and the three conditions nearly indistinguishable.
Fig. 03And the metric everyone quotes, which has nothing left to say. Note the axis starts at 97.8.