Active Prompting

Active prompting is an adaptive approach to few-shot example selection that borrows from active learning. Rather than picking demonstrations at random or by surface similarity, the method runs the model on a pool of unlabeled examples, measures uncertainty — often as the variance of answers across temperature-sampled runs — and selects the most uncertain examples for human annotation. Those annotated examples then become the few-shot demonstrations used at inference. It targets the model's specific gaps instead of padding the prompt with easy cases the model already handles, and typically beats random or similarity-based selection on reasoning benchmarks when the annotation budget is small.