Randomness is the Root of All Evil: More Reliable Evaluation of Deep Active Learning

Abstract

Using deep neural networks for active learning (AL) poses significant challenges for the stability and the reproducibility of experimental results. Inconsistent settings continue to be the root causes for contradictory conclusions and in worst cases, for incorrect appraisal of methods. Our community is in search of a unified framework for exhaustive and fair evaluation of deep active learning. We provide just such a framework, one which is built upon systematically fixing, containing and interpreting sources of randomness. We isolate different influence factors, such as neural-network initialization or hardware specifics, to assess their impact on the learning performance. We then use our framework to analyze the effects of basic AL settings, such as the query-batch size and the use of subset selection, and different datasets on AL performance. Our findings enable us to derive specific recommendations for the reliable evaluation of deep active learning, thus helping advance the community toward a more normative evaluation of results.

For further details please consult the conference publication.

Below you see an overview of the influence factors considered by our work in comparison to related work. Up to now, the community has only observed isolated effects that become apparent in the evaluation of AL. We are the first to systematically tie these effects to various sources of randomness.

As an example, the used GPU model can change the ranking of AL methods despite enforcing deterministic computations in the learning framework. The lower the averaged results in the "pair-wise penalty matrix" the better.

.

Team

• Yilin Ji (Contact)
• Daniel Kaestner
• Oliver Wirth
• Christian Wressnegger

Proof-of-Concept Implementations

For the sake of reproducibility and to foster future research, we make our framework for evaluting deep active learning methods publicly available at:

https://github.com/intellisec/eval-al

Publication

A detailed description of our work will been presented at the (WACV 2023) in January 2023. If you would like to cite our work, please use the reference as provided below:

@InProceedings{Ji2023Randomness,
author    = {Yilin Ji and Daniel Kaestner and Oliver Wirth
and Christian Wressnegger},
booktitle = {Proc. of the {IEEE} Winter Conference on Applications
of Computer Vision ({WACV})},
title     = {Randomness is the Root of All Evil:
More Reliable Evaluation of Deep Active Learning},
year      = {2023},
month     = jan
}

A preprint of the paper is available here.