Quantitative Methods in Defense and National Security 2007

Factors that Influence Algorithm Performance in the Face Recognition Grand Challenge
Jonathon Phillips, (NIST), jonathon@nist.gov
Ross Beveridge (Colorado State University),
Geof H. Givens (Colorado State University),
and Bruce A. Draper (Colorado State University)


Over the last two decades the effort to develop effective automatic face recognition has resulted in hundreds, if not thousands of papers. Typically, these papers report performance on a single data set in order to draw comparisons between alternative approaches. This type of analysis is valuable when the goal is to conclude that a particular approach is superior to another on a very specific task as exemplified by the data set. However, this style of analysis tells us little about underlying factors that make recognition easier or harder.

When it is addressed at all, the question of what factors affect recognition performance is almost invariably addressed by dataset partitioning. Consider studying the effects of pose and illumination. Several carefully constructed datasets have been developed that lend themselves to studying these factors with partitioned data. Work with the Yale data set and subsequently the PIE data set typically falls into this category. In other words, studies look at relative performance across changes in pose, illumination, or both as exemplified by performance on distinct data partitions. Partitioned data set analysis has also been applied to the question of whether women or men are easier to recognize.

Unfortunately, this approach is less effective the moment one begins to ask questions about more than a handful of factors. It is also far less practical, due to the combinatorial explosion of partitions over multiple factors. Of greater importance is the fact that the practical limitations of partitioning impose important limitations on the ability to control for confounding factors. If one skirts around the combinatorial problem by resorting to marginal analysis (i.e., abandoning control via partitioning), control of confounding effects is eliminated altogether. More sophisticated multi-factor statistical techniques provide greater control and permit more thorough evaluation of factor effects.

Generalized linear mixed models (GLMMs) are one such technique. This paper uses GLMMs to provide the largest statistical analysis to date of factors that influence face recognition performance. Our analysis investigates how a set of 12 factors, henceforth called covariates, predict verification rate at various false accept rates. Performance data for three algorithms from the Face Recognition Grand Challenge Experiment 4 are used in our analysis. The covariates include such things as gender, race, age, distance between eyes in pixels, and apparent focus of the imagery.

Our analysis shows that covariates have a significant effect on performance and that overall performance reported in evaluations does not give a complete picture of the performance properties of face recognition algorithms. The analysis shows for the first time the effect of image based covariates such as size, tilt, and focus of a face on performance. Our analysis shows that a number of the assumptions of the automatic face recognition community are wrong. Our analysis also provides additional scientific evidence supporting observed effects of subject covariates on performance.

Take me back to the main conference page.