IJCB2014 tutorial - Biometric Menagerie banner

System Design and Performance Assessment: A Biometric Menagerie Perspective

IJCB2014 Tutorial
29 Sept 2014, 2pm-5pm
Norman Poh
Slide in pdf


One of the major sources of variability in assessing the performance of a biometric system is the subject variability. Testing the same system on two disjoint populations of users almost always invariably yields two different results. This phenomenon was first described by Doddington et al when they evaluated speaker recognition systems in 1998. The users, or more precisely the statistical models associated with them, who are difficult to be recognised by a biometric system are given names such as goats and lambs; whereas users whose biometric traits are likely to be successful at impersonating others are called wolves. This phenomenon is dubbed “Doddington Zoo” and “Biometric Menagerie”. Subsequent studies then either aim at characterising the phenomenon, or at reducing the phenomenon that leads to better tailored decisions such as user-specific decision threshold and user-specific score calibration, and fusion strategies.

In this tutorial, we will describe biometric menagerie and explain how and why it has a direct impact on how the system performance is characterised; how confidence intervals can be estimated; and why performance prediction is difficult.

The significance of biometric menagerie is that it has impact on all biometric modalities. Furthermore, by reducing the phenomenon through user-specific strategies, a relative performance gain of about 30% has been observed for a unimodal biometric system; and up to 50% for a multimodal biometric system.

Video lecture

Talk outline


1. The basics in biometrics performance reporting

  • False Acceptance Rate
  • False Rejection Rate
  • Equal Error Rate 

2. Characterising Biometric Menagerie

  • Doddington’s classification
  • Yager and Dustone’s classification
  • Some explanations and conjectures of the causes of Menagerie
  • Biometric Menagerie Index
  • User-ranking and its applications

3. Performance Intervals Estimation in the Presence of Biometric Menagerie

  • A two-step bootstrapping procedure that takes Biometric Menagerie into consideration
  • Worst-case scenario DET curves
  • Performance generalisation under different subject compositions and application settings

4. Reducing the Impact of Biometric Menagerie

  • User-specific decision threshold
  • User-specific score calibration
    • F-norm, Z-norm, T-norm
    • Group-based normalisation
    • Discriminative normalisation
  • User-specific fusion
    • Direct design of user-specific fusion
    • A two-step design of user-specific fusion
    • User-specific and modality-selective fusion via B-ratio

5. Open-ended questions involving Biometric Menagerie

  • Spoofing and Menagerie
  • Biometric Menagerie – an algorithmic, modality, or quality issue?
  • Estimating unimodal and multikodal biometric performance over time


  • G. Doddington, W. Liggett, A. Martin, M. Przybocki, and D. Reynolds. Sheep, goats, lambs and wolves: A statistical analysis of speaker performance in the nist 1998 speaker recognition evaluation. In Int’l Conf. Spoken Language Processing (ICSLP), Sydney, Australia, November 1998.
  • A. Hicklin, C. Watson, and B. Ulery. The myth of goats: How many people have fi ngerprints that are hard to match. Technical report, NIST, 2005. IR7271.
  • N. Poh and S. Bengio, An Investigation of F-ratio Client-Dependent Normalisation on Biometric Authentication Tasks, in IEEE Int'l Conf. Acoustics, Speech, and Signal Processing (ICASSP), volume I, pages 721-724, 2005.
  • J. Fierrez-Aguilar, J. Ortega-Garcia and J. Gonzalez-Rodriguez, "Target dependent score normalization techniques and their application to signature verification", IEEE Trans. on Systems, Man & Cybernetics - Part C, Vol. 35, n. 3, pp. 418-425, August 2005.
  • J. Fierrez-Aguilar, D. Garcia-Romero, J. Ortega-Garcia and J. Gonzalez-Rodriguez, "Bayesian adaptation for user-dependent multimodal biometric authentication", Pattern Recognition, Vol. 38, n. 8, pp. 1317-1319, August 2005.
  • J. Fierrez-Aguilar, J. Ortega-Garcia, J. Gonzalez-Rodriguez and J. Bigun, "Discriminative multimodal biometric authentication based on quality measures", Pattern Recognition, Vol. 38, n. 5, pp. 777-779, May 2005.
  • M. Wittman, P. Davis, and P. J. Flynn. Empirical studies of the existence of the biometric menagerie in the frgc 2.0 color image corpus. In Computer Vision and Pattern Recognition Workshop (CVPRW ’06), June 2006.
  • N. Poh and S. Bengio, Performance Generalization in Biometric Authentication Using Joint User-specific and Sample Bootstraps , IEEE Trans. on Pattern Analysis and Machine Intelligence, 29(3):492-498, March 2007
  • N. Poh and J. Kittler, A Method for Estimating authentication Performance Over Time , in 12th Iberoamerican Congress on Pattern Recognition CIARP, Via del Mar-Valparaiso, Chile, pp. 360-369, 2007
  • M. Une, A. Otsuka, and H. Imai. Wolf attack probability: A new security measure in biometric authentication systems. In Advances in Biometrics, volume 4642, pages 396–406. LNCS, Springer-Verlag Berlin Heidelberg, 2007.
  • N. Poh and J. Kittler, A Methodology for Separating Sheep from Goats for Controlled Enrollment and Multimodal Fusion, in 6th Biometrics Symposium, pp. 17–22, 2008.
  • N. Poh and J. Kittler, Incorporating Variation of Model-specific Score Distribution in Speaker Verification Systems, IEEE Trans. Audio, Speech and Language Processing, 16(3):594-606, 2008.
  • N. Poh and J. Kittler, A Biometric Menagerie Index for Characterising Template/Modelspecific Variation, in Int’l Conf. on Biometrics (ICB'09), 2009.
  • N Poh, A. Merati and J. Kittler, Adaptive Client-Impostor Centric Score Normalization: A Case Study in Fingerprint Verification, in IEEE Conference on Biometrics: Theory, Applications and Systems (BTAS'09), pages 1-7, 2009.
  • Beveridge, J. Ross, et al. "Factors that influence algorithm performance in the face recognition grand challenge." Computer Vision and Image Understanding 113.6 (2009): 750-762.
  • N. Poh, User-specific Score Normalization and Fusion for Biometric Person Recognition, in Advanced Topics in Biometrics, Chap 16, 401-418, edited by Haizhou Li, Kar-Ann Toh and Liyuan Li, World Scientific Publishing, 2010
  • Yager, Neil, and Ted Dunstone. "The biometric menagerie." Pattern Analysis and Machine Intelligence, IEEE Transactions on 32.2 (2010): 220-230.
  • Tabassi, Elham. "Image specific error rate: A biometric performance metric." Pattern Recognition (ICPR), 2010 20th International Conference on. IEEE, 2010.
  • N. Poh, A. Rattani, M. Tistarelli and J. Kittler, Group-specific Score Normalization for Biometric Systems, in IEEE Computer Society Workshop on Biometrics (CVPR), pages 38-45, 2010.
  • Klare, Brendan F., and Anil K. Jain. "Face recognition: Impostor-based measures of uniqueness and quality." Biometrics: Theory, Applications and Systems (BTAS), 2012 IEEE Fifth International Conference on. IEEE, 2012.
  • N. Poh and M. Tistarelli, Customizing Biometric Authentication Systems via Discriminative Score Calibration, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
  • A. Merati, N. Poh and J. Kittler, User-Specific Cohort Selection and Score Normalization for Biometric Systems, IEEE Trans. on Information Forensics and Security, 7(4):1270-1277, 2012.
  • Wayman, James L., Antonio Possolo, and Anthony J. Mansfield. "Modern statistical and philosophical framework for uncertainty assessment in biometric performance testing." IET biometrics 2.3: 85-96,2013
  • Paone, Jeffrey Richard. Liberating the Biometric Menagerie Through Score Normalization Improvements. Diss. University of Notre Dame, 2013.
  • DeCann, B.; Ross, A, "Relating ROC and CMC curves via the biometric menagerie," Biometrics: Theory, Applications and Systems (BTAS), 2013 IEEE Sixth International Conference on , vol., no., pp.1,8, Sept. 29 2013-Oct. 2 2013.
  • Howard, John J., and Delores Etter. "The effect of ethnicity, gender, eye color and wavelength on the biometric menagerie." In Technologies for Homeland Security (HST), 2013 IEEE International Conference on, pp. 627-632. IEEE, 2013.
  • Li, Shengzhe, Hakil Kim, Changlong Jin, Stephen Elliott, and Mingjie Ma. "Assessing the level of difficulty of fingerprint datasets based on relative quality measures." Information Sciences 268 (2014): 122-132.