IJCB2014 tutorial - Biometric Menagerie
Design and Performance Assessment: A Biometric Menagerie Perspective
29 Sept 2014, 2pm-5pm
Slide in pdf
One of the major sources of variability in assessing the performance of
a biometric system is the subject variability. Testing the same system
on two disjoint populations of users almost always invariably yields
two different results. This phenomenon was first described by
Doddington et al when they evaluated speaker recognition systems in
1998. The users, or more precisely the statistical models associated
with them, who are difficult to be recognised by a biometric system are
given names such as goats and lambs; whereas users whose biometric
traits are likely to be successful at impersonating others are called
wolves. This phenomenon is dubbed “Doddington Zoo” and “Biometric
Menagerie”. Subsequent studies then either aim at characterising the
phenomenon, or at reducing the phenomenon that leads to better tailored
decisions such as user-specific decision threshold and user-specific
score calibration, and fusion strategies.
In this tutorial, we will describe biometric menagerie and explain how
and why it has a direct impact on how the system performance is
characterised; how confidence intervals can be estimated; and why
performance prediction is difficult.
The significance of biometric menagerie is that it has impact on all
biometric modalities. Furthermore, by reducing the phenomenon through
user-specific strategies, a relative performance gain of about 30% has
been observed for a unimodal biometric system; and up to 50% for a
multimodal biometric system.
1. The basics in biometrics
- False Acceptance
- False Rejection
- Equal Error
2. Characterising Biometric Menagerie
- Yager and
- Some explanations
and conjectures of the causes of Menagerie
- User-ranking and
3. Performance Intervals Estimation
in the Presence of Biometric Menagerie
- A two-step
bootstrapping procedure that takes Biometric Menagerie into
scenario DET curves
generalisation under different subject compositions and application
4. Reducing the Impact of Biometric
- F-norm, Z-norm,
- Direct design of
- A two-step
design of user-specific fusion
and modality-selective fusion via B-ratio
5. Open-ended questions involving
- Spoofing and
Menagerie – an algorithmic, modality, or quality issue?
unimodal and multikodal biometric performance over time
- G. Doddington, W. Liggett, A. Martin, M. Przybocki, and D.
Reynolds. Sheep, goats, lambs and wolves: A statistical analysis of
speaker performance in the nist 1998 speaker recognition evaluation. In
Int’l Conf. Spoken Language Processing (ICSLP), Sydney, Australia,
- A. Hicklin, C. Watson, and B. Ulery. The myth of goats: How
many people have fi ngerprints that are hard to match. Technical
report, NIST, 2005. IR7271.
- N. Poh and S. Bengio, An Investigation of F-ratio
Client-Dependent Normalisation on Biometric Authentication Tasks, in
IEEE Int'l Conf. Acoustics, Speech, and Signal Processing (ICASSP),
volume I, pages 721-724, 2005.
- J. Fierrez-Aguilar, J. Ortega-Garcia and J.
Gonzalez-Rodriguez, "Target dependent score normalization techniques
and their application to signature verification", IEEE Trans. on
Systems, Man & Cybernetics - Part C, Vol. 35, n. 3, pp.
418-425, August 2005.
- J. Fierrez-Aguilar, D. Garcia-Romero, J. Ortega-Garcia and
J. Gonzalez-Rodriguez, "Bayesian adaptation for user-dependent
multimodal biometric authentication", Pattern Recognition, Vol. 38, n.
8, pp. 1317-1319, August 2005.
- J. Fierrez-Aguilar, J. Ortega-Garcia, J. Gonzalez-Rodriguez
and J. Bigun, "Discriminative multimodal biometric authentication based
on quality measures", Pattern Recognition, Vol. 38, n. 5, pp. 777-779,
- M. Wittman, P. Davis, and P. J. Flynn. Empirical studies of
the existence of the biometric menagerie in the frgc 2.0 color image
corpus. In Computer Vision and Pattern Recognition Workshop (CVPRW
’06), June 2006.
- N. Poh and S. Bengio, Performance Generalization in
Biometric Authentication Using Joint User-specific and Sample
Bootstraps , IEEE Trans. on Pattern Analysis and Machine Intelligence,
29(3):492-498, March 2007
- N. Poh and J. Kittler, A Method for Estimating
authentication Performance Over Time , in 12th Iberoamerican Congress
on Pattern Recognition CIARP, Via del Mar-Valparaiso, Chile, pp.
- M. Une, A. Otsuka, and H. Imai. Wolf attack probability: A
new security measure in biometric authentication systems. In Advances
in Biometrics, volume 4642, pages 396–406. LNCS, Springer-Verlag Berlin
- N. Poh and J. Kittler, A Methodology for Separating Sheep
from Goats for Controlled Enrollment and Multimodal Fusion, in 6th
Biometrics Symposium, pp. 17–22, 2008.
- N. Poh and J. Kittler, Incorporating Variation of
Model-specific Score Distribution in Speaker Verification Systems, IEEE
Trans. Audio, Speech and Language Processing, 16(3):594-606, 2008.
- N. Poh and J. Kittler, A Biometric Menagerie Index for
Characterising Template/Modelspecific Variation, in Int’l Conf. on
Biometrics (ICB'09), 2009.
- N Poh, A. Merati and J. Kittler, Adaptive Client-Impostor
Centric Score Normalization: A Case Study in Fingerprint Verification,
in IEEE Conference on Biometrics: Theory, Applications and Systems
(BTAS'09), pages 1-7, 2009.
- Beveridge, J. Ross, et al. "Factors that influence
algorithm performance in the face recognition grand challenge."
Computer Vision and Image Understanding 113.6 (2009): 750-762.
- N. Poh, User-specific Score Normalization and Fusion for
Biometric Person Recognition, in Advanced Topics in Biometrics, Chap
16, 401-418, edited by Haizhou Li, Kar-Ann Toh and Liyuan Li, World
Scientific Publishing, 2010
- Yager, Neil, and Ted Dunstone. "The biometric menagerie."
Pattern Analysis and Machine Intelligence, IEEE Transactions on 32.2
- Tabassi, Elham. "Image specific error rate: A biometric
performance metric." Pattern Recognition (ICPR), 2010 20th
International Conference on. IEEE, 2010.
- N. Poh, A. Rattani, M. Tistarelli and J. Kittler,
Group-specific Score Normalization for Biometric Systems, in IEEE
Computer Society Workshop on Biometrics (CVPR), pages 38-45, 2010.
- Klare, Brendan F., and Anil K. Jain. "Face recognition:
Impostor-based measures of uniqueness and quality." Biometrics: Theory,
Applications and Systems (BTAS), 2012 IEEE Fifth International
Conference on. IEEE, 2012.
- N. Poh and M. Tistarelli, Customizing Biometric
Authentication Systems via Discriminative Score Calibration, IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
- A. Merati, N. Poh and J. Kittler, User-Specific Cohort
Selection and Score Normalization for Biometric Systems, IEEE Trans. on
Information Forensics and Security, 7(4):1270-1277, 2012.
- Wayman, James L., Antonio Possolo, and Anthony J.
Mansfield. "Modern statistical and philosophical framework for
uncertainty assessment in biometric performance testing." IET
biometrics 2.3: 85-96,2013
- Paone, Jeffrey Richard. Liberating the Biometric Menagerie
Through Score Normalization Improvements. Diss. University of Notre
- DeCann, B.; Ross, A, "Relating ROC and CMC curves via the
biometric menagerie," Biometrics: Theory, Applications and Systems
(BTAS), 2013 IEEE Sixth International Conference on , vol., no.,
pp.1,8, Sept. 29 2013-Oct. 2 2013.
- Howard, John J., and Delores Etter. "The effect of
ethnicity, gender, eye color and wavelength on the biometric
menagerie." In Technologies for Homeland Security (HST), 2013 IEEE
International Conference on, pp. 627-632. IEEE, 2013.
- Li, Shengzhe, Hakil Kim, Changlong Jin, Stephen Elliott,
and Mingjie Ma. "Assessing the level of difficulty of fingerprint
datasets based on relative quality measures." Information Sciences 268