Voice recognition Part 2

Thanks Nick.  It's good to be able to continue this thread.

The figure of 99.6% "first time authorisation" sounds good. Can you provide details of how that's measured please?  For comparison's sake, what's the Equal Error Rate (which the FBI reckons is 2% best case for voice). And is it measured under the "Zero Effort Imposter" assumption?

Anyway, I am not trying to drag us into specmanship.  Rather, I am keen to debate the 'fine print' for the sake of policy people and non technologists who are making important decisions, without properly comprehending how biometrics work, and how and why they fail.  I'm sure that good progress will continue to be made on most biometrics, and I agree they have a role to play in some applications.  But I believe that use of the term "unique" may mislead people, perhaps inadvertently, perhaps not.  It doesn't quite gel to claim on the one hand that a certain trait is "unique" while on the other hand, real life performance of the biometric apparatus falls short of 100%.  Sometimes, it falls way short.

[For instance, only recently, former US Homeland Security chief Michael Chertoff, in a speech about identity technology , stated without qualification that “your fingerprint is unique”.  Even if that were true (and some researchers actually think it’s a myth) the fact is that fingerprint scanners don’t have the precision to deliver “uniqueness”.  The UK Government CESG testing I mentioned last time showed typical False Accept Rates of 1 in a thousand.  That sounds nearly unique, but the corresponding False Reject Rates for two of the scanners tested were 10% and 25%, which would render them unusable in automatic banking.  So in practice, the False Accept Rate will have to be de-tuned, maybe to 1 in a hundred, where the False Reject Rate is still worryingly high, between 5% and 15%. Getting this tradeoff right in unattended banking security applications will involve tough choices, hopefully made transparently.]

Is my criticism of abuse of the word "unique" mere pedantry?  I don't think so.  We're talking about security, and the first thing anyone needs to know -- as Nick has properly reinforced -- is that no security is perfect.  Yet loose talk around "uniqueness" is giving all sorts of lay people (policy makers, politicians, bankers ...) a false sense of security. “Uniqueness” just doesn’t manifest in practical biometric performance.  To make systems not falsely reject unreasonably high numbers of genuine customers, the False Match Rate needs to be compromised.  Uniqueness is deliberately sacrificed by engineers, while the marketing departments continue to bang on about it.

Worse than this, published error rates for biometrics are chronically optimistic.  As I mentioned previously, biometric performance measurement almost always uses the “zero effort imposter” assumption, which causes systemic over-estimation of their strength.  False Match Rates are worked out by counting accidental matches, and do not look at scenarios where someone is trying to get matched falsely.

The recent FBI report State of the Art Biometrics Excellence Roadmap, Technology Assessment October 2008 says:

The intentional spoofing or manipulation of biometrics invalidates the “zero effort imposter” assumption commonly used in performance evaluations. When a dedicated effort is applied toward fooling biometrics systems, the resulting performance can be dramatically different. (see Vol 1, page 1.4).

And

For all biometric technologies, error rates are highly dependent upon the population and application environment. The technologies do not have known error rates outside of a controlled test environment. Therefore, any reference to error rates applies only to the test in question and should not be used to predict performance in a different application. (page Vol 1, 2.10).

In other words, the stated performance specifications of biometric solutions don’t tell us much about how well they stand up to criminal attack.  This failing is nothing short of remarkable.  Shouldn’t resistance to would-be robbers be top of mind when commissioning banking system security?

Cheers,

Stephen.

Regarding the technical issue, we're looking into it to see if there are any problems. But I'd just like to remind community members that commenting on a blog is a two-step process: Once you've written the comment in the box provided, the button at the bottom is labelled save. Click this and your comment is then displayed to you as a type of preview, but you then need to click the publish button below the preview to actually make it live.

Indeed - the idea is you get the chance to review and fix any problems with your comment before it goes live. If you forget to click publish - just go to your personal home page and you'll get the chance to fix it.