ZEBRA: Zero Evidence Biometric Recognition Assessment

Overview

ZEBRA: Zero Evidence Biometric Recognition Assessment

license: LGPLv3 - please reference our paper
version: 2020-06-11
author: Andreas Nautsch (EURECOM)

Disclaimer - this toolkit is a standalone implementation of our paper

Nautsch, Patino, Tomashenko, Yamagishi, Noe, Bonastre, Todisco and Evans:
"The Privacy ZEBRA: Zero Evidence Biometric Recognition Assessment"
in Proc. Interspeech 2020

[Online] pre-print

This work is academic (non-for-profit);
a reference implementation without warranty.

What is the ZEBRA framework?

How can we assess for privacy preservation in the processing of human signals, such as speech data?

Mounting privacy legislation calls for the preservation of privacy in speech technology, though solutions are gravely lacking. While evaluation campaigns are long-proven tools to drive progress, the need to consider a privacy adversary implies that traditional approaches to evaluation must be adapted to the assessment of privacy and privacy preservation solutions. This paper presents the first step in this direction: metrics.

We propose the ZEBRA framework which is inspired by forensic science.

On the contrary to method validation in modern cryptography, which is backboned by zero-knowledge proofs (see Shanon), we need to tackle zero-evidence. The former defines input data (e.g., an A is represented by the number 65); the latter models input data (we can only describe e.g., acoustic data, biometric identities, and semantic meaning).

Communication is more than the written word; we need to leave the stiff perspective of the written word behind, when the medium changes to speech and to other human signals (e.g., video surveillance).

Privacy preservation for human data is not binary

Only levels of privacy preservation can be quantified (theoretic proofs for a yes/no decision are unavailble).

The ZEBRA framework compares candidate privacy safeguards in an after-the-fact evaluation:

  • candidate algorithms protect human signals (e.g., speech) regarding the disclosure of specific sensitive information (e.g., the biometric identity);
  • knowing the facts of how much sensitive information could be exposed, how much is exposed after using each candidate safeguard?

Privacy & the realm of the adversary

The conventional signal processing or machine learning perspectives as system evaluators does not suffice anymore!

Adversaries are evaluators of safeguard evaluators.

We need to shift our perspective.

To optimize algorithms and their parameters, we are used to improve some average/expected performance loss.

  1. Expectation values reflect on a population level, yet, privacy is a fundamental human right that is: for each individual - how badly is information disclosed for those who belong to a minority in the eyes of a candidate privacy safeguard?
  2. An adversary can only infer information based on observations - figuratively speaking like a judge/jury assesses evidence.
  3. By formalizing decision inference based on evidence, the strength of evidence is estimated - it allows to reflect to which extent one of two decisions should be favored over another; given the circumstances of a case - an individual performance.
  4. Given the circumstances is formalized by the prior belief; like forensic practitioners cannot know the prior belief of a judge/jury, we cannot know the prior belief of an adversary.
  5. Empirical Cross-Entropy (ECE) plots are introduced in forensic voice biometrics to simulate ECE for all possible prior beliefs, such that one can report an expected gain in relative information - an average/expected performance.
  6. Categorical tags are introduced in forensic science and constantly refined since the 1960s to summarize different levels of strength of evidence into a scale that is easier to digest by the human mind.

ZEBRA, a zero-evidence framework to assess for preserving privacy on empirical data

The proposed ZEBRA framework has two metrics:

  • on the population level, the expected ECE is quantified by integrating out all possible prior beliefs; the result is: expected empirical cross-entropy [in bits] which is 0 bit for full privacy; and 1/log(4) ~ 0.721 bit for no privacy.
  • on the individual level, the worst-case strength of evidence is quantified. In forensic science, the strength of evidence is referred to a so-called log-likelihood ratio (LLR) which symmetrically encodes the relative strength of evidence of one possible decision outcome over the other; an LLR of 0 means zero strength of evidence for either possible deicision outcome - on the contrary, values towards inifnity would resembe towards 'inifnitely decisive' evidence (no privacy). The worst-case strength of evidence is the maximum(absolute(LLR)).

Categorical tags summarizes the maximum(absolute(LLR)) value; an example adopted from the literature:

Tag Description (for a 50:50 prior belief)
0 50:50 decision making of the adversary
A adversary makes better decisions than 50:50
B adversary makes 1 wrong decision out of 10 to 100
C adversary makes 1 wrong decision out of 100 to 1000
D adversary makes 1 wrong decision out of 1000 to 100.000
E adversary makes 1 wrong decision out of 100.000 to 1.000.000
F adversary makes 1 wrong decision in at least 1.000.000

The better an adversary can make decisions, despite the privacy preservation of a candidate safeguard applied, the worse is the categorical tag.

Scope of this ZEBRA reference implementation

  1. Computation and visualization of the ZEBRA framework.

    • Metrics in ZEBRA profile: (population, individual, tag)
    • ZEBRA profile in ECE plots
      full privacy: black profile
      no privacy: y = 0 (profiles equal to the x-axis)
      alt text

    For display only, LLRs are in base 10.

  2. Saving to: LaTeX, PDF and PNG formats.

  3. Automatic assessment of the 2020 VoicePrivacy Challenge
    ReadMe: use ZEBRA for kaldi experiments

  4. Computation and visualizations of conventional metrics:
    ReadMe: conventional plots & metrics

    • ECE plots (Ramos et al.)
      metrics: ECE & min ECE
    • APE plots (Brümmer et al.)
      metrics: DCF & min DCF
    • Computation only
      metrics: Cllr, min Cllr & ROCCH-EER

Installation

The installation uses Miniconda, which creates Python environments into a folder structure on your hard drive.

Deinstallation is easy: delete the miniconda folder.

  1. install miniconda, see:
    https://docs.conda.io/projects/conda/en/latest/user-guide/install/#regular-installation
  2. create a Python environment

    conda create python=3.7 --name zebra -y

  3. activate the environment

    conda activate zebra

  4. installing required packages

    conda install -y numpy pandas matplotlib seaborn tabulate

HowTo: use

A quick reference guide for using Python, the command line and to customization.

Command line: metric computation

Computing the metrics (command structure):

python zero_evidence.py -s [SCORE_FILE] -k [KEY_FILE]

An example is provided with scores.txt and key.txt as score and key files:

scr=exp/Baseline/primary/results-2020-05-10-14-29-38/ASV-libri_test_enrolls-libri_test_trials_f/scores
key=keys-voiceprivacy-2020/libri_test_trials_f

python zero_evidence.py -s $scr -k $key

Result:

ZEBRA profile
Population: 0.584 bit
Individual: 3.979 (C)

Command line: visualization

Display each plot: -p option: python zero_evidence.py -s $scr -k $key -p

Command line: customization

  1. Custom label for an experiment: -l option:

    python zero_evidence.py -s $scr -k $key -l "libri speech, primary baseline"
    

    libri speech, primary baseline
    Population: 0.584 bit
    Individual: 3.979 (C)

  2. Save the profile visualization (without their display): -e png

    python zero_evidence.py -s $scr -k $key -l "profile" -e png
    

    -l profile for a file name: ZEBRA-profile
    note: "ZEBRA-" is an automatic prefix to the exported plot file names

    Supported file types:

    • -e tex: LaTeX
    • -e pdf: PDF
    • -e png: PNG
  3. To save a plot with its display, use both options: -p -e png

Python: high-level implementation

Calling the API provided by zebra.py

from zebra import PriorLogOddsPlots, zebra_framework, export_zebra_framework_plots

# initialize the ZEBRA framework 
zebra_plot = PriorLogOddsPlots()  

# declare score & key paths
scr = 'exp/Baseline/primary/results-2020-05-10-14-29-38/ASV-libri_test_enrolls-libri_test_trials_f/scores'
key = 'keys-voiceprivacy-2020/libri_test_trials_f'

# run the framework
zebra_framework(plo_plot=zebra_plot, scr_path=scr, key_path=key)

# saving the ZEBRA plot
export_zebra_framework_plots(plo_plot=zebra_plot, filename='my-experiment', save_plot_ext='png')

Python: low-level implementation

Code snippets from zebra.py

Let's assume classA_scores & classB_scores are numpy arrays of scores.

from numpy import log, abs, hstack, argwhere
from zebra import PriorLogOddsPlots

zebra_plot = PriorLogOddsPlots(classA_scores, classB_scores)

# population metric
dece = zebra_plot.get_delta_ECE()

# individual metric
max_abs_LLR = abs(hstack((plo_plot.classA_llr_laplace, plo_plot.classB_llr_laplace))).max()

# categorical tag
max_abs_LLR_base10 = max_abs_LLR / log(10)
cat_idx = argwhere((cat_ranges < max_abs_LLR_base10).sum(1) == 1).squeeze()
cat_tag = list(categorical_tags.keys())[cat_idx]

# nicely formatted string representations
str_dece = ('%.3f' if dece >= 5e-4 else '%.e') % dece
str_max_abs_llr = ('%.3f' if max_abs_LLR >= 5e-4 else '%.e') % max_abs_LLR

if dece == 0:
    str_dece = '0'

if max_abs_LLR == 0:
    str_max_abs_llr = '0'

For getting the privacy related version on DCF plots, simply run: zebra_plot.get_delta_DCF().

Python: on changing categorical tags

  1. Make a copy of zebra.py
  2. Edit the following part to your liking

    the arrays such as array([0, eps]) contain lower and upper bounds; for numerical convenience only, an epsilon value is used.
    The limits are in base-10 LLR intervals.

    # Here are our categorical tags, inspired by the ENFSI sacle on the stength of evidence
    # Please feel free to try out your own scale as well :)
    # dict: { TAG : [min max] value of base10 LLRs }
    categorical_tags = {
        '0': array([0, eps]),
        'A': array([eps, 1]),
        'B': array([1, 2]),
        'C': array([2, 4]),
        'D': array([4, 5]),
        'E': array([5, 6]),
        'F': array([6, inf])
    }
    
    # pre-computation for easier later use
    cat_ranges = vstack(list(categorical_tags.values()))
    

Documentation

This package is based on:

In performance.py, one can find derived and adjusted code snippets.

For legacy compatability, the code is structured to also provide DCF and ECE visualizations.

Naturally, Cllr, min Cllr and the ROCCH-EER can also be computed with the ZEBRA toolkit. For optimization, please see pyBOSARIS; eventually, to optimize ZEBRA, we would recommend to optimize Cllr. (see Niko Brümmer's disseration or the BOSARIS toolkit user guide regarding convexity).

This toolkit is organized as follows:

  • demo_conventional_plots.py
    Creation of conventional ECE & APE plots
  • demo_voiceprivacy_challenge.py
    Automatic ZEBRA evaluation of an entire challenge
  • demo_zebra.py
    Example on creating a ZEBRA plot and exporting to tex, pdf, png
  • helpers.py
    Helpers to read score files from the kaldi folder structure
  • performance.py
    Library of integrated performance functions, see related software
  • plo_plots.py
    Implementation of ECE & APE plots in one class: PriorLogOddsPlots; with plot export functionality
  • zebra.py
    Wrapper functions to interact with PriorLogOddsPlots in ZEBRA style
  • zero_evidence.py
    Command line script for ZEBRA framework

Acknowledgements

This work is partly funded by the projects: ANR-JST VoicePersonae, ANR Harpocrates and ANR-DFG RESPECT.

You might also like...
Implementation of "StrengthNet: Deep Learning-based Emotion Strength Assessment for Emotional Speech Synthesis"

StrengthNet Implementation of "StrengthNet: Deep Learning-based Emotion Strength Assessment for Emotional Speech Synthesis" https://arxiv.org/abs/2110

Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Source code for paper "Deep Superpixel-based Network for Blind Image Quality Assessment"

DSN-IQA Source code for paper "Deep Superpixel-based Network for Blind Image Quality Assessment" Requirements Python =3.8.0 Pytorch =1.7.1 Usage wit

A user-friendly research and development tool built to standardize RL competency assessment for custom agents and environments.
A user-friendly research and development tool built to standardize RL competency assessment for custom agents and environments.

Built with ❤️ by Sam Showalter Contents Overview Installation Dependencies Usage Scripts Standard Execution Environment Development Environment Benchm

Lightweight Face Image Quality Assessment

LightQNet This is a demo code of training and testing [LightQNet] using Tensorflow. Uncertainty Losses: IDQ loss PCNet loss Uncertainty Networks: Mobi

MRQy is a quality assurance and checking tool for quantitative assessment of magnetic resonance imaging (MRI) data.
MRQy is a quality assurance and checking tool for quantitative assessment of magnetic resonance imaging (MRI) data.

Front-end View Backend View Table of Contents Description Prerequisites Running Basic Information Measurements User Interface Feedback and usage Descr

No-reference Image Quality Assessment(NIQA) Algorithms (BRISQUE, NIQE, PIQE, RankIQA, MetaIQA)
No-reference Image Quality Assessment(NIQA) Algorithms (BRISQUE, NIQE, PIQE, RankIQA, MetaIQA)

No-Reference Image Quality Assessment Algorithms No-reference Image Quality Assessment(NIQA) is a task of evaluating an image without a reference imag

Pytorch implementation of "MOSNet: Deep Learning based Objective Assessment for Voice Conversion"

MOSNet pytorch implementation of "MOSNet: Deep Learning based Objective Assessment for Voice Conversion" https://arxiv.org/abs/1904.08352 Dependency L

To propose and implement a multi-class classification approach to disaster assessment from the given data set of post-earthquake satellite imagery.
To propose and implement a multi-class classification approach to disaster assessment from the given data set of post-earthquake satellite imagery.

To propose and implement a multi-class classification approach to disaster assessment from the given data set of post-earthquake satellite imagery.

Owner
Voice Privacy Challenge
The VoicePrivacy initiative aims to promote the development of privacy preservation tools for speech technology.
Voice Privacy Challenge
[CVPR 2021] Released code for Counterfactual Zero-Shot and Open-Set Visual Recognition

Counterfactual Zero-Shot and Open-Set Visual Recognition This project provides implementations for our CVPR 2021 paper Counterfactual Zero-S

null 144 Dec 24, 2022
Rethinking of Pedestrian Attribute Recognition: A Reliable Evaluation under Zero-Shot Pedestrian Identity Setting

Pytorch Pedestrian Attribute Recognition: A strong PyTorch baseline of pedestrian attribute recognition and multi-label classification.

Jian 79 Dec 18, 2022
Easy and comprehensive assessment of predictive power, with support for neuroimaging features

Documentation: https://raamana.github.io/neuropredict/ News As of v0.6, neuropredict now supports regression applications i.e. predicting continuous t

Pradeep Reddy Raamana 93 Nov 29, 2022
Code for paper "A Critical Assessment of State-of-the-Art in Entity Alignment" (https://arxiv.org/abs/2010.16314)

A Critical Assessment of State-of-the-Art in Entity Alignment This repository contains the source code for the paper A Critical Assessment of State-of

Max Berrendorf 16 Oct 14, 2022
[CVPRW 2021] Code for Region-Adaptive Deformable Network for Image Quality Assessment

RADN [CVPRW 2021] Code for Region-Adaptive Deformable Network for Image Quality Assessment [Paper on arXiv] Overview Update [2021/5/7] add codes for W

IIGROUP 53 Dec 28, 2022
Repo for 2021 SDD assessment task 2, by Felix, Anna, and James.

SoftwareTask2 Repo for 2021 SDD assessment task 2, by Felix, Anna, and James. File/folder structure: helloworld.py - demonstrates various map backgrou

null 3 Dec 13, 2022
NIMA: Neural IMage Assessment

PyTorch NIMA: Neural IMage Assessment PyTorch implementation of Neural IMage Assessment by Hossein Talebi and Peyman Milanfar. You can learn more from

Kyryl Truskovskyi 293 Dec 30, 2022
A PyTorch Implementation of Neural IMage Assessment

NIMA: Neural IMage Assessment This is a PyTorch implementation of the paper NIMA: Neural IMage Assessment (accepted at IEEE Transactions on Image Proc

yunxiaos 418 Dec 29, 2022
Official PyTorch implementation of the paper "Recycling Discriminator: Towards Opinion-Unaware Image Quality Assessment Using Wasserstein GAN", accepted to ACM MM 2021 BNI Track.

RecycleD Official PyTorch implementation of the paper "Recycling Discriminator: Towards Opinion-Unaware Image Quality Assessment Using Wasserstein GAN

Yunan Zhu 23 Nov 5, 2022
[ICCV 2021] Group-aware Contrastive Regression for Action Quality Assessment

CoRe Created by Xumin Yu*, Yongming Rao*, Wenliang Zhao, Jiwen Lu, Jie Zhou This is the PyTorch implementation for ICCV paper Group-aware Contrastive

Xumin Yu 31 Dec 24, 2022