Thanks for your baseline! Here's a doubt: The competition rules say that: (a) to any individual, entity, or country prohibited by any applicable U.S. or non-U.S. export controls and trade sanctions; (b) to anyone on U.S. or non-U.S. government restricted parties lists; or (c) for any purpose prohibited by applicable export controls and trade sanctions, including nuclear, chemical or biological weapons, or missile technology applications without the required government authorizations. Could the Chinese team win the final award? What's the detailed prohibited list?

• Added setup.py so that pip install -e . works
• Added matplotlib requirement to requirements.txt, which is required for compute_metrics script
• Added some instructions for baseline #1

I have now run all 5 baselines & verified they work as expected on my environment, which I set up using pip install -e.

Increased the Pillow version because <8.3.2 contained security issues. Also changed the faiss version to the most recent on pypi; the one specified can no longer be found.

Successfully ran pip install -e .

If predictions have the same score, the current implementation of the metric will score them differently if they are submitted in a different order.

https://github.com/facebookresearch/isc2021/blob/dab82c0381a3198270a8fa3a8ab722c3eba81b58/vcd/vcd/metrics.py#L351

We have fixed this by grouping by score and updating precision and recall for a particular score threshold as a whole rather than for individual scores:

from itertools import groupby
...
def match_metric(gts: Collection[Match], predictions: Collection[Match]):
    r"""V2 metric:

    Computes the AP based on the VCSL approach for the
    calculation of Precision and Recall.

    AP = \sum_{i=1}^N P(i) ΔR(i)

    where, P(i) = sqrt(P_q * P_r) and R(i) = sqrt(R_q * R_r)
    calculated as in the VCSL.
    """

    predictions = sorted(predictions, key=lambda x: x.score, reverse=True)

    # Initialize video pairs and load their gt bboxs
    video_pairs = defaultdict(VideoPairEvaluator)
    for gt in gts:
        video_pairs[gt.pair_id].add_gt(gt)

    # Get the total gt length for each axis
    gt_total_lengths = {axis: 0 for axis in Axis}
    for _, v in video_pairs.items():
        for axis in Axis:
            gt_total_lengths[axis] += v.total_gt_length(axis)

    # Loop through the predictions
    recall = 0.0
    metric = 0.0
    intersections = {axis: 0 for axis in Axis}
    totals = {axis: 0 for axis in Axis}

    # Group predictions by score to break ties consistently
    for _, preds in groupby(predictions, key=lambda x: x.score):
        # Update precision and recall within a given group before updating metric
        for pred in preds:
            pair_id = pred.pair_id
            # Given a new prediction, we only need the differences in the intersection with
            # gt and total video length covered for both query and reference axes.
            intersection_deltas, total_deltas = video_pairs[pair_id].add_prediction(pred)

            recalls = {}
            precisions = {}
            for axis in Axis:
                # Accumulate the differences to the corresponding values
                intersections[axis] += intersection_deltas[axis]
                totals[axis] += total_deltas[axis]
                recalls[axis] = intersections[axis] / gt_total_lengths[axis]
                precisions[axis] = intersections[axis] / totals[axis]

        new_recall = sqrt(recalls[Axis.QUERY] * recalls[Axis.REF])
        precision = sqrt(precisions[Axis.QUERY] * precisions[Axis.REF])

        # Compute metric
        delta_recall = new_recall - recall
        metric += precision * delta_recall
        recall = new_recall

    return metric
...