• Add a graffiti_guess column to the database
• Estimate the confusion matrix for each client by comparing classifier results against the graffiti guess
• When running the API server, disable the (slow) classifier load if the DISABLE_CLASSIFIER environment variable is set

I think this is quite prone to overfitting, as in my most recent training data set there are very few misclassifications, even for Nimbus and Lodestar (very suspicious)

curl "http://localhost:8000/confusion/Lighthouse/0/10000000"
{"true_pos": 21925, "true_neg": 7931, "false_pos": 16, "false_neg": 0}                                                                                                                                                         

curl "http://localhost:8000/confusion/Prysm/0/10000000"
{"true_pos": 279, "true_neg": 29584, "false_pos": 0, "false_neg": 9}

curl "http://localhost:8000/confusion/Teku/0/10000000"
{"true_pos": 6684, "true_neg": 23188, "false_pos": 0, "false_neg": 0}

curl "http://localhost:8000/confusion/Nimbus/0/10000000"
{"true_pos": 928, "true_neg": 28944, "false_pos": 0, "false_neg": 0}

curl "http://localhost:8000/confusion/Lodestar/0/10000000"
{"true_pos": 40, "true_neg": 29825, "false_pos": 0, "false_neg": 7}

It would be cool to have features named and capable of being toggled on and off from the CLI.

It would also be nice to include feature names on the matplotlib graphs, with a 2D cross-section for each pair of features.

The version of click in black is unpinned. A recent click upgrade caused widespread frustration in the world of Python. This PR pins click. For more context: https://github.com/dask/distributed/issues/6013

Fixes the calculation of gaps in the presence of re-orgs and missing blocks.

Blockprint was getting stuck when block n - 2 was missing, which was parent to both n - 1 and n. The gap calculator was calculating an invalid gap with bounds [n, n - 1].

The new calculator takes care to fetch the oldest prior block with slot less than the missing parent n - 2, which yields the correct gap [n - 2, n - 1].

Add a new /classify/no_store endpoint that classifies block rewards and returns their classification without storing anything in the database. I am using this in blockgauge.

This PR adds an API endpoint at validators/blocks/latest. This endpoint returns a list of each validator, their most recently proposed block and the best_guess_single for that block.

Usage examples can be seen in docs/validator_api.md

This PR improves the cross validation score to around 0.95 by computing new features based on the ordering of AttestationData within blocks. Previously we relied entirely on the ordering of attestation rewards, but several clients seem to order based on attestation slot instead.

From now on, to use blockprint with these new difflib features will require a version of Lighthouse containing this commit: https://github.com/sigp/lighthouse/commit/53b2b500dbb9f00b5d6dd4da065e52d7ba186f9a (at time of writing, no release includes this commit but it will be in v2.3.2/v2.4.0).

Finally, the graffiti regex were also updated to support recent versions of RocketPool which include an extra letter like RP-NL for Nethermind-Lighthouse.

Blockprint backfill is stuck on this block with 0 attestations: https://beaconcha.in/block/3952526

This PR unblocks it by not panicking in statistics.median on the 0-length array.

Add an option to the KNN classifier to exclude some clients from the classifier itself, instead classifying them based only on their graffiti matches.

• Add interactive Jupyter notebook for exploring plots of the feature space more easily
• Add several new features with better performance than the previous set:
  • percent_redundant_boost: the same as percent_redundant but adds +0.2 to any non-zero measurement to create separation between the blocks with no-redundancy and the blocks with redundancy
  • mean_density: same as median_density but using the mean to allow the measurement to be influenced by outliers
  • difflib_sorted_distance: based on the same idea as percent_pairwise_ordered but taking into account more of the ordering characteristics. Python's difflib is part of the standard library and provides a metric for sequence similarity in the range 0-1.
  • spearman_correlation: similar to difflib_sorted_distance, calculates Spearman's rank correlation coefficient for the list of per attestation rewards relative to its sorted version.
• Updates the default feature set to [percent_redundant_boost, percent_pairwise_ordered, spearman_correlation, mean_density]. This feature set performed the best during cross validation on slots 3481601 to 3702784, with the training data balanced using balance.py to a max-imbalance of 2x. This improved CV performance also carried over into improved performance when classifying the most recent portion of the chain, with a minimum sensitivity of 0.80 (Prysm) and a min precision of 0.62 (Prysm). See run 13 in the spreadsheet here: https://docs.google.com/spreadsheets/d/1ly8dT8ngWeU0KQFm3Ec_SKcyxs_jzEGopQYRCGt8NXc.

• Add graffiti regex for Lodestar
• Add a proper arg parser to prepare_training_data. This changes the syntax for the parallel workers flag from a positional arg to an optional --num-workers flag
• Add a --disable flag to prepare_training_data that causes training data not to be produced for the listed clients
• Add a --disable flag to knn_classifier that ignores training data for the listed clients

I don't think we urgently need a --disable flag on build_db because it can be used with training data that simply omits certain clients, and that will be handled gracefully.

Currently blockprint's API will return 0 values when querying a section of the database for which history is missing. This can occur if the beacon node backing blockprint has been falling in and out of sync or is re-syncing and lacks a portion of history.

At the moment these gaps can be detected via the /sync/gaps API, but this is error-prone and not obvious. It would be better if blockprint detected queries affected by the gaps and failed them explicitly so that the user doesn't have to check /sync/gaps manually. E.g. if there is a gap from slot 3623682 to 3687456 (as there is currently) then blockprint should 404 on queries like /blocks_per_client/113625/113850 rather than returning a response with 0 blocks per client for all clients.

The reason the API is the way it is currently is that the SQL query to detect the gaps takes several seconds to evaluate, because it scans the entire table. If this query can be made to run quickly, then it will be feasible to run it for every query.

Some graffiti regex should only be enabled on e.g. Prater. The ./prepare_training_data.py script should take a --prater or --mainnet flag to switch regex on/off, or a path to a file containing the regex.