Overview of changes:

• Created vis folder
• Added PCA visualization of random dataset samples embedded
• Extended above with spherical coordinates
• Copied carp_cloob config (maybe unnecessary?)

We're conducting experiments in using an MLM objective to improve data efficiency.

This PR will include:

A new data pipeline API for collating and processing input data. A new learning rate scheduler API for custom LR schedulers that are model state aware. CARP MLM, which optimizes MLM for a fixed number of epochs before switching to a different CARP objective. MixedMLMEncoder, which will start as an MLM encoder and switch to some other encoder after a fixed number of epochs.

We need to be able to split the contrastive parallel batch between GPUs for the multiGPU setting rather than using DeepSpeed's naive data parallelism.

In particular, using vanilla CARP as reference,

The passage encoder contrastive parallel

https://github.com/EleutherAI/magiCARP/blob/f4f880f7ede3420226d3e41267c1c41d48264a2e/carp/pytorch/model/architectures/carp.py#L61

and the review encoder contrastive parallel

https://github.com/EleutherAI/magiCARP/blob/f4f880f7ede3420226d3e41267c1c41d48264a2e/carp/pytorch/model/architectures/carp.py#L73

can be split between GPUs by sending a subset of the microbatches to its own GPU.

Similarly, the review and passage encoder do not need to be computed in serial- as long as we gather in parallel.

Currently we cannot scale CLOOB past declutr-base for unknown reasons. Current hypothesis is that we need to pretrain or fine-tune a language model using a contrastive loss. Kharr and I are exploring using SimCTG as a method to fine-tune a base model before training a CARP CLOOB model.

I added makefile for code styling and static analysis. You can style code by executing 'style' in the makefile. It would be great to style code before pushing code ;)

Two fixes

Context Length Fix:

• Any trainer inheriting from BaseTrainer was calling create_tok, a method that truncates all strings to only look at last n_ctx (generally 512) characters, as opposed to last n_ctx tokens
• BaseEncoder was fixed previously to implicitly truncate when required
• create_tok is now just lowering data quality by cutting all inputted strings short
• This bug likely effected many training results

Vicreg Trainer:

• Previous KeyError bug in BaseTrainer was patched in BaseTrainer but not in other trainers
• Changed a key in vicreg trainer to reflect the patch to BaseTrainer

Had an absolute value so the cosine similarity would always be positive, removed that...

Also, don't know why normalize=False is the default, I think normalize=True is a better default. If you are not normalized, you need it to be, and if it is already normalized to a unit vector, it shouldn't do anything, right? so normalize=True sounds like a better default to me...

In several scripts, string length was being used instead of number of tokens for context truncation. I have not checked the entire codebase for more instances of this issue (these may exist) but have resolved the cases within the example scripts. Additionally, the tokenizer in the BaseEncoder has been updated to have automatic truncation to model context length. A one-off bug within data pipeline that seems to be result of old code being left in has also been resolved.

Added example scripts that can be used to generate passage or review encodings en masse, with indexes of which dataset items were encoded. This is useful for many things so I think it should be incorporated into main branch.

Useful for uni-encoder experiments.

Current results are indecisive. Merging this into main so that we can use it for a multitude of future experiments.

See a declutr-base run here https://wandb.ai/eleutherai/magicarp-carp_pytorch_training/runs/1z41hm0i?workspace=user-louiscastricato

Namely:

1. Linear head classification task on embeddings. Test accuracy from trained linear head.
2. WANDB ready plot of UMAP'd embeddings to see latent space.

Made error of not creating a default value for "force_fresh". This means that running script without command line argument of "FRESH" resulted in no value being set, causing a runtime error. There was effectively no way to have a non fresh run unless a redundant argument was passed.

not sure what the etiology is here, but figured we should document it and maybe centralize repair discussion here since it's apparently a known issue. I think with CARPFilip we first observed this when gradient checkpointing was turned off, prior to which biases had feasible gradient distributions but weights gradients were already reporting as collapsed to zero (or more likely, had exploded and the nan's were interpreted as zero by wandb or the gradient checkpointing mechanism).

alternatively, we could move everything to config files and use command line for overriding config entries. Requiring that certain arguments be provided via the command line is confusing, at least for me.