A PyTorch-based library for fast prototyping and sharing of deep neural network models.

We need to restructure the documentation.

• Intro (feature overview: supported tasks and commands)
• Getting Started (big classy tutorial):
  • Installation (maybe move before "getting started"? right after Intro)
  • Basic (everything that does not require touching code):
    • Intro (explanation of raw dataset and end goal)
    • Formatting the data (convert data to tsv / jsonl)
    • Choosing a profile (high level - the profile list goes in the reference manual)
    • Train (quick overview of how to train from the CLI with classy train - no frills)
    • Inference (quick overview of inference commands + ref to CLI sections)
  • Customizing things
    • Template (we introduce the template as the go-to way to work with classy)
    • Changing profile
    • Custom data format
    • Custom model
    • [...] all other customizations
• Reference manual:
  • CLI (detailed description) of classy commands
    • train
    • predict
    • other inference commands (evaluate, serve, demo)
    • up/download
    • describe (maybe as first?)
  • Tasks and input formats (joined together, for each task we show both formats - currently they're separated)
  • Profiles
  • Mixins? (what should we say about them?)

Hi, I'm always frustrated when fine-tuning the checkpoint. There has no access to it, let alone continuing to train the SOTA model by a different loss or use it as a submodule. I would be much grateful if you can provide this function.

Hello, Using the current token annotation format, I can't set the boundaries of the following entities (they all have the same label):

name personal_data address personal_data id card number personal_data

...in the following example please tell me your name address and identity card number

The resulting labeled sentence would be O O O O PERSONAL_DATA PERSONAL_DATA O PERSONAL_DATA PERSONAL_DATA PERSONAL_DATA

With this format, it is impossible to know if name and address are one or two different entities. Did you consider this issue?

Thanks!

Describe the bug If you try to start a training modifying from the command line a parameter that has been introduced (i.e. was not in the default profile for the task you are doing) in a custom profile the process crashes.

To Reproduce Create a custom model (CustomModel) with a custom parameter (custom_parameter). Then, create your own profile for the task (custom_profile). The model section of your profile will be something like this:

model:
  _target_: my_project.my_models.CustomModel
  transformer_model: bert-base-cased
  custom_parameter: custom_value

Then run from the command line:

classy train my_task my_data -n my_model --profile custom_profile -c model.custom_parameter=new_value

Expected behaviour The model should start with the model.custom_parameter set to the new value (new_value).

Actual behaviour The training process crashes with the following hydra error:

Could not override 'model.custom_parameter'.
To append to your config use +model.custom_parameter=new_value
Key 'custom_parameter' is not in struct
    full_key: model.custom_parameter
    object_type=dict

Desktop (please complete the following information):

• OS: Ubuntu Server 20.04
• PyTorch Version: 1.9.0
• Classy Version: 0.1.0

updates:

• github.com/pre-commit/pre-commit-hooks: v2.3.0 → v4.1.0
• github.com/psf/black: 21.12b0 → 22.3.0
• github.com/asottile/blacken-docs: v1.12.0 → v1.12.1

Is your feature request related to a problem? Please describe. How do we easily move specific runs across machines (e.g., train on a docker instance, run a demo on local machine)? At the moment, this has to be done manually, which is painful considering that we have to include everything under a certain folder and then we need to recreate the same structure in the target machine.

Describe the solution you'd like classy export <experiment> should produce a zip/tgz containing everything necessary to move experiments (or single runs) across machines. We sort of do this already with classy upload / download, but just rely on the hub. Instead of uploading, we can just create the zip in the local folder (or a path specified by the user).

Then, we should be able to run classy import <classy-exported-file.zip> to find our experiment in our local folder (better yet, every classy inference command could work with the export! :) )

Custom profiles are only recognized if their extension is yaml (and the error message was completely unrelated and impossible to debug easily).

We should allow profiles to end in yml as well :)

Fixes the bug: The trainer parameters were not loaded correctly when the model was loaded from a checkpoint. E.g. val_check_interval was not set, so validation callbacks were not called rightly.

Describe the bug The trainer parameters were not loaded correctly when the model was loaded from a checkpoint. E.g. val_check_interval was not set, so validation callbacks were not called rightly.

To Reproduce Steps to reproduce the behaviour:

1. Train a model for x steps
2. Stop the training
3. Restore the training from the checkpoint using resume_from parameter

Expected behaviour The trainer parameters should be preserved. E.g if val_check_interval is set, the validation callbacks should be called accordingly.

Actual behaviour The validation callbacks are not called and val_check_interval is set to 0.

Desktop (please complete the following information):

• OS: Ubuntu 20.04
• PyTorch Version: 1.11

Describe the bug Title. In classy/data/dataset/hf/classification.py#L89 we invoke self.tokenize (#L109) which correctly truncates the input.

The issue arises due to tuple(tok_encoding.word_to_tokens(wi)) for wi in range(len(tokens)): when a token is not included in the input due to truncation, word_to_tokens returns None, and tuple(None) raises a TypeError, which triggers the catch condition and makes the function return None, which cannot be unpacked in input_ids, token_offsets = self.tokenize(token_sample.tokens), resulting in another unhandled exception that finally crashes classy.

To Reproduce In the token classification setting, input a sentence that has too many tokens (or reduce truncation to obtain the same effect).

Expected behaviour I think there is a way to know how many of the original tokens were kept, and we can iterate over them instead of len(tokens), otherwise we can just iterate until word_to_tokens(wi) is not None. Comments?

Should be pretty easy to implement: we could employ two methods / properties / classmethods for Evaluation, one for a white list of keys to log and the other for a black list, with the white list taking precedence over the black list (so if the user specifies both, we first get the subset of the white list and then remove from them the keys in the black list). By default, everything is logged to pb (as it is now).

Is your feature request related to a problem? Please describe. Huggingface Transformers has several task-pretrained models in its hub (e.g. facebook/bart-large-xsum). classy automatically supports using them at training time for further fine-tuning, but doesn't support using them directly at inference time. The current way to go about it is essentially weight porting.

Rather, classy should support something like classy demo ... facebook/bart-large-xsum ...

Describe the solution you'd like Personally, I'd like a solution that's symmetric with classy train. So, if you train bart with classy train generation data/xsum --profile bart-large -c transformer_model=facebook/bart-large-xsum, you could do inference tasks such as classy demo with something like classy demo "generation data/xsum --profile bart-large -c transformer_model=facebook/bart-large-xsum".

I am unsure whether this would be friendly (same syntax) or unfriendly (overkill syntax), and suggestions/help are welcome.