A PyTorch-based open-source framework that provides methods for improving the weakly annotated data and allows researchers to efficiently develop and compare their own methods.

Overview

Python Version license GitHub Release build status PyPI codecov

Knodle (Knowledge-supervised Deep Learning Framework) - a new framework for weak supervision with neural networks. It provides a modularization for separating weak data annotations, powerful deep learning models, and methods for improving weakly supervised training.

More details about Knodle are in our recent paper.


Latest news

Installation

pip install knodle

Usage

knodle offers various methods for denoising weak supervision sources and improve them. There are several methods available for denoising. Examples can be seen in the tutorials folder.

There are four mandatory inputs for knodle:

  1. model_input_x: Your model features (e.g. TF-IDF values) without any labels. Shape: (n_instances x features)
  2. mapping_rules_labels_t: This matrix maps all weak rules to a label. Shape: (n_rules x n_classes)
  3. rule_matches_z: This matrix shows all applied rules on your dataset. Shape: (n_instances x n_rules)
  4. model: A PyTorch model which can take your provided model_input_x as input. Examples are in the model folder.

If you know which denoising method you want to use, you can directly call the corresponding module (the list of currently supported methods is provided below).

Example for training the baseline classifier:

from knodle.model.logistic_regression_model import LogisticRegressionModel
from knodle.trainer.baseline.majority import MajorityVoteTrainer

NUM_OUTPUT_CLASSES = 2

model = LogisticRegressionModel(model_input_x.shape[1], NUM_OUTPUT_CLASSES)

trainer = MajorityVoteTrainer(
  model=model,
  mapping_rules_labels_t=mapping_rules_labels_t,
  model_input_x=model_input_x,
  rule_matches_z=rule_matches_z,
  dev_model_input_x=X_dev,
  dev_gold_labels_y=Y_dev
)

trainer.train()

trainer.test(X_test, Y_test)

A more detailed example of classifier training is here.

Main Principles

The framework provides a simple tensor-driven abstraction based on PyTorch allowing researchers to efficiently develop and compare their methods. The emergence of machine learning software frameworks is the biggest enabler for the wide spread adoption of machine learning and its speed of development. With Knodle we want to empower researchers in a similar fashion.

Knodle main goals:

  • Data abstraction. The interface is a tensor-driven data abstraction which unifies a large number of input variants and is applicable to a large number of tasks.
  • Method independence. We distinguish between weak supervision and prediction model. This enables comparability and accounts for a domain-specific inductive biases.
  • Accessibility. There is a high-level access to the library, that makes it easy to test existing methods, incorporate new ones and benchmark them against each other.

Apart from that, Knodle includes a selection of well-known data sets from prior work in weak supervision. Knodle ecosystem provides modular access to datasets and denoising methods (that can, in turn, be combined with arbitrary deep learning models), enabling easy experimentation.

Datasets currently provided in Knodle:

  • Spam Dataset - a dataset, based on the YouTube comments dataset from Alberto et al. (2015). Here, the task is to classify whether a text is relevant to the video or holds spam, such as adver- tisement.
  • Spouse Dataset - relation extraction dataset is based on the Signal Media One-Million News Articles Dataset from Corney et al. (2016).
  • IMDb Dataset - a dataset, that consists of short movie reviews. The task is to determine whether a review holds a positive or negative sentiment.
  • TAC-based Relation Extraction Dataset - a dataset built over Knowledge Base Population challenges in the Text Analysis Conference. For development and test purposes the corpus annotated via crowdsourcing and human labeling from KBP is used (Zhang et al. (2017). The training is done on a weakly-supervised noisy dataset based on TAC KBP corpora (Surdeanu (2013)).

All datasets are added to the Knodle framework in the tensor format described above and could be dowloaded here. To see how the datasets were created please have a look at the dedicated tutorial.

Denoising Methods

There are several denoising methods available.

Trainer Name Module Description
MajorityVoteTrainer knodle.trainer.baseline This builds the baseline for all methods. No denoising takes place. The final label will be decided by using a simple majority vote approach and the provided model will be trained with these labels.
AutoTrainer knodle.trainer This incorporates all denoising methods currently provided in Knodle.
KNNAggregationTrainer knodle.trainer.knn_aggregation This method looks at the similarities in sentence values. The intuition behind it is that similar samples should be activated by the same rules which is allowed by a smoothness assumption on the target space. Similar sentences will receive the same label matches of the rules. This counteracts the problem of missing rules for certain labels.
WSCrossWeighTrainer knodle.trainer.wscrossweigh This method weighs the training samples basing on how reliable their labels are. The less reliable sentences (i.e. sentences, whose weak labels are possibly wrong) are detected using a DS-CrossWeigh method, which is similar to k-fold cross-validation, and got reduced weights in further training. This counteracts the problem of wrongly classified sentences.
SnorkelTrainer knodle.trainer.snorkel A wrapper of the Snorkel system, which incorporates both generative and discriminative Snorkel steps in a single call.

Each of the methods has its own default config file, which will be used in training if no custom config is provided.

Details about negative samples

Tutorials

We also aimed at providing the users with basic tutorials that would explain how to use our framework. All of them are stored in examples folder and logically divided into two groups:

  • tutorials that demonstrate how to prepare the input data for Knodle Framework...
    • ... on the example of a well-known ImdB dataset. A weakly supervised dataset is created by incorporating keywords as weak sources (link).
    • ... on the example of a TAC-based dataset in .conll format. A relation extraction dataset is created using entity pairs from Freebase as weak sources (link).
  • tutorials how to work with Knodle Framework...
    • ... on the example of AutoTrainer. This trainer is to be called when user wants to train a weak classifier, but has no intention to use any specific denoising method, but rather try all currently provided in Knodle (link).
    • ... on the example of WSCrossWeighTrainer. With this trainer a weak classifier with WSCrossWeigh denoising method will be trained (link).

Compatibility

Currently the package is tested on Python 3.7. It is possible to add further versions. The CI/CD pipeline needs to be updated in that case.

Structure

The structure of the code is as follows

knodle
โ”œโ”€โ”€ knodle
โ”‚    โ”œโ”€โ”€ evaluation
โ”‚    โ”œโ”€โ”€ model
โ”‚    โ”œโ”€โ”€ trainer
โ”‚          โ”œโ”€โ”€ baseline
โ”‚          โ”œโ”€โ”€ knn_aggregation
โ”‚          โ”œโ”€โ”€ snorkel
โ”‚          โ”œโ”€โ”€ wscrossweigh
โ”‚          โ””โ”€โ”€ utils
โ”‚    โ”œโ”€โ”€ transformation
โ”‚    โ””โ”€โ”€ utils
โ”œโ”€โ”€ tests
โ”‚    โ”œโ”€โ”€ data
โ”‚    โ”œโ”€โ”€ evaluation
โ”‚    โ”œโ”€โ”€ trainer
โ”‚          โ”œโ”€โ”€ baseline
โ”‚          โ”œโ”€โ”€ wscrossweigh
โ”‚          โ”œโ”€โ”€ snorkel
โ”‚          โ””โ”€โ”€ utils
โ”‚    โ””โ”€โ”€ transformation
โ””โ”€โ”€ examples
     โ”œโ”€โ”€ data_preprocessing
           โ”œโ”€โ”€ imdb_dataset
           โ””โ”€โ”€ tac_based_dataset
     โ””โ”€โ”€ training
           โ”œโ”€โ”€ simple_auto_trainer
           โ””โ”€โ”€ wscrossweigh

License

Licensed under the Apache 2.0 License.

Contact

If you notices a problem in the code, you can report it by submitting an issue.

If you want to share your feedback with us or take part in the project, contact us via [email protected].

And don't forget to follow @knodle_ai on Twitter :)

Authors

Citation

@misc{sedova2021knodle,
      title={Knodle: Modular Weakly Supervised Learning with PyTorch}, 
      author={Anastasiia Sedova, Andreas Stephan, Marina Speranskaya, and Benjamin Roth},
      year={2021},
      eprint={2104.11557},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

Acknowledgments

This research was funded by the WWTF though the project โ€œKnowledge-infused Deep Learning for Natural Language Processingโ€ (WWTF Vienna Research Group VRG19-008).

Comments
  • Feature/#297 mimic cxr dataset

    Feature/#297 mimic cxr dataset

    I will add some more comments to the jupiter notebook, this is still WIP. So far, I only got the X and T matrix, since I need some advice for the Z matrix.

    Linked issue: #297

    dataset student project 
    opened by marlenesteiner 5
  • Feature/#295 chexpert dataset

    Feature/#295 chexpert dataset

    I added the .py file and the Jupyter notebook for the preprocessing of CheXpert. Please let me know if you have any suggestions for improvement.

    Linked issue: #295

    dataset student project 
    opened by LenaZellinger 2
  • Evaluation method

    Evaluation method

    Implement an evaluate method

    
      def evaluate(
            self, eval_dataset: Optional[Dataset] = None, ignore_keys: Optional[List[str]] = None
        ) -> Dict[str, float]:
        pass
    
    opened by AlessandroVol23 2
  • Bump numpy from 1.21.4 to 1.22.0 in /examples/data_preprocessing/police_killing_dataset

    Bump numpy from 1.21.4 to 1.22.0 in /examples/data_preprocessing/police_killing_dataset

    Bumps numpy from 1.21.4 to 1.22.0.

    Release notes

    Sourced from numpy's releases.

    v1.22.0

    NumPy 1.22.0 Release Notes

    NumPy 1.22.0 is a big release featuring the work of 153 contributors spread over 609 pull requests. There have been many improvements, highlights are:

    • Annotations of the main namespace are essentially complete. Upstream is a moving target, so there will likely be further improvements, but the major work is done. This is probably the most user visible enhancement in this release.
    • A preliminary version of the proposed Array-API is provided. This is a step in creating a standard collection of functions that can be used across application such as CuPy and JAX.
    • NumPy now has a DLPack backend. DLPack provides a common interchange format for array (tensor) data.
    • New methods for quantile, percentile, and related functions. The new methods provide a complete set of the methods commonly found in the literature.
    • A new configurable allocator for use by downstream projects.

    These are in addition to the ongoing work to provide SIMD support for commonly used functions, improvements to F2PY, and better documentation.

    The Python versions supported in this release are 3.8-3.10, Python 3.7 has been dropped. Note that 32 bit wheels are only provided for Python 3.8 and 3.9 on Windows, all other wheels are 64 bits on account of Ubuntu, Fedora, and other Linux distributions dropping 32 bit support. All 64 bit wheels are also linked with 64 bit integer OpenBLAS, which should fix the occasional problems encountered by folks using truly huge arrays.

    Expired deprecations

    Deprecated numeric style dtype strings have been removed

    Using the strings "Bytes0", "Datetime64", "Str0", "Uint32", and "Uint64" as a dtype will now raise a TypeError.

    (gh-19539)

    Expired deprecations for loads, ndfromtxt, and mafromtxt in npyio

    numpy.loads was deprecated in v1.15, with the recommendation that users use pickle.loads instead. ndfromtxt and mafromtxt were both deprecated in v1.17 - users should use numpy.genfromtxt instead with the appropriate value for the usemask parameter.

    (gh-19615)

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    wontfix dependencies 
    opened by dependabot[bot] 1
  • [WIP] Feature/CheXpert and CheXbert labelers

    [WIP] Feature/CheXpert and CheXbert labelers

    I ran into some issues when trying to adjust the Z matrix & was not able to resolve them until now, therefore the code is unfinished. (This will definitely be resolved by the end of the holidays) Missing: pattern transformation, adjusting Z matrix depending on neg/unc patterns; improving code & file architecture

    The build fails at the moment, but if I understand it correctly, this happens, because I use a file where all constants are defined and always just use import * from constants at the beginning of each file.

    enhancement student project 
    opened by elisabear 5
  • First try, PoC PyTorchLightning

    First try, PoC PyTorchLightning

    Goal: Try to use PyTorchLightning

    Subgoals:

    • Build MajorityTrainer, SnorkelTrainer
    • Train with TfIDf + Log. regression and Huggingface Trafo DistilBert
    • Use abstractions.

    Still very problematic:

    • Data Container. How to abstract properly from input (e.g. tfidf vs ids + attention mask)
    • Test Step for Transformers
    hackathon optimization 
    opened by AndSt 0
  • Feature/#316 cosine

    Feature/#316 cosine

    The COSINE trainer is added.

    How to use it?

    • knodle\examples\trainer\cosine\cosine_training_example.py contains an example code for training and testing cosine on the TREC dataset. The TREC dataset is downloaded and preprocessed in Knodle dataset format.
    • All cosine related code components can be found in knodle\knodle\trainer\cosine
    • After the training is completed (i.e. after trainer.train()), the best model during training obtained via early stopping is stored in trainer.model.

    Remarks

    • training with COSINE on the TREC dataset (via weak supervision) is tested.
    • The function convert_text_to_transformer_input() in knodle\examples\trainer\preprocessing.py is updated, now it takes an additional parameter max_sen_len
    enhancement hackathon 
    opened by davidpig 1
Releases(v0.1.3)
A PyTorch implementation of Sharpness-Aware Minimization for Efficiently Improving Generalization

sam.pytorch A PyTorch implementation of Sharpness-Aware Minimization for Efficiently Improving Generalization ( Foret+2020) Paper, Official implementa

Ryuichiro Hataya 102 Dec 28, 2022
Sharpness-Aware Minimization for Efficiently Improving Generalization

Sharpness-Aware-Minimization-TensorFlow This repository provides a minimal implementation of sharpness-aware minimization (SAM) (Sharpness-Aware Minim

Sayak Paul 54 Dec 8, 2022
Weakly Supervised Dense Event Captioning in Videos, i.e. generating multiple sentence descriptions for a video in a weakly-supervised manner.

WSDEC This is the official repo for our NeurIPS paper Weakly Supervised Dense Event Captioning in Videos. Description Repo directories ./: global conf

Melon(Xuguang Duan) 96 Nov 1, 2022
Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. It can use GPUs and perform efficient symbolic differentiation.

============================================================================================================ `MILA will stop developing Theano <https:

null 9.6k Dec 31, 2022
Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. It can use GPUs and perform efficient symbolic differentiation.

============================================================================================================ `MILA will stop developing Theano <https:

null 9.6k Jan 6, 2023
Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. It can use GPUs and perform efficient symbolic differentiation.

============================================================================================================ `MILA will stop developing Theano <https:

null 9.3k Feb 12, 2021
The official implementation of CVPR 2021 Paper: Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation.

Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation This repository is the official implementation of CVPR 2021 paper:

null 9 Nov 14, 2022
BisQue is a web-based platform designed to provide researchers with organizational and quantitative analysis tools for 5D image data. Users can extend BisQue by implementing containerized ML workflows.

Overview BisQue is a web-based platform specifically designed to provide researchers with organizational and quantitative analysis tools for up to 5D

Vision Research Lab @ UCSB 26 Nov 29, 2022
Code for Private Recommender Systems: How Can Users Build Their Own Fair Recommender Systems without Log Data? (SDM 2022)

Private Recommender Systems: How Can Users Build Their Own Fair Recommender Systems without Log Data? (SDM 2022) We consider how a user of a web servi

joisino 20 Aug 21, 2022
Using this codebase as a tool for my own research. Making some modifications to the original repo for my own purposes.

For SwapNet Create a list.txt file containing all the images to process. This can be done with the GNU find command: find path/to/input/folder -name '

Andrew Jong 2 Nov 10, 2021
null 113 Nov 28, 2022
Annotated, understandable, and visually interpretable PyTorch implementations of: VAE, BIRVAE, NSGAN, MMGAN, WGAN, WGANGP, LSGAN, DRAGAN, BEGAN, RaGAN, InfoGAN, fGAN, FisherGAN

Overview PyTorch 0.4.1 | Python 3.6.5 Annotated implementations with comparative introductions for minimax, non-saturating, wasserstein, wasserstein g

Shayne O'Brien 471 Dec 16, 2022
This YoloV5 based model is fit to detect people and different types of land vehicles, and displaying their density on a fitted map, according to their coordinates and detected labels.

This YoloV5 based model is fit to detect people and different types of land vehicles, and displaying their density on a fitted map, according to their

Liron Bdolah 8 May 22, 2022
PaddleRobotics is an open-source algorithm library for robots based on Paddle, including open-source parts such as human-robot interaction, complex motion control, environment perception, SLAM positioning, and navigation.

็ฎ€ไฝ“ไธญๆ–‡ | English PaddleRobotics paddleRoboticsๆ˜ฏๅŸบไบŽpaddle็š„ๆœบๅ™จไบบๅผ€ๆบ็ฎ—ๆณ•ๅบ“้›†๏ผŒๅŒ…ๆ‹ฌไบบๆœบไบคไบ’ใ€ๅคๆ‚่ฟๅŠจๆŽงๅˆถใ€็Žฏๅขƒๆ„Ÿ็Ÿฅใ€slamๅฎšไฝๅฏผ่ˆช็ญ‰ๅผ€ๆบ็ฎ—ๆณ•้ƒจๅˆ†ใ€‚ ไบบๆœบไบคไบ’ ไธปๅŠจๅคšๆจกไบคไบ’ๆŠ€ๆœฏTFVT-HRI ไธปๅŠจๅคšๆจกไบคไบ’ๆŠ€ๆœฏๆ˜ฏ้€š่ฟ‡่ง†่ง‰ใ€่ฏญ้Ÿณใ€่งฆๆ‘ธไผ ๆ„Ÿๅ™จ็ญ‰่พ“ๅ…ฅๆœบๅ™จไบบ

null 185 Dec 26, 2022
Compare outputs between layers written in Tensorflow and layers written in Pytorch

Compare outputs of Wasserstein GANs between TensorFlow vs Pytorch This is our testing module for the implementation of improved WGAN in Pytorch Prereq

Hung Nguyen 72 Dec 20, 2022
Pytorch Implementation for CVPR2018 Paper: Learning to Compare: Relation Network for Few-Shot Learning

LearningToCompare Pytorch Implementation for Paper: Learning to Compare: Relation Network for Few-Shot Learning Howto download mini-imagenet and make

Jackie Loong 246 Dec 19, 2022
A lightweight library to compare different PyTorch implementations of the same network architecture.

TorchBug is a lightweight library designed to compare two PyTorch implementations of the same network architecture. It allows you to count, and compar

Arjun Krishnakumar 5 Jan 2, 2023
A GPU-optional modular synthesizer in pytorch, 16200x faster than realtime, for audio ML researchers.

torchsynth The fastest synth in the universe. Introduction torchsynth is based upon traditional modular synthesis written in pytorch. It is GPU-option

torchsynth 229 Jan 2, 2023