Jury
Simple tool/toolkit for evaluating NLG (Natural Language Generation) offering various automated metrics. Jury offers a smooth and easy-to-use interface. It uses datasets for underlying metric computation, and hence adding custom metric is easy as adopting datasets.Metric.
Main advantages that Jury offers are:
- Easy to use for any NLG system.
- Calculate many metrics at once.
- Metrics calculations are handled concurrently to save processing time.
- It supports evaluating multiple predictions.
To see more, check the official Jury blog post.
Installation
Through pip,
pip install jury
or build from source,
git clone https://github.com/obss/jury.git
cd jury
python setup.py install
Usage
API Usage
It is only two lines of code to evaluate generated outputs.
from jury import Jury
jury = Jury()
# Microsoft translator translation for "Yurtta sulh, cihanda sulh." (16.07.2021)
predictions = ["Peace in the dormitory, peace in the world."]
references = ["Peace at home, peace in the world."]
scores = jury.evaluate(predictions, references)
Specify metrics you want to use on instantiation.
jury = Jury(metrics=["bleu", "meteor"])
scores = jury.evaluate(predictions, references)
CLI Usage
You can specify predictions file and references file paths and get the resulting scores. Each line should be paired in both files.
jury eval --predictions /path/to/predictions.txt --references /path/to/references.txt --reduce_fn max
If you want to specify metrics, and do not want to use default, specify it in config file (json) in metrics key.
{
"predictions": "/path/to/predictions.txt",
"references": "/path/to/references.txt",
"reduce_fn": "max",
"metrics": [
"bleu",
"meteor"
]
}
Then, you can call jury eval with config argument.
jury eval --config path/to/config.json
Custom Metrics
You can use custom metrics with inheriting jury.metrics.Metric, you can see current metrics on datasets/metrics. The code snippet below gives a brief explanation.
from jury.metrics import Metric
CustomMetric(Metric):
def compute(self, predictions, references):
pass
Contributing
PRs are welcomed as always :)
Installation
git clone https://github.com/obss/jury.git
cd jury
pip install -e .[develop]
Tests
To tests simply run.
python tests/run_tests.py
Code Style
To check code style,
python tests/run_code_style.py check
To format codebase,
python tests/run_code_style.py format
License
Licensed under the MIT License.
2.3k Jan 7, 2023
409 Oct 28, 2022
55 Nov 22, 2022
3 May 25, 2022
431 Dec 19, 2022
4 Jul 1, 2022
119 Oct 8, 2022
652 Jan 6, 2023
132 Nov 25, 2022
12 Nov 11, 2022
1.2k Dec 18, 2022
125 Dec 20, 2022
2 Jan 17, 2022
12.3k Dec 31, 2022