Sign Language Transformers (CVPR'20)
This repo contains the training and evaluation code for the paper Sign Language Transformers: Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation.
This code is based on Joey NMT but modified to realize joint continuous sign language recognition and translation. For text-to-text translation experiments, you can use the original Joey NMT framework.
Requirements
-
Download the feature files using the
data/download.shscript. -
[Optional] Create a conda or python virtual environment.
-
Install required packages using the
requirements.txtfile.pip install -r requirements.txt
Usage
python -m signjoey train configs/sign.yaml
! Note that the default data directory is ./data. If you download them to somewhere else, you need to update the data_path parameters in your config file.
ToDo:
- Initial code release.
- Release image features for Phoenix2014T.
- Share extensive qualitative and quantitative results & config files to generate them.
- (Nice to have) - Guide to set up conda environment and docker image.
Reference
Please cite the paper below if you use this code in your research:
@inproceedings{camgoz2020sign,
author = {Necati Cihan Camgoz and Oscar Koller and Simon Hadfield and Richard Bowden},
title = {Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation},
booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2020}
}
Acknowledgements
This work was funded by the SNSF Sinergia project "Scalable Multimodal Sign Language Technology for Sign Language Learning and Assessment" (SMILE) grant agreement number CRSII2 160811 and the European Union’s Horizon2020 research and innovation programme under grant agreement no. 762021 (Content4All). This work reflects only the author’s view and the Commission is not responsible for any use that may be made of the information it contains. We would also like to thank NVIDIA Corporation for their GPU grant.
107 Dec 27, 2022
58 Dec 21, 2022
128 Dec 8, 2022
26 Oct 14, 2022
15 Aug 20, 2022
444 Jan 3, 2023
4 Mar 30, 2022
3 Feb 24, 2022
63 Nov 18, 2022
440 Jan 2, 2023
20 Dec 6, 2022
77.2k Jan 2, 2023
20 Feb 13, 2022
62 Dec 24, 2022
1.3k Dec 31, 2022
38 Dec 12, 2022
46 Jun 30, 2022
434 Dec 30, 2022