Optimization of a gesture representation network for Sign Language analysis

Valentin Belissen (1, 2, 3) , Michèle Gouiffes (3) , Annelies Braffort (3)

1 Université Paris-Saclay
2 UP11 - Université Paris-Sud - Paris 11
3 LIMSI - Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur

This paper presents the manufacturing and optimization of a convolutional-recurrent neural network, in order to jointly learn the detection of numerous Sign Language linguistic features in ordinary RGB videos. The proposed architecture can learn generic temporal-gestural features from a compact representation of people producing continuous Sign Language. These generic features make it possible to detect both lexical signs and higher-level linguistic patterns simultaneously. New pattern types can be added to the model and accurately detected without retraining the gestural features, that is with few training instances. The network is trained and tested on a continuous dialog corpus of French Sign Language. It gets localized F1-scores up to 80%, depending on the optimization of the network architecture.

Mots clés: Sign Language; Computer Vision; Sign Language Recognition; Sign Language Linguistics

Domaines: Intelligence artificielle [cs.AI]; Vision par ordinateur et reconnaissance de formes [cs.CV]; Traitement des images [eess.IV]; Informatique [cs]

Valentin Belissen, Michèle Gouiffes, Annelies Braffort. Optimization of a gesture representation network for Sign Language analysis. 2019. ⟨hal-02146369⟩