JP6306899B2 - Non-finger movement detection device and program - Google Patents

Description

  The present invention relates to a non-finger movement detection device and a program.

  Research on automatic translation from Japanese into sign language is ongoing with the aim of expanding sign language services. In sign language, linguistic information is expressed by finger movements such as hand and finger movements as well as non-finger movements such as facial expressions and head movements. Translation from text to sign language is performed based on a bilingual corpus of Japanese sentences and sign language sentences. In general, the sign language sentence mainly describes finger movements, and only a part of non-hand movements is transcribed. This is because a finger movement is easy to identify a word, but a non-finger movement is difficult. Non-finger movements also have the role of linguistic and spoken language prosody, and it is not easy for humans to write with distinction.

  Conventionally, as a technique for detecting a non-finger movement, there is a technique for automatically recognizing a facial expression using a face recognition technique and detecting the position as a non-finger movement (for example, Patent Documents 1 to 3).

JP 2014-021896 A JP 2014-021846 A JP 2011-150381 A

Social Welfare Corporation National Sign Language Training Center, Japanese Sign Language Research Institute, “New Japanese-Sign Language Dictionary”, All-Japan Samurai Federation, June 10, 2011

  However, in sign language, facial expressions also have the role of a spoken language prosody, and facial expressions may appear even in non-finger movements. Therefore, even if there is a facial expression, it does not necessarily have a linguistic meaning, and a non-hand movement may not be detected correctly.

  Accordingly, an object of the present invention made in view of such a point is to provide a non-hand movement detection device and a program for detecting a word to be non-hand movement based on a bilingual corpus of a Japanese sentence and a sign language sentence.

In order to solve the above-described problems, a non-finger movement detection device according to the present invention is a storage unit that stores a bilingual corpus including a Japanese sentence and a sign language sentence corresponding to the Japanese sentence, The storage unit is divided into Japanese words and stored in association with numbers indicating the word order, and the sign language sentence is divided into sign language words corresponding to finger movements, and stored in association with numbers indicating the word order , said Japanese words contained in Japanese sentences, a alignment unit for associating the said sign language word included in the sign language sentence, the matching character or string between the sign language words and the Japanese word, the based Japanese words and the sentence length of said sign language words comprises associating the said sign language word and the Japanese word, and the alignment portion, of the Japanese word, Do not associated with the sign language word From Japanese word comprises a non-finger movement detection unit for detecting the content words representing the lexical meaning as the Japanese word to be non-finger operation, the.

In addition, when the Japanese word composed of a plurality of kanji characters is translated into another language word that is equal to or more than the number of kanji characters of the Japanese word, the alignment unit is configured to include the plurality of kanji characters. Preferably, the word word is divided into kanji characters, and the kanji characters and the sign language words are further associated with each other based on the translation probabilities between the kanji characters and the sign language words.

  In order to solve the above problems, a program according to the present invention causes a computer to function as the non-finger movement detection device.

  According to the non-finger movement detection apparatus and program according to the present invention, it is possible to detect a word that should be non-finger movement based on a bilingual corpus of a Japanese sentence and a sign language sentence.

It is a figure which shows schematic structure of the non-finger movement detection apparatus which concerns on one Embodiment of this invention. It is a flowchart which shows the process of a non-finger movement detection apparatus.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Here, there is no established notation for sign language, but in this paper, Japanese labels that are commonly used notation are used (see Non-Patent Document 1). In a Japanese label, a sign language word is described by enclosing a Japanese word semantically close to a sign language word in parentheses “{}”, such as “{rain}”. Japanese labels have the advantage that they are easy to handle in machine translation because they are written for each word. An example of a bilingual corpus of Japanese sentences and sign language sentences (Japanese labels) is expressed as follows.
[Bilingual Corpus]
Japanese sentence: Calling attention to heavy snow and heavy thunderstorms Sign language: {Very} {Snowfall} {Thunder} {Rain} {Caution} {Promotion} {Medium}

  FIG. 1 is a diagram showing a schematic configuration of a non-finger movement detection device according to an embodiment of the present invention. The non-finger movement detection device includes a morphological analysis unit 11, a word pre-alignment unit 12, a character pre-alignment unit 13, a sentence distance calculation unit 14, a kanji division unit 15, a translation probability estimation unit 16, a translation probability alignment unit 17, and a non-hand The control part 10 which has the operation | movement detection part 18, and the storage part 20 which has the bilingual corpus storage part 21 and the translation probability storage part 22 are provided. In the control unit 10, the word pre-alignment unit 12, the character pre-alignment unit 13, the sentence distance calculation unit 14, the kanji division unit 15, the translation probability estimation unit 16, and the translation probability alignment unit 17 constitute an alignment unit A. Each processing unit 11 to 18 of the control unit 10 is configured by a suitable processor, and each processing unit 11 to 18 can be implemented by a common processor or can be implemented as an individual processor. The storage unit 20 is a suitable storage device, and may be an external storage device via a communication interface as well as a built-in non-hand motion detection device.

  The morpheme analyzer 11 divides the Japanese sentence in the bilingual corpus into words by performing morphological analysis. In addition, when the sign language sentence in the bilingual corpus is not divided into words like Japanese labels, the morphological analysis unit 11 can divide the sign language sentence into words by performing morphological analysis. Hereinafter, each word in the Japanese sentence will be appropriately referred to as a Japanese word, and each word in the sign language sentence will be appropriately referred to as a sign language word.

  The alignment unit A associates (aligns) the Japanese words included in the Japanese sentence with the sign language words included in the sign language sentence.

  The word pre-alignment unit 12 associates Japanese words and sign language words that are superficially matched. That is, the word pre-alignment unit 12 associates Japanese words and sign language words whose characters (character strings) completely match.

  The character pre-alignment unit 13 associates the characters constituting the Japanese word and the sign language words that are superficially matched. That is, the character pre-alignment unit 13 associates a Japanese word with the same sign language word as the character constituting the Japanese word.

The in-sentence distance calculation unit 14 calculates the closeness (in-sentence distance) between a position where a certain Japanese word appears in a Japanese sentence and a position where a certain sign language word appears in a sign language sentence. In general, Japanese sentences and sign language sentences are similar in word order. Therefore, when the distance in a sentence between a certain Japanese word and a certain sign language word is close, there is a high possibility that the words correspond to each other. In addition, when the distance in a sentence between a certain Japanese word and a certain sign language word is long, the possibility that the words correspond to each other is low. Therefore, words whose sentence distance exceeds a predetermined threshold can be excluded from alignment targets. The in-sentence distance dist (j _p , s _q ) calculated by the in-sentence distance calculation unit 14 is defined as in Expression (1). Here, N _J is the number of Japanese words in the Japanese sentence, j _p is the p-th Japanese word, N _S is the number of sign language words in the sign language sentence, and s _q is the q-th sign language word.
dist (j _p , s _q ) = | p / N _J -q / N _S | (1)

  When a Japanese word composed of a plurality of kanji characters is translated into another language word that is greater than or equal to the number of kanji in the Japanese word, the kanji dividing unit 15 converts the Japanese word composed of a plurality of kanji characters. Divide into each kanji. Here, the other language is, for example, English, and when the Japanese word composed of two kanji characters is translated into two or more English words, the kanji dividing unit 15 Divide into kanji. Various known techniques can be used for translating Japanese words into other languages. For example, the kanji dividing unit 15 is provided with a translation dictionary from Japanese words into other languages.

  The translation probability estimation unit 16 estimates a translation probability between a Japanese word and a sign language word. Various well-known techniques can be used for estimating the translation probability. For example, the translation probability estimating unit 16 can estimate the translation probability using an EM algorithm. In addition, the Japanese word said here includes each Chinese character (Japanese word which consists of one Chinese character) divided | segmented by the Chinese character division | segmentation part 15. FIG.

  The translation probability alignment unit 17 associates the Japanese word with the sign language word based on the translation probability between the Japanese word and the sign language word. Various known techniques can be used for the association based on the translation probabilities. For example, the translation probability alignment unit 17 can perform word association using the Greedy algorithm. Since the Japanese word includes each kanji divided by the kanji dividing unit 15, the translation probability alignment unit 17 associates each kanji with the sign language word based on the translation probability between each kanji and the sign language word. Is to do.

  The non-finger movement detector 18 detects a Japanese word that performs a non-finger movement from Japanese words that are not associated with a sign language word. In particular, the non-hand motion detection unit 18 is a Japanese word that is a target of non-hand motion for content words that mainly represent lexical meaning, such as nouns, verbs, and adjectives, among Japanese words that are not associated with sign language words. Detect as.

  The bilingual corpus storage unit 21 stores a bilingual corpus including a Japanese sentence and a sign language sentence corresponding to the Japanese sentence. The bilingual corpus storage unit 21 appropriately stores the association result between the Japanese word and the sign language word by the alignment unit A.

  The translation probability storage unit 22 stores the translation probability between the Japanese word and the sign language word estimated by the translation probability estimation unit 16.

FIG. 2 is a flowchart illustrating processing of the non-finger movement detection device. In this flowchart, the processing of each functional unit will be described using the following parallel corpus 1 as an example.
[Bilingual Corpus 1]
Japanese sentence: Calling attention to heavy snow and heavy thunderstorms Sign language: {Very} {Snowfall} {Thunder} {Rain} {Caution} {Promotion} {Medium}

In S <b> 1, when a bilingual corpus including a Japanese sentence and a sign language sentence is input, the morphological analysis unit 11 performs morphological analysis on the Japanese sentence of the bilingual corpus and divides the word into words, and stores the division result in the bilingual corpus storage unit 21. To do. For example, the bilingual corpus 1 is divided into words by the morphological analyzer 11 as follows.
[Bilingual Corpus 1 (word division example)]
Japanese statement: heavy snow ₁ / and ₂ / strong ₃ / thunderstorm _4/5 / Note ₆ / to _7/8 so / called ₉ / Te ₁₀ / have ₁₁ / mass ₁₂
Sign language: {Very} ₁ {Snowfall} ₂ {Thunder} ₃ {Rain} ₄ {Caution} ₅ {Promotion} ₆ {Medium} ₇

In S2, the word pre-alignment unit 12 discovers the Japanese words and sign language words divided in S1 that match in the surface layer, and the in-sentence distance calculated by the in-sentence distance calculation unit 14 is less than or equal to the threshold value. If there is, the alignment result is stored in the bilingual corpus storage unit 21 in association with the words. For example, among the Japanese words and sign language words of the bilingual corpus 1 (word division example), the words that match surfacely are “Caution ₆ ” and {Caution} ₅ .

Here, the in-sentence distance calculation unit 14 calculates the in-sentence distance dist (attention ₆ , {attention} ₅ ) between “attention ₆ ” and {attention} ₅ . Assuming that the threshold of the distance in the sentence is 0.3, dist (note ₆ , {attention} ₅ ) is a value below the threshold 0.3 as shown below, so the word pre-alignment unit 12 sets “attention ₆ ” and {attention} ₅ to And the alignment result is stored in the bilingual corpus storage unit 21.
dist (Note ₆ , {Note} ₅ ) = | _6/ 12-5/7 | = 0.21 ≤ 0.3

In S3, the character pre-alignment unit 13 finds the characters that make up the Japanese words and the sign language words in a superficial manner for unaligned Japanese words, and the in-sentence distance calculation unit 14 If the calculated value is less than or equal to the threshold value, the words are aligned and the alignment result is stored in the bilingual corpus storage unit 21. For example, among the Japanese words in the bilingual corpus 1 (word division example) that are not aligned in S2, the character that constitutes the Japanese word and the sign language word are superficially matched is the next word.
Thunder ₄ Thunderstorm = {Thunder} ₃
Thunderstorm ₄ rain = {rain} ₄

Here, the sentence distance calculation unit 14, the sentence distance dist (thunderstorm ₄ , {thunder} ₃ ) between “thunderstorm ₄ ” and {thunder} _3, and the sentence distance dist (thunderstorm) between “thunderstorm ₄ ” and {rain} _{4. 4} , {rain} ₄ ) As described below, since dist (thunderstorm ₄ , {thunder} ₃ ) and dist (thunderstorm ₄ , {rain} ₄ ) are both below the threshold value 0.3, the character pre-alignment unit 13 performs “thunder” in thunderstorm ₄ And {thunder} ₃ , “rain” of thunderstorm ₄ and {rain} ₄ are associated with each other, and the alignment result is stored in the parallel corpus storage unit 21.
dist (Thunderstorm ₄ , {thunder} ₃ ) = | 4/12-3/7 | = 0.10 ≦ 0.3
dist (Thunderstorm ₄ , {Rain} ₄ ) = | 4/12-4/7 | = 0.24 ≤ 0.3

In S4, when the Japanese word composed of a plurality of kanji characters is translated into other language words equal to or more than the number of kanji of the Japanese word among the unaligned Japanese words, the kanji dividing unit 15 starts from the plurality of kanji characters. The constructed Japanese word is divided into kanji characters, and each divided kanji character is stored in the bilingual corpus storage unit 21. For example, among the Japanese words of the bilingual corpus 1 (word division example) that are not aligned in S2 and S3, the word composed of a plurality of Chinese characters is “Daiyuki ₁ ”. Kanji dividing unit 15, when you see, for example, Japanese-English bilingual dictionary, each Chinese character for "heavy snow _1" is translated into two of the English word "heavy snow", "heavy snow _1" and "large", "snow" The result of the division is stored in the bilingual corpus storage unit 21.

In S <b> 5, the translation probability estimation unit 16 estimates a translation probability for a set of words whose sentence distance is equal to or less than a threshold among unaligned Japanese words and sign language words, and stores them in the translation probability storage unit 22. To do. Here, in the bilingual corpus 1 (word division example), Japanese words and sign language words that are not aligned in S2 to S4 are as follows.
[Bilingual corpus (unaligned words)]
Japanese statement: large ₁ / snow ₁ / and ₂ / strong _3/5 / to _7/8 so / called ₉ / Te ₁₀ / have ₁₁ / mass ₁₂
Sign language: {Very} ₁ {Snowfall} ₂ {Promotion} ₆ {Medium} ₇

First, the sentence distance calculation unit 14 calculates the sentence distance between each Japanese word and each sign language word. For example, the distance in the sentence between the Japanese word “Dai ₁ ” and each sign language word can be calculated as follows. As a result, for “large ₁ ”, the translation probabilities are estimated by the translation probability estimator 16 for {very} ₁ and {snowfall} ₂ where the distance in the sentence is equal to or less than the threshold (0.3).
dist (large ₁ , {very} ₁ ) = | ₁ /12-1/7 | = 0.20 ≤ 0.3
dist (large ₁ , {snowfall} ₂ ) = | ₁ /12-2/7 | = 0.20 ≤ 0.3
dist (large ₁ , {advertisement} ₆ ) = | 1/12-6/7 | = 0.77> 0.3
dist (large ₁ , {medium} ₇ ) = | 1/12-3/ ₇ | = 0.92> 0.3

The translation probability estimation unit 16 estimates the following translation probabilities for a pair of Japanese words and sign language words whose distance in the sentence is equal to or less than a threshold, for example, using the EM algorithm, and stores the estimated translation probabilities in the translation probability storage unit 22. To do.
Large ₁ : {very} ₁ (translation probability = 0.54)
Snow ₁ : {Snowfall} ₂ (Translation probability = 0.83)
Call ₉ : {Promotion} ₆ (Translation probability = 0.65)
₁₁ : {Medium} ₇ (Translation probability = 0.16)

In S6, the translation probability alignment unit 17 performs alignment of words that are not yet aligned using the translation probability estimated in S5, and stores the alignment result in the bilingual corpus storage unit 21. The translation probability alignment unit 17 performs, for example, the following alignment by a greedy algorithm using only a translation probability equal to or higher than a certain threshold, and stores the alignment result in the bilingual corpus storage unit 21.
Large ₁ = {Very} ₁
Snow ₁ = {Snowfall} ₂
Strong ₃ = (no alignment)
₅ = (no alignment)
₇ = (No alignment)
Call ₉ = {Promotion} _6
11 = {Medium} _7
12 = (No alignment)

The non-finger movement detector 18 detects a Japanese word that performs a non-finger movement from Japanese words that are not yet associated with a sign language word after the alignment of S2-6. In particular, the non-finger movement detection unit 18 detects content words that mainly represent lexical meaning, such as nouns, verbs, and adjectives, among Japanese words that are not aligned as Japanese words that are subject to non-finger movement. For example, in the above example, the Japanese words determined to have no alignment are “strong ₃ ”, “ni ₅ ”, “do ₇ ”, and “mas ₁₂ ”. The non-hand motion detection unit 18 detects “strong ₃ ”, which is a content word, as a Japanese word that is a target of non-hand motion.

In S8, the non-finger movement detection unit 18 outputs the Japanese word that is the target of the non-finger movement detected in S7 to a subsequent sign language display device or the like (not shown). For example, in the case of the bilingual corpus 1, “strong ₃ ” in the Japanese sentence relates to “thunderstorm ₄ ”, and such dependency information can be acquired by the morphological analysis of S1. For this reason, the non-finger movement detection unit 18 outputs the dependency information in addition to the Japanese word that is the target of the non-finger movement, so that the non-finger movement “strong” is performed in the sign language display device or the like in the subsequent stage. Can be performed at the timing of the “thunderstorm” finger movement.

  Thus, according to the present embodiment, the alignment unit A associates the Japanese words included in the Japanese sentence with the sign language words included in the sign language sentence, and the non-finger movement detection unit 18 includes Japanese words that perform non-finger movement are detected from Japanese words that are not associated with sign language words. Thereby, it is possible to detect a word that should be operated by a non-finger based on a bilingual corpus of a Japanese sentence and a sign language sentence.

  The alignment unit A also includes a Japanese word composed of a plurality of kanji characters when a Japanese word composed of a plurality of kanji characters is translated into another language word equal to or more than the number of kanji in the Japanese word. Can be divided into each Chinese character, and each Chinese character can be associated with the sign language word based on the translation probability between each Chinese character and the sign language word. As a result, even if a Japanese word and a sign language word that do not coincide with each other on the surface layer, if the meanings of the kanji and the sign language word constituting the Japanese word are similar, the words can be associated with each other. For this reason, the correspondence accuracy between the words in the bilingual corpus can be improved, and as a result, the number of unaligned words is reduced, so that it is possible to detect a word that should be non-finger-operated more accurately.

  In addition, the alignment unit A detects a content word representing a lexical meaning among Japanese words that are not associated with a sign language word as a Japanese word that is a target of non-hand movement. This makes it possible to express content words that mainly represent lexical meaning, such as nouns, verbs, and adjectives, by non-finger movements.

  Note that a computer can be used to function as the above-described non-finger movement detection device, and such a computer stores a program describing processing contents for realizing each function of the non-finger movement detection device. This program can be realized by reading out and executing this program by the CPU of the computer. This program can be recorded on a computer-readable recording medium.

  Although the present invention has been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications and corrections based on the present disclosure. Therefore, it should be noted that these variations and modifications are included in the scope of the present invention. For example, the functions included in each functional unit, each step, etc. can be rearranged so as not to be logically contradictory, and a plurality of means and steps can be combined into one or divided. is there.

DESCRIPTION OF SYMBOLS 10 Control part 11 Morphological analysis part 12 Word pre-alignment part 13 Character pre-alignment part 14 Sentence distance calculation part 15 Kanji division part 16 Translation probability estimation part 17 Translation probability alignment part 18 Non-finger movement detection part A Alignment part 20 Storage part 21 Parallel translation Corpus storage unit 22 Translation probability storage unit

Claims (3)

A storage unit for storing a bilingual corpus including a Japanese sentence and a sign language sentence corresponding to the Japanese sentence, the Japanese sentence is divided into Japanese words, stored in association with a number indicating a word order, and The storage unit that divides a sign language sentence into sign language words corresponding to finger movements and stores them in association with numbers indicating word orders ;
Said Japanese words contained in the Japanese sentence, said a alignment unit for associating the said sign language words included in the sign language sentence, match character or string between the sign language words and the Japanese word, The alignment unit includes associating the Japanese word with the sign language word based on a sentence distance between the Japanese word and the sign language word ;
A non-finger movement detector for detecting a content word representing a lexical meaning as a Japanese word to be subjected to a non-finger movement from a Japanese word that is not associated with the sign language word among the Japanese words ; A non-finger movement detection device provided. The alignment unit, when a Japanese word composed of a plurality of Chinese characters among the Japanese words is translated into another language word that is equal to or more than the number of Chinese characters of the Japanese word, The non-finger movement detection device according to claim 1, further comprising: associating each kanji with the sign language word based on a translation probability between each kanji and the sign language word. A program for causing a computer to function as a non-manual motion detection apparatus according to claim 1 or 2.

Images