JPH04222014A - Automatic finger talking translating device using neural network and its translating method - Google Patents

Abstract

PURPOSE:To automatically translate words, meaning, sentences, etc., expressed by the fingers of a finger talking person by using a neural network. CONSTITUTION:The angles of respective joints of respective fingers of the hands and the positions and directions of the hands are detected by data globes and positional direction sensors, each of an input layer, an intermediate layer and an output layer consists of plural units and the units of respective adjacent layers are completely mutually connected with an inherent weight coefficient. In the neural network, a value obtained by multiplying the output value of the unit in the preceding layer by said weight coefficient is inputted to the units in the succeeding layer and each unit in the succeeding layer calculates the sum of values inputted from all the units in the preceding layer, applies non-linear conversion defined as an output function to the sum and then outputs the converted data to the succeeding layer. In said constitution, detection signals outputted from respective sensors are supplied to the neural network and finger talking translated results outputted from the neural network are outputted as language levels through a display and a speaker.

Description

[Detailed description of the invention]

0001

[Industrial Application Field] The present invention relates to an automatic sign language translation device and its translation method using a neural network to automatically translate words, meanings, sentences, etc. expressed by a sign language person in sign language. Regarding.

0002

[Prior Art] Conventionally, robots that perform sign language movements have been considered at the research stage, but on the other hand, robots that perform sign language movements automatically translate words, meanings, sentences, etc. expressed by sign language users into everyday or conversational language. There is no equipment. Such devices are also necessary for communication between ordinary people who do not know sign language and sign language users.

0003

[Problem to be solved by the invention] There is no device that automatically translates the words, meanings, sentences, etc. expressed by a sign language person into everyday or conversational language, and it is difficult for the sign language person to communicate with ordinary people who do not know sign language. Such devices are also necessary for communication between people.

The present invention has been made in view of the above, and
The aim is to create an automatic sign language translation device using a neural network that automatically translates the words, meanings, sentences, etc. expressed by sign language speakers into everyday language so that people who do not know sign language can understand them. The purpose is to provide a method.

[0005]

[Means for Solving the Problems] In order to achieve the above object, an automatic sign language translation device using a neural network according to the present invention includes a first detection means for detecting the angle of each joint of each finger of the hand; a second detection means for detecting position and direction; an input layer comprising a plurality of units supplied with detection signals from the first and second detection means; and an input layer configured to be supplied with output signals from the input layer. an output consisting of at least one or more intermediate layers consisting of a plurality of units coupled to each other, and a plurality of units coupled so as to be supplied with output signals from the intermediate layer, each unit outputting a sign language translation result; The units in each adjacent layer are connected with each other with a unique weighting coefficient, and the value obtained by multiplying the output value of the unit in the previous layer by the weight is input to the unit in the next layer, and the unit in the next layer is The unit of the layer calculates the sum of the values from all the units of the previous layer, and after applying a nonlinear transformation defined as an output function, the neural network outputs to the next layer, and the output from the neural network. The gist of the present invention is to have a conversion means for converting a sign language translation result into a language level.

[0006] Furthermore, the automatic sign language translation method using a neural network of the present invention uses time-series data of each joint angle of each finger of the hand and hand position and direction information corresponding to each measured sign language translation result as a teacher. The error between the output signal of the output layer obtained by feeding the input layer of the neural network as a signal and the ideal output signal is used to change the weight value between each unit of the neural network, and calculate the weight value of each finger of the hand. A neural network is trained to output correct sign language translation results corresponding to the input information of each joint angle and hand position and direction, and the input layer of this trained neural network is Input time series data of hand position and direction information,
The gist is to output a sign language translation result corresponding to the input information from the output layer of the neural network.

[0007]

[Operation] The automatic sign language translation device using the neural network of the present invention detects the angle of each joint of each finger of the hand and the position and direction of the hand. The units in each adjacent layer are connected with each other using a unique weighting coefficient, and the value obtained by multiplying the output value of the unit in the previous layer by the weight is input to the unit in the next layer, and the unit in the next layer is A unit of a layer calculates the sum of values from all units of the previous layer, and after applying a nonlinear transformation defined as an output function,
The detection signal is supplied to a neural network that outputs to the next layer, and the sign language translation result output from the neural network is converted to a linguistic level.

In addition, in the automatic sign language translation method using a neural network of the present invention, time series data of each joint angle of each finger of the hand and hand position and direction information corresponding to each measured sign language translation result is used as a teacher. The error between the output signal of the output layer obtained by feeding the input layer of the neural network as a signal and the ideal output signal is used to change the weight value between each unit of the neural network, and calculate the weight value of each finger of the hand. A neural network is trained to output correct sign language translation results corresponding to the input information of each joint angle and hand position and direction, and the input layer of this trained neural network is Time series data of hand position and direction information is input, and a sign language translation result corresponding to the input information is output from the output layer of the neural network.

[0009]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Examples of the present invention will be described below with reference to the drawings.

FIG. 1 is an overall configuration diagram of an automatic sign language translation device using a neural network according to an embodiment of the present invention. This automatic sign language translation device uses a data glove 1 that is a sensor that detects the angle of each joint of each finger of a signer's hand.
and a position/direction sensor 3 that detects the position and direction of the hand.
has.

The data glove 1 is a glove made of stretchable cloth with optical fibers attached along each finger as shown in the figure. When the fibers are bent, the amount of light passing through the fibers increases. It uses the principle of change to detect the angle of each joint of each finger. Further, the position/direction sensor 3 detects the distance from a certain reference position and the direction with respect to the reference direction.

The joint angle signals of each finger detected by the data glove 1 and the hand position and direction signals detected by the position and direction sensor 3 are transmitted as time series data to a neural network 10 via an input side conversion device 5. is supplied to

The neural network 10 is composed of a plurality of units, which are neurons indicated by circles, and includes an input layer 7 to which each joint angle of each finger, hand position and direction signals are supplied, and an output signal from the input layer 7. The intermediate layer 9 consists of at least one intermediate layer 9 consisting of a plurality of units coupled to be supplied with a As shown in the figure, it is composed of three layers: an output layer 11 that outputs sign language translation results such as word 1, word 2, ... word 7, and the units in each adjacent layer are connected with each other using unique weighting coefficients. has been done. In addition, in this neural network 10, the value obtained by multiplying the output value of the unit in the previous layer by the weight is input to the unit in the subsequent layer, and the unit in the subsequent layer receives the values from all the units in the previous layer. After calculating the sum and applying a nonlinear transformation defined as an output function, it is output to the next layer.

The output signal from the neural network 10 is converted into language level or auditory level information by the output side conversion device 13, and is supplied to the display 15 and the speaker 17. is output as audio.

The output layer 11 of the neural network 10 shown in FIG. 1 has seven units, each of which outputs seven words or meanings. The number of words or meanings can be freely set by changing the number of units in the output layer 11. When a sign language person performs sign language, if a certain unit of the output layer 11 outputs a value of approximately "1" and other units approximately output a value of "0", the sign language person The word or meaning expressed by the data globe 1 means that the word or meaning represented by the unit whose approximate value is "1" is output.

Automatic sign language translation by the neural network 10 configured as described above is roughly divided into two processes.

The first process is a learning process. In this learning process, the neural network 10 uses the output signals of the data glove 1 and the position/direction sensor 3 as teacher signals when a sign language person performs sign language as a sample.
The output of the output layer obtained by inputting the input to the input layer 7 of 0'') is used to improve the neural network 10 and construct a correct neural network 10. The configuration of this neural network is to change the weight values between units using a method called back propagation learning rule, and finally the data glove 1 and the position/direction sensor 3 when a sign language person signs. When inputting the output value of , a neural network is constructed that outputs "1" only for the correct word or unit of meaning that the sign language person was trying to express.

The second process uses the neural network completed by the learning described above to input the output signals of the data glove 1 and the position/direction sensor 3 when an unknown sign language person signs to the input layer of the neural network 10. 7 and the output from the output layer 11 is used to determine the word or meaning that the sign language person is trying to express.

Therefore, if the output signals from the data glove 1 and the position/direction sensor 3 when a sign language person signs are input to the input layer 7 of the neural network 10 and learned by the back propagation learning rule, the target sign language can be learned. A neural network is formed to determine the After the learning is completed, if the output signals from the data glove 1 and position/direction sensor 3 when an unknown sign language person signs are input to the neural network 10, the words or meanings of the sign language can be instantly determined. can be determined.

Next, actual sign language will be explained, and how this automatic sign language translation device is applied will be explained. Sign language is a visual form of expression that combines several units of body and hand movements. A simple action unit is a finger letter. This is used to express particles, auxiliary verbs, conjugated endings, etc., and can express spoken words in sign language.

FIG. 2 shows an example of the 50-sound finger letters, and FIG. 3 shows the alphabet letters. This kind of finger writing can be expressed with one hand. Therefore, in order to translate these finger letters with the automatic sign language translation device, the data glove 1 is put on, the shape of each finger letter is created, a teacher signal corresponding to each finger letter is given, and the neural network 10 You can learn it using .

However, unlike other finger characters, the movements of finger characters such as "no" and "ri" are not static but dynamic, so in this case, for example, as shown in FIG. The data division switch 19 and the operation division switch 21, which are comprised of a foot switch or the like, may be used to divide the detection output signal during the operation, convert it into time series data, and input it into the neural network 10 for learning. However, since the speed of movement etc. differs depending on the person, it is necessary to devise a method of division. In the case of finger characters, since it is a simple dynamic motion, the data at the beginning and end of the motion can be cut out using the motion division switch 21 and input.

[0023]Actual sign language is performed as shown in FIG. FIG. 4 shows a state in which a person is using sign language to say, "This is my friend, Mr. Hayashi." In this case, first, the neural network 10 is trained to learn the words and meanings for each action unit used in sign language, and when translating, the sign language person is asked to separate each action unit using the action division switch 21. If the detection output signals of the data glove 1 and the position/direction sensor 3 are input to the trained neural network 10, the words and meanings of each action unit are outputted from the neural network 10 as an output signal, so this output signal is used as the output signal. By supplying the sign language translation result to the display 15 and speaker 17 via the side conversion device 13, the sign language translation result is displayed on the display 15 and outputted as audio from the speaker 17. In addition,
In this case, since two hands are used, it is necessary to use two data gloves 1 and two position/direction sensors 3. In addition, as shown in the third frame of FIG. 4, there is a considerable amount of dynamic movement involved, so the method explained using finger letters is used for learning.

[0024]

[Effects of the Invention] As explained above, according to the present invention,
Since words, meanings, sentences, etc. expressed by a sign language person in sign language can be automatically translated into everyday language, communication between the sign language person and ordinary people who do not know sign language can be carried out smoothly.

[Brief explanation of the drawing]

FIG. 1 is an overall configuration diagram of an automatic sign language translation device using a neural network according to an embodiment of the present invention.

FIG. 2 is a diagram showing an example of a 50-sound finger alphabet.

FIG. 3 is a diagram showing finger letters of the alphabet.

FIG. 4 is a diagram showing an example of actual sign language.

[Explanation of symbols]

1. Data glove 3 Position direction sensor 7 Input layer 9. Middle class 10 Neural network 11 Output layer 15 Display 17 Speaker

Claims (3)

[Claims] [Claim 1] A first sensor that detects the angle of each joint of each finger of the hand.
an input layer comprising a detection means, a second detection means for detecting the position and direction of the hand, and a plurality of units supplied with detection signals from the first and second detection means; At least one intermediate layer consisting of a plurality of units coupled to each other so as to be supplied with an output signal, and a plurality of units coupled to each other so as to be supplied with an output signal from the intermediate layer, from each unit a sign language It has an output layer in which the translation result is output, and the units in each adjacent layer are all connected with a unique weighting coefficient, and the value obtained by multiplying the output value of the unit in the previous layer by the weight is the output value in the next layer. A neural network in which the unit in the next layer calculates the sum of the values from all the units in the previous layer, performs a nonlinear transformation defined as an output function, and then outputs it to the next layer. 1. A sign language automatic translation device using a neural network, comprising: a conversion means for converting a sign language translation result outputted from the neural network into a language level. 2. Data for creating time series data values of each joint angle information of each finger of the hand detected by the first detection means and the position and direction information of the hand detected by the second detection means. 2. The automatic sign language translation device using a neural network according to claim 1, further comprising a classification switch and an action division switch that divides the information into action units. [Claim 3] Output obtained by supplying time-series data of each joint angle of each finger of the hand and position and direction information of the hand corresponding to each measured sign language translation result to the input layer of the neural network as a teacher signal. The error between the output signal of the layer and the ideal output signal is used to change the weight value between each unit of the neural network, corresponding to the input information of each joint angle of each finger of the hand and the position and direction of the hand. A neural network is trained to output the correct sign language translation result, and time series data of each joint angle of each finger of an unknown hand and hand position and direction information is input to the input layer of this trained neural network. An automatic sign language translation method using a neural network, characterized in that a sign language translation result corresponding to the input information is output from an output layer of a neural network.