JP7422535B2 - Conversion device and program - Google Patents

Description

The present invention relates to a conversion device and a program.

Technology that automatically recognizes the content shown in videos is expected to be used as a means of assisting human communication. As an example, technology that automatically recognizes sign language images captured by a camera or the like is expected to be used for communication between hearing-impaired people and normal hearing people.

Non-Patent Document 1 describes research on automatically recognizing German sign language, which is one sign language, and converting it into German. For example, Figure 2 in Non-Patent Document 1 shows a schematic configuration of a sign language translator for translating sign language into colloquial language. The sign language translator shown in Figure 2 consists of an encoder and a decoder. The encoder and decoder each use a recurrent neural network (RNN). The encoder receives a sequence of frame images and generates a feature vector. The decoder receives the feature vectors generated by the encoder and generates a sequence of words.

Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, Richard Bowden “Neural Sign Language Translation” In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.

BACKGROUND ART Technology for recognizing the content of images captured using a camera (for example, human gestures, etc.) has been put into practical use, for example, in application areas where a non-contact human-machine interface is desired. Areas where contactless interfaces are desired include, for example, areas where hygiene considerations are required, such as food factories and medical sites. However, technology that automatically recognizes continuous and complex human movements and converts them into another language, such as in sign language, has not yet reached a practical level.

No practical examples have been reported regarding automatic recognition of Japanese Sign Language, one of the sign languages used in Japan.

Furthermore, if the input sign language video is not divided into word units in advance, it is even more difficult to automatically recognize the sign language words by automatically dividing the video into sign language word units based on the video.

The present invention was made based on the above-mentioned problem recognition, and it is possible to process input data (for example, video (series of frame images) that is not divided into predetermined units (for example, divisions of words to be converted)). The present invention aims to provide a conversion device and a program that can input data and output a symbol string (for example, a word string in a predetermined linguistic expression) corresponding to the input data.

[1] In order to solve the above problems, a conversion device according to one aspect of the present invention includes an encoder unit that generates state data based on input data, a decoder unit that generates output data based on the state data, a learning data supply unit that supplies a pair of learning input data that is input to the encoder unit and correct answer data that is a correct answer of the output data corresponding to the learning input data; the difference between the learning output data generated by the decoder section based on the state data generated by the encoder section and the correct answer data supplied by the learning data supply section corresponding to the learning input data. a second encoder unit that generates estimated state data that is state data estimated based on the correct data; and a second encoder unit that generates estimated state data that is state data that is estimated based on the correct data; Calculating a second loss representing the difference between the state data and the estimated state data generated by the second encoder unit based on the correct data supplied by the learning data supply unit in response to the learning input data. a second loss calculation unit that performs learning; and a control unit that controls the operation to switch between a first learning mode, a second learning mode, and a conversion execution mode as appropriate; The internal parameters of the encoder section and the decoder section are adjusted based on the loss calculated by the loss calculation section based on the learning input data and the correct data supplied by the data supply section, and the second learning is performed. In the mode, the encoder unit and the second encoder unit calculate the second loss based on the second loss calculated by the second loss calculation unit based on the learning input data supplied by the learning data supply unit and the correct answer data. In the conversion execution mode, the encoder section generates state data based on the input data, and the decoder section generates output data based on the state data generated by the encoder section. It is a converting device that generates.

[2] Further, in one aspect of the present invention, in the above conversion device, each of the encoder section, the decoder section, and the second encoder section includes a neural network therein, and the first learning mode In the second learning mode, the internal parameters of the encoder section and the decoder section are adjusted by performing error backpropagation of the respective neural networks of the encoder section and the decoder section based on the loss, and in the second learning mode, The internal parameters of the encoder section and the second encoder section are adjusted by performing error backpropagation in neural networks of the encoder section and the second encoder section based on the second loss.

[3] Further, in one aspect of the present invention, in the above-mentioned conversion device, the control unit, during the learning process, performs the following for each pair of the learning input data and the correct answer data supplied by the learning data supply unit. , the first learning mode and the second learning mode are controlled to be executed repeatedly.

[4] Moreover, one aspect of the present invention is that in the above conversion device, the input data is a series of images, and the output data is a series of predetermined symbols.

[5] Further, in one aspect of the present invention, in the above conversion device, the series of images is a series of images representing a sign language, and the series of symbols is a series of words in gloss notation corresponding to the sign language. It is called a column.

[6] Further, one aspect of the present invention provides an encoder unit that generates state data based on input data, a decoder unit that generates output data based on the state data, and a learning device that is input to the encoder unit. a learning data supply unit that supplies a pair of input data for learning and correct answer data that is a correct answer of the output data corresponding to the input data for learning, and state data generated by the encoder unit based on the input data for learning. a loss calculation unit that calculates a loss representing a difference between the learning output data generated by the decoder unit based on , and the correct answer data supplied by the learning data supply unit corresponding to the learning input data; , a second encoder unit that generates estimated state data that is state data estimated based on the correct data; the state data that the encoder unit generates based on the learning input data; and the learning input data. and the estimated state data generated by the second encoder unit based on the correct data supplied by the learning data supply unit in response to the second loss calculation unit; a control unit that controls the operation by appropriately switching between a first learning mode, a second learning mode, and a conversion execution mode, and in the first learning mode, the learning data supply unit supplies the learning data. The internal parameters of the encoder section and the decoder section are adjusted based on the loss calculated by the loss calculation section based on the input data and the correct data, and in the second learning mode, the learning data supply section Adjusting the internal parameters of the encoder unit and the second encoder unit based on the second loss calculated by the second loss calculation unit based on the supplied learning input data and the correct answer data, and performing the conversion. In the execution mode, the computer functions as a conversion device in which the encoder section generates state data based on input data, and the decoder section generates output data based on the state data generated by the encoder section. This is a program that allows you to

According to the present invention, recognition accuracy can be improved in the process of automatically recognizing data (video) and outputting a symbol string corresponding to the video.

FIG. 1 is a block diagram showing a schematic functional configuration of a conversion device according to an embodiment of the present invention. FIG. 3 is a schematic diagram showing the flow of data through processing by an encoder section and a decoder section in the conversion device according to the embodiment. FIG. 7 is a schematic diagram showing the flow of data through learning processing using a second encoder unit, which is an additional network, of the conversion device according to the embodiment. FIG. 2 is a block diagram showing a more detailed configuration example of an encoder unit in the conversion device according to the same embodiment. FIG. 2 is a block diagram showing a more detailed configuration example of a decoder section in the conversion device according to the same embodiment. FIG. 3 is a block diagram showing a more detailed configuration example of a second encoder section in the conversion device according to the same embodiment. It is a flowchart which shows the processing procedure when the conversion apparatus by the same embodiment performs machine learning processing. This is a graph showing the results of an evaluation experiment, and is a graph for comparing the conversion error rate between the conversion device of this embodiment and the conversion device according to the conventional technology.

Next, embodiments of the present invention will be described.

The conversion device according to this embodiment inputs a sign language video and converts the sign language represented by the video into an intermediate representation called gloss notation. Gloss notation is a symbol string in a sign language that does not have letters, in which a series of actions that make up a sign language phrase or sentence is separated into short intervals corresponding to words in sign language and transcribed using letters. In Japanese Sign Language gloss notation, Japanese words that are close in meaning to the sign language words are used as labels. That is, the conversion device according to this embodiment inputs a sign language video, performs automatic recognition processing on the video, and outputs a label string (symbol string) corresponding to the video.

Note that the sign language video input to the conversion device is not divided into units such as words in advance. Also, meta information indicating the break position is not provided.

FIG. 1 is a block diagram showing a schematic functional configuration of a conversion device according to this embodiment. As illustrated, the conversion device 1 includes an input section 10, an encoder section 20, a decoder section 30, an output section 40, a loss calculation section 50, a second encoder section 60, and a second loss calculation section 70. , a learning data supply section 80, and a control section 90. Each of these functional units can be realized by, for example, a computer and a program. Furthermore, each functional section has a storage means, if necessary. The storage means is, for example, variables on the program or memory allocated by executing the program. Furthermore, if necessary, nonvolatile storage means such as a magnetic hard disk device or a solid state drive (SSD) may be used. Furthermore, at least some of the functions of each functional unit may be realized as a dedicated electronic circuit instead of a program. The functions of each functional section will be explained next.

The input unit 10 acquires input data and supplies it to the encoder unit 20. This input data may be a sequence of images. Furthermore, this series of images (video) may be a series of images (video) representing sign language. Note that the frame rate of the video is arbitrary, and may be, for example, about 30 frames per second (fps).

The encoder unit 20 generates a state vector (state data) based on input data. The encoder unit 20 has a function of updating an internal model (adjusting parameters) by performing machine learning processing. In this embodiment, the encoder unit 20 has an internal neural network and can update internal parameters through error backpropagation processing. Note that the error backpropagation method itself can be implemented using existing technology.

The decoder unit 30 generates output data based on the state vector (state data). The decoder unit 30 has a function of performing machine learning processing to update an internal model (adjust parameters). In this embodiment, the decoder unit 30 has an internal neural network, and can update internal parameters through error backpropagation processing.

The output unit 40 outputs the output data (estimated symbol string) generated by the decoder unit 30. The output data may be, for example, a word string (symbol string) in gloss notation corresponding to a sign language video.

The loss calculation unit 50 includes output data (estimated word strings, learning output data) generated by the encoder unit 20 and the decoder unit 30 based on the learning input data input to the encoder unit 20, and the learning input data. A loss representing the difference between the data and the correct data supplied by the learning data supply unit is calculated. When the learning output data generated by the encoder unit 20 and the decoder unit 30 and the correct answer data are respectively considered to be vectors corresponding to symbol strings, the loss calculated by the loss calculation unit 50 is, for example, It is the norm between both vectors.

The second encoder unit 60 generates estimated state data, which is an estimated state vector (state data), based on the correct data supplied by the learning data supply unit 80 (correct answer of the output data output by the decoder unit 30). . The second encoder unit 60 has a function of performing machine learning processing to update the internal model (adjust parameters). In this embodiment, the second encoder unit 60 has an internal neural network and can update internal parameters through error backpropagation processing.

The second loss calculation unit 70 calculates the second encoder based on the state data generated by the encoder unit 20 based on the learning input data and the correct data supplied by the learning data supply unit 80 in response to the learning input data. A second loss representing the difference between the estimated state data generated by the unit 60 and the estimated state data generated by the unit 60 is calculated. When both the state data and the estimated state data are regarded as vectors, the second loss calculated by the second loss calculation unit 70 is, for example, the norm between these two vectors.

The learning data supply unit 80 supplies learning data for the encoder unit 20, decoder unit 30, and second encoder unit 60 to perform machine learning. Specifically, the learning data supply unit 80 supplies a pair of learning input data that is input to the encoder unit 20 and correct answer data that is the correct answer to the output data corresponding to the learning input data. The above learning input data is also used as input to the second encoder section 60. The learning data supply unit 80 supplies many pairs of learning input data and correct answer data.

The control unit 90 controls the entire operation of the conversion device 1 . The control unit 90 performs control based on at least the operation mode of the conversion device 1. As a specific example, the control unit 90 controls each unit of the conversion device 1 to operate the conversion device 1 by appropriately switching between the first learning mode, the second learning mode, and the conversion execution mode. The operation of the conversion device 1 in each mode is as follows. In the first learning mode, the internal parameters of the encoder section 20 and the decoder section 30 are calculated based on the loss calculated by the loss calculation section 50 based on the learning input data and correct answer data supplied by the learning data supply section 80. adjust. In the second learning mode, the encoder unit 20 and the second encoder unit 60 calculate the second loss based on the learning input data supplied by the learning data supply unit 80 and the correct data. Adjust the internal parameters of. The adjustment of the internal parameters in each of the first learning mode and the second learning mode is, for example, a process of updating the internal parameters of each part by error backpropagation processing in a neural network. The internal parameters of a neural network are vectors of weight values that are applied to input values when calculating the output value at each node. In the conversion execution mode, the encoder unit 20 generates a state vector (state data) based on the input data, and the decoder unit 30 generates output data based on the state vector (state data) generated by the encoder unit 20. (estimated transformation result corresponding to input data) is generated.

Note that the control procedure by which the control unit 90 switches between the first learning mode and the second learning mode will be further described later with reference to FIG. 7 (flowchart).

FIG. 2 is a schematic diagram showing the flow of data in the operation of the encoder unit 20 and decoder unit 30 in the conversion device 1 of this embodiment. As explained below, the encoder section 20 and the decoder section 30 operate in a learning mode and a conversion mode.

The encoder section 20 has a neural network 201 inside. A series of frame images frame ₁ , frame ₂ , . . . , frame _r included in the input video is input to the neural network 201 . The neural network 201 outputs a state vector calculated based on a series of frame images frame ₁ , frame ₂ , . . . , frame _r . The encoder unit 20 passes the state vector generated based on the input video to the decoder unit 30.

The decoder section 30 has a neural network 301 inside. The state vector generated by the neural network 201 of the encoder unit 20 is input to the neural network 301 . The neural network 301 outputs a string of words word ₁ , word ₂ , . . . , word _u-1 , word _u calculated based on the input state vector. All of these words are symbols in the gloss notation described above. Further, the neural network 301 outputs a special symbol <eos> at the end of the word string. <eos> is a symbol representing the end of sentence. The word string output by the neural network 301 is also called an estimated word string.

When operating in learning mode, each of the neural networks 201 and 301 adjusts internal parameters by performing machine learning processing based on learning data. When operating in the conversion mode, each of the neural networks 201 and 301 calculates an output using adjusted internal parameters in the machine learning process. When the encoder unit 20 and the decoder unit 30 operate in the conversion mode, the estimated word string output by the neural network 301 is the conversion result corresponding to the input video.

Write more details about machine learning processing. The estimated word string output by the neural network 301 can be compared with the correct word string that is correct data. The correct word string is supplied by the learning data supply unit 80 in a form corresponding to the input video. The loss calculation unit 50 calculates the loss from the estimated word string output by the neural network 301 and the correct word string supplied from the learning data supply unit 80. Based on the loss calculated by the loss calculation unit 50, the neural networks 201 and 301 perform error backpropagation and update internal parameters.

FIG. 3 is a schematic diagram showing the flow of learning processing using the second encoder section 60, which is an additional network. The second encoder section 60 is used only for the purpose of assisting the learning process of the encoder section 20. In other words, the second encoder section 60 is used only in the learning mode and not in the conversion mode.

As already explained, when the conversion device 1 operates in the learning mode, the encoder unit 20 inputs a series of frame images and outputs a state vector. When the conversion device 1 operates in the learning mode, a correct word string is also input to the second encoder section 60. This correct word string is supplied by the learning data supply section 80. As shown in the figure, the correct word string is a string of words such as word ₁ , word ₂ , . . . , word _u-1 , word _u . Additionally, the special symbol <bos> (beginning of sentence) is added to the beginning of the correct word string, and the special symbol <eos> (end of sentence) is added to the end of the correct word string. sentence) is added. The second encoder section 60 has a neural network 601 inside. The neural network 601 calculates and outputs a state vector based on the input correct word string and internal parameters at that time.

The second loss calculation unit 70 obtains the state vector output by the encoder unit 20 and the state vector output by the second encoder unit 60, and calculates a loss (second loss) from both of these vectors. Each of the neural networks 201 and 601 performs error backpropagation based on the loss calculated by the second loss calculation unit 70, and updates each internal parameter.

That is, the encoder unit 20 adjusts internal parameters by performing error backpropagation based on the loss calculated by the second loss calculation unit 70. This learning process enables the encoder unit 20 to output a good state vector based on the input video. As shown in FIG. 2, the encoder unit 20 also performs error backpropagation based on the loss calculated by the loss calculation unit 50. However, the path of backpropagation when error backpropagation is performed based on the loss calculated by the loss calculation unit 50 is relatively long, and the path of backpropagation when error backpropagation is performed based on the loss calculated by the second loss calculation unit 70 is relatively long. The path of propagation is relatively short. In other words, even if sufficient machine learning effect cannot be obtained by backpropagation of the error based on the loss calculated by the loss calculation unit 50 because the path is too long, the error based on the loss calculated by the second loss calculation unit 70 By using backpropagation in combination, the encoder unit 20 can perform better learning.

That is, by using the second encoder section 60, which has a configuration unique to this embodiment, the learning effect of the encoder section 20 can be improved.

FIG. 4 is a block diagram showing a more detailed configuration example of the encoder section 20. As shown in FIG. As illustrated, the encoder unit 20 is configured to include a recurrent neural network (RNN) therein. In the figure, the temporal recursive structure of the RNN is expanded from left to right. In the illustrated configuration example, the encoder unit 20 has RNNs from the first layer to the Nth layer corresponding to each frame of the input frame image sequence frame ₁ , frame ₂ , . . . , frame _r . N is a positive integer. For example, N may be set to a value of approximately 2 or more and 4 or less. However, N is not limited to the range illustrated here. To configure the encoder unit 20, N-layer RNN circuits are sequentially reused as time progresses (as frame images progress). A frame image is input to the first layer RNN. Instead of directly inputting the frame image to the first layer RNN, the frame image is input in advance to a circuit that extracts features such as a CNN (convolutional neural network) (not shown), and the output feature vector is It may also be input to the first layer RNN. The output from the first layer RNN is passed to the second layer RNN corresponding to the same frame image and the first layer RNN corresponding to the next frame image. In addition, the i-th layer (1<i<N) RNN outputs the output from the (i-1) layer RNN corresponding to the same frame image and the i-th layer RNN corresponding to the previous frame image. Receive output. Then, the output from the i-th layer RNN is passed to the (i+1)-th layer RNN corresponding to the same frame image and the i-th layer RNN corresponding to the next frame image. Further, the Nth layer RNN receives an output from the (N-1)th layer RNN corresponding to the same frame image and an output from the Nth layer RNN corresponding to the previous frame image. Then, the output from the Nth layer RNN is passed to the Nth layer RNN corresponding to the next frame image. The output from the RNN corresponding to the last frame image (frame _r in FIG. 4) is a state vector. The encoder section 20 passes the generated state vector to the decoder section 30 and the second loss calculation section 70.

As described with reference to FIG. 4, the encoder unit 20 is logically configured using RNNs arranged in a matrix of N rows and r columns. Here, N is the number of layers, and r is the length of the sequence of input images.

FIG. 5 is a block diagram showing a more detailed configuration example of the decoder section 30. As shown in FIG. As illustrated, the decoder section 30 includes an RNN therein. In the figure, the temporal recursive structure of the RNN is expanded from left to right. In the illustrated configuration example, the decoder unit 30 outputs word strings (estimated word strings) word ₁ , word ₂ , . . . , word _u-1 , word _u , and corresponding to each symbol of <eos>, It has an RNN from the first layer to the Nth layer. The value of N here is matched to the value of N of the encoder section 20 (see FIG. 4). That is, the decoder section 30 is logically configured using RNNs arranged in a matrix of N rows and (u+1) columns, similar to the internal configuration of the encoder section 20. The flow of data exchange within the RNN matrix in the decoder section 30 is also similar to that within the RNN matrix in the encoder section 20. Here, (u+1) is the length of the output sequence. However, the length of this output series may include a special symbol such as <eos>.

The decoder unit 30 obtains the state vector generated by the encoder unit 20 as input data. Further, the N-th layer RNN of the decoder unit 30 sequentially outputs estimated word sequences (word ₁ , word ₂ , . . . , word _u-1 , word _u , and <eos>). The decoder unit 30 passes the generated estimated word string to the output unit 40 and the loss calculation unit 50.

FIG. 6 is a block diagram showing a more detailed configuration example of the second encoder section 60. As illustrated, the second encoder section 60 is configured to include an RNN therein. In the figure, the temporal recursive structure of the RNN is expanded from left to right. In the illustrated configuration example, the second encoder unit 60 corresponds to the input correct word strings <bos>, word ₁ , word ₂ , . . . , word _u-1 , word _u , and <eos>, It has an RNN from the first layer to the Nth layer. The value of N here is matched to the value of N of the encoder section 20 (see FIG. 4). That is, the second encoder section 60 is logically configured using RNNs arranged in a matrix of N rows and (u+2) columns, similar to the internal configuration of the encoder section 20. The flow of data exchange within the RNN matrix in the second encoder section 60 is also similar to that within the RNN matrix in the encoder section 20. Here, (u+2) is the length of the output sequence. However, the length of this output series may include special symbols such as <bos> and <eos>.

The second encoder unit 60 obtains as input the correct word string data passed from the learning data supply unit 80 . The first layer RNN of the second encoder unit 60 sequentially receives the correct word strings (<bos>, word ₁ , word ₂ , . . . , word _u-1 , word _u , and <eos>). The second encoder unit 60 passes the state vector generated based on the correct word string to the second loss calculation unit 70.

FIG. 7 is a flowchart illustrating an example of a procedure when the conversion device 1 performs machine learning processing. Below, the procedure of the learning process will be explained with reference to this flowchart.

In step S101, the learning data supply unit 80 supplies a pair of input and output data as learning data. The input data is video data. The learning data supply unit 80 passes the input data to the encoder unit 20 as a series of frame image data. The output data is correct word string data. The learning data supply unit 80 passes the correct word string, which is output data, to the second encoder unit 60 and the loss calculation unit 50.

Next, in step S102, the encoder unit 20 performs forward propagation based on the series of frame image data passed in step S101. The encoder unit 20 outputs a state vector as a result of forward propagation.

Next, in step S103, the second encoder unit 60 performs forward propagation based on the data of the correct word string passed in step S101. The second encoder unit 60 outputs a state vector as a result of forward propagation.

Next, in step S104, the second loss calculation unit 70 calculates, based on the state vector output from the encoder unit 20 (step S102) and the state vector output from the second encoder unit 60 (step S103), Calculate loss.

Next, in step S105, the encoder unit 20 performs error backpropagation based on the loss calculated by the second loss calculation unit 70 in step S104. Through this error backpropagation, the encoder unit 20 updates internal parameters.

Next, in step S106, the second encoder unit 60 performs error backpropagation based on the loss calculated by the second loss calculation unit 70 in step S104. Through this error backpropagation, the second encoder section 60 updates internal parameters.

As described above, the series of processes from steps S102 to S106 are based on the difference between the output of the encoder section 20 and the output of the second encoder section 60, and the neural network that each of the encoder section 20 and the second encoder section 60 has internally is executed. This is the process of adjusting parameters. In other words, this is the second learning mode processing described above.

Next, in step S107, the encoder unit 20 performs forward propagation based on the series of frame image data passed in step S101. The encoder unit 20 outputs a state vector as a result of forward propagation. The encoder unit 20 passes the state vector generated in this step to the decoder unit 30.

Next, in step S108, the decoder unit 30 performs forward propagation based on the state vector output by the encoder unit 20 in step S107. As a result, the decoder unit 30 outputs a word string (estimated word string). This estimated word string may include special symbols such as <eos>.

Next, in step S109, the loss calculation unit 50 calculates a loss based on the correct word string data passed in step S101 and the estimated word string data obtained in step S108.

Next, in step S110, the decoder unit 30 performs error backpropagation based on the loss calculated in step S109. Through this error backpropagation, the decoder unit 30 updates internal parameters.

Next, in step S111, the encoder unit 20 performs error backpropagation of the neural network included in the encoder unit 20, as an extension of the error backpropagation process of the decoder unit 30 in step S110. Through this error backpropagation, the encoder unit 20 updates internal parameters.

The series of processes from steps S107 to S111 described above are based on the difference between the estimated word string obtained by the forward propagation process of the encoder section 20 and decoder section 30 and the correct word string given from the learning data supply section 80. , is a process of adjusting the parameters of the neural network that each of the encoder section 20 and the decoder section 30 has internally. That is, this is the process of the first learning mode described above.

In step S112, the control unit 90 determines whether the machine learning process using all the learning data has been completed. If all the learning data has been processed (step S112: YES), the process advances to the next step S113. If there is still learning data (input/output data pair) remaining (step S112: NO), the process returns to step S101 to process the next data.

If the process proceeds to step S113, the control unit 90 determines whether or not the learning process using the current set of learning data has been repeated a predetermined number of times. Note that this number of times is, for example, predetermined. If the predetermined number of processes have been completed (step S113: YES), the entire process of this flowchart is ended. If the predetermined number of processes have not been completed (step S113: NO), the process returns to step S101 to perform the next process. Note that in this step, instead of determining whether or not to end the entire process based on a predetermined number of times, the determination may be made based on another determination criterion. As an example, it may be determined whether or not to end the entire process based on the convergence status (whether or not the values of the parameter set of the neural network to be updated are sufficiently converged).

The learning of the encoder section 20 and the decoder section 30 progresses through the above processing procedure. Learning adjusts the internal parameters of the encoder unit 20 and decoder unit 30, so that the encoder unit 20 and decoder unit 30 can process input data (for example, a series of images) more accurately. For example, a video representing sign language. Become.

In the procedure described above, the parameters of the encoder section 20 are not only updated based on the loss calculated by the loss calculation section 50, but also the parameters of the encoder section 20 are updated based on the loss calculated by the second loss calculation section 70. Update. The second loss calculating section 70 generates the difference between the state vectors calculated by the encoder section 20 and the second encoder section 60 as a loss. This method allows the encoder unit 20 to learn more effectively. In other words, the state vector generated by the encoder unit 20 better represents the relationship between the input video and the correct word string. Therefore, the conversion device 1 is expected to generate a highly accurate estimated word string corresponding to the input video.

In the procedure shown in FIG. 7, learning based on the loss calculated by the second loss calculation unit 70 (learning of the encoder unit 20 and the second encoder unit 60 from steps S102 to S106, second learning mode) and loss calculation Learning based on the loss calculated by the unit 50 (learning of the encoder unit 20 and decoder unit 30 from steps S107 to S111, first learning mode) is performed individually and alternately. This is an example of mode switching by the control unit 90 described above. That is, during the learning process, the control unit 90 repeatedly executes the first learning mode and the second learning mode for each pair of learning input data and correct answer data supplied by the learning data supply unit 80. Control. However, both of these learning may be performed simultaneously on the calculation graph.

[Example of evaluation experiment]
An evaluation experiment was conducted on the conversion device according to the above embodiment. The results are described below.

The conditions of the experiment were as follows. As a CNN (convolutional neural network), parameters learned using the ImageNet data set using AlexNet were used as initial values. Furthermore, as the RNN, a 4-layer, 1000-unit residual GRU (Gated Recurrent Unit) was adopted for all of the encoder section 20, decoder section 30, and second encoder section 60.

Note that the conversion device to be compared is a conversion device using a conventional technique. In other words, the conversion device to be compared does not have the second loss calculation unit 70 and does not perform error backpropagation based on the loss calculated by the second loss calculation unit 70. Further, therefore, the conversion device to be compared does not have the second encoder unit 60.

As learning data used in the evaluation experiment, 16,000 pairs of input video (sign language video) and correct word strings based on gross expressions were used. Similarly, 1000 pairs of input video and correct word strings based on gross representation were used as evaluation data. The input video used as learning data is a video of the Japan Broadcasting Corporation's sign language news (broadcast from 2009 to 2015). The target domain is weather information and weather-related topics. The maximum length of the input video is 10 seconds per sentence. The frame rate of the input video is 29.97 frames per second. The size of the image is 256 pixels x 256 pixels, and the image includes the upper body, right hand, and left hand. The number of vocabulary is 6695 words.

FIG. 8 is a graph showing the results of the evaluation experiment. The graph in the figure shows the correspondence between the number of learning data and the word error rate (conversion error rate) for each of the conventional technique and the technique of this embodiment. The horizontal axis of the graph is the total number of learning data (unit: number of sentences). The vertical axis of the graph is the word error rate. In both methods, the word error rate tends to decrease as the number of training data increases. However, in all areas of the number of training data, the word error rate is lower when using the method of this embodiment than when using the conventional technique. The minimum word error rate using the prior art method was 0.424. On the other hand, the minimum word error rate when using the method of this embodiment was 0.403. In other words, it was confirmed that by using this embodiment, a lower word error rate could be achieved.

Note that at least some of the functions of the conversion device in the embodiments described above can be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed. Note that the "computer system" herein includes hardware such as an OS and peripheral devices. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, magneto-optical disks, ROM, CD-ROM, DVD-ROM, and USB memory, and storage devices such as hard disks built into computer systems. Say something. Furthermore, a "computer-readable recording medium" refers to a medium that temporarily and dynamically stores a program, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In that case, it may also include something that retains a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client. Further, the program may be one for realizing a part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

Although the embodiments have been described above, the present invention can be further implemented in the following modifications. For example, the content shown in the input video may be other than sign language. A conversion device that outputs symbol strings based on images of movements such as arbitrary gestures (not limited to movements of people or living things), not limited to sign language, may be implemented. Furthermore, the word string to be output is not limited to gloss notation. The word string to be output may be an arbitrary linguistic expression or a more general symbol string. Furthermore, input data is not limited to video. For example, it may be any series data. Furthermore, the method for machine learning used by the encoder section 20, decoder section 30, and second encoder section 60 is not limited to neural networks. That is, instead of a neural network, any means that can perform machine learning based on learning data may be used.

As described above, according to the present embodiment (including modified examples), the conversion device 1 includes a second encoder that is configured using a neural network and generates a state vector based on the input correct symbol string. section 60, and a second loss calculation section 70 that calculates the difference between the state vector generated by the encoder section 20 and the state vector generated by the second encoder section 60. Then, based on the loss calculated by the second loss calculation unit 70, backpropagation of errors in the encoder unit 20 and the second encoder unit 60 can be performed. In other words, the machine learning process of the encoder unit 20 can be performed using a relatively short error backpropagation path. Thereby, the learning effect is clearly expressed, and the conversion accuracy of the conversion device 1 is improved. Alternatively, the learning cost of the conversion device 1 can be reduced.

Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configuration is not limited to these embodiments, and includes designs within the scope of the gist of the present invention.

The present invention can be used, for example, in any application area that generates a symbol string based on a video (an example is video understanding, etc.). In particular, when processing sign language videos, it can be used for communication between hearing-impaired people and normal hearing people, and for the education of sign language learners. However, the scope of use of the present invention is not limited to what is exemplified here.

1 Conversion device 10 Input section 20 Encoder section 30 Decoder section 40 Output section 50 Loss calculation section 60 Second encoder section 70 Second loss calculation section 80 Learning data supply section 90 Control section 201, 301, 601 Neural network

Claims (6)

an encoder section that generates state data based on input data;
a decoder unit that generates output data based on the state data;
a learning data supply unit that supplies a pair of learning input data that is input to the encoder unit and correct answer data that is a correct answer of the output data that corresponds to the learning input data;
state data generated by the encoder unit based on the learning input data; learning output data generated by the decoder unit based on the learning output data; and the learning output data supplied by the learning data supply unit in response to the learning input data. a loss calculation unit that calculates a loss representing the difference between the correct data and the
a second encoder unit that generates estimated state data that is state data estimated based on the correct data;
The state data is generated by the encoder unit based on the learning input data, and the second encoder unit generates the state data based on the correct data supplied by the learning data supply unit corresponding to the learning input data. a second loss calculation unit that calculates a second loss representing the difference between the estimated state data and the estimated state data;
a control unit that controls the operation by appropriately switching between a first learning mode, a second learning mode, and a conversion execution mode;
Equipped with
In the first learning mode, the inside of the encoder section and the decoder section is calculated based on the loss calculated by the loss calculation section based on the learning input data supplied by the learning data supply section and the correct answer data. Adjust the parameters and
In the second learning mode, the encoder section and the Adjust the internal parameters of the second encoder section,
In the conversion execution mode, the encoder section generates state data based on input data, and the decoder section generates output data based on the state data generated by the encoder section.
conversion device. Each of the encoder section, the decoder section, and the second encoder section includes a neural network therein,
In the first learning mode, the internal parameters of the encoder section and the decoder section are adjusted by performing error backpropagation of the respective neural networks of the encoder section and the decoder section based on the loss;
In the second learning mode, the internal parameters of the encoder section and the second encoder section are determined by performing error backpropagation of the respective neural networks of the encoder section and the second encoder section based on the second loss. adjust,
The conversion device according to claim 1. The control unit repeatedly executes the first learning mode and the second learning mode for each pair of the learning input data and the correct answer data supplied by the learning data supply unit during the learning process. to control,
The conversion device according to claim 1 or 2. The input data is a series of images;
the output data is a series of predetermined symbols;
Conversion device according to any one of claims 1 to 3. The series of images is a series of images representing sign language,
the series of symbols is a series of words in glossary notation corresponding to the sign language;
The conversion device according to claim 4. an encoder section that generates state data based on input data;
a decoder unit that generates output data based on the state data;
a learning data supply unit that supplies a pair of learning input data that is input to the encoder unit and correct answer data that is a correct answer of the output data that corresponds to the learning input data;
state data generated by the encoder unit based on the learning input data; learning output data generated by the decoder unit based on the learning output data; and the learning output data supplied by the learning data supply unit in response to the learning input data. a loss calculation unit that calculates a loss representing the difference between the correct data and the
a second encoder unit that generates estimated state data that is state data estimated based on the correct data;
The state data is generated by the encoder unit based on the learning input data, and the second encoder unit generates the state data based on the correct data supplied by the learning data supply unit corresponding to the learning input data. a second loss calculation unit that calculates a second loss representing the difference between the estimated state data and the estimated state data;
a control unit that controls the operation by appropriately switching between a first learning mode, a second learning mode, and a conversion execution mode;
Equipped with
In the first learning mode, the inside of the encoder section and the decoder section is calculated based on the loss calculated by the loss calculation section based on the learning input data supplied by the learning data supply section and the correct answer data. Adjust the parameters and
In the second learning mode, the encoder section and the Adjust the internal parameters of the second encoder section,
In the conversion execution mode, the encoder section generates state data based on input data, and the decoder section generates output data based on the state data generated by the encoder section.
A program that allows a computer to function as a converter.

Images