JP6754154B1 - Translation programs, translation equipment, translation methods, and wearable devices - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique capable of speeding up a translation process when translating a word into a word of another language based on the movement of the lip when the speaker utters the word. SOLUTION: In a translation program capable of translating and outputting a word spoken by a speaker into a word in another language having the same meaning as the word, a computer, a word spoken by the speaker, and a lip when the word is spoken. It functions as a translation unit that cannot be read from the movement and can be translated and output into the words of the other language associated with the movement of the lip. [Selection diagram] Fig. 4

Description

The present invention relates to a technique for translating words spoken by a speaker into another language, and more particularly to a translation program, a translation device, a translation method, and a wearable terminal that execute translation processing based on the movement of the speaker's lips. is there.

Research and development of technology to make the words spoken by the speaker recognized by a computer, not by humans, is being conducted.
For example, there is known a technique of analyzing and recognizing a word spoken by a speaker by analyzing an image of the speaker.
Specifically, there is known a video analysis technique (lip reading technique) that identifies the movement of the speaker's lip from the image and reads the words spoken by the speaker based on the specified lip movement (for example, lip reading technique). See Patent Document 1).
Based on such video analysis technology, there is a technology that can perform translation by outputting the speaker's language (Japanese) that can be read from the movement of the speaker's lips and other languages (English) that correspond to it. It has been proposed (see, for example, Patent Document 2).

Japanese Unexamined Patent Publication No. 2004-15250 Japanese Unexamined Patent Publication No. 2013-45282

By the way, conventionally, the translation process into another language based on the movement of the speaker's lips has a problem in the following points.
For example, when the speaker utters the word "I have a pen" in English, the movement of the speaker's lips is identified by video analysis, and the speaker says "I have a pen" from the identified lip movement. Recognizing what the speaker said, that is, "I have a pen" in English, then translated this English "I have a pen" into Japanese "I have a pen." Processing is performed.
However, in this case, firstly, the word "English" is recognized from the movement of the speaker's lip, and secondly, the recognized "English" word is translated into the word "Japanese". Processing and processes are required, which has been a cause of delay in translation processing.
In other words, if "Japanese" can be translated directly from the movement of the speaker's lips, there will be no delay problem, but useless processing such as recognizing the word "English" once from the movement of the speaker's lips. There was a problem of delay due to the fact that there was a process.

In addition, the interpreting system of Patent Document 2 associates the text data of Japanese "warm" corresponding to the movement (feature amount) of the lips with the text data of English "It is warm" corresponding to this in advance. By registering the mode database, it is possible to output both Japanese and English when the speaker says "It's warm" in Japanese.
However, in this case, the database is composed of a plurality of data such as Japanese text data and English text data corresponding thereto, and is configured to include useless data.
In other words, if the "Japanese" can be translated directly from the movement of the speaker's lips, the database contains "English" text data that is not necessary, which wastefully occupies the storage area of the storage unit. I had no choice but to prepare more storage capacity than necessary.

As described above, the conventional translation technique for other languages based on the movement of the speaker's lips is not configured to directly translate the movement of the speaker's lips into another language, which delays the translation. And various problems related to storage capacity have occurred.
This is because, as is the case with lip-reading technicians, the movement of the lips of a speaker (for example, an English speaker) can read the words of the language (English) actually spoken by the speaker, Words in other languages (Japanese) different from the language spoken by the speaker cannot be read from the movement of the speaker's lips, in other words, the speaker actually speaks from the movement of the speaker's lips. The background is that it is impossible to read words in other languages.
In other words, due to such common general knowledge, many people, including those skilled in the art, have not been able to come up with a composition of translation processing that directly links the movement of the speaker's lips with other languages, which delays the process. And various problems related to storage capacity were latent.

The present invention has been made in view of the above circumstances, and is a translation program, a translation device, a translation method, and a wearable terminal that directly translates into another language based on the movement of the speaker's lips. The purpose is to provide.

In order to achieve the above object, the translation program of the present invention is a translation program capable of translating a word spoken by a speaker into a word in another language having the same meaning as the word, when the speaker speaks a word on a computer . A storage unit that stores video information that can identify the movement of the lips and information of words in other languages corresponding to the words in association with each other, and the movement of the lips that is stored in the storage unit can be specified. A translation model generator that generates a trained translation model by learning teacher data that inputs video information and outputs information of words in other languages, and words spoken by the speaker on the speaker's lips. By inputting video information that can specify the movement into the translation model , it functions as a translation unit that translates into the words of the other language output from the translation model .

According to the present invention, translation into another language based on the movement of the speaker's lips can be performed at high speed.

It is an external view of the wearable terminal which is an example of the translation apparatus of this invention. It is a block diagram which shows the hardware configuration of a translation apparatus. It is a block diagram which shows the software structure of a translation apparatus. This is an example of a data set (teacher data) used for machine learning. It is a schematic diagram which shows the trained translation model. It is a figure which shows the display example of the translation information. It is a flowchart which shows the processing procedure which concerns on the translation method of this invention. It is a figure which shows an example of the translation apparatus composed of a plurality of apparatus. (A) is an example of a translation device composed of a wearable terminal and a smartphone, and (b) is an example of a translation device composed of a server and various terminal devices.

An embodiment of the translation apparatus 1 of the present invention will be described with reference to the drawings.
FIG. 1 is an external view of a wearable terminal which is an example of the translation device 1 of the present invention.
As shown in FIG. 1, the translation device 1 of the present invention can be configured by, for example, a glasses-type wearable terminal provided with a photographing device 16.
In the present embodiment, while the user who is the listener is wearing the wearable terminal and the speaker facing the other is photographed by the photographing device 16, the words spoken by the speaker are translated by the wearable terminal and the translation result is displayed. (Simultaneous interpretation) is assumed.
Further, in the present embodiment, it is assumed that the speaker is an American, the user is a Japanese, and the English words spoken by the speaker are translated into Japanese.
The translation device 1 can also be applied to an information processing terminal such as a smartphone, a tablet terminal, or a personal computer.

FIG. 2 is a block diagram showing a hardware configuration of a wearable terminal which is an example of the translation device 1.
As shown in FIG. 2, the translation device 1 is a computer including a processor 11, a memory 12, a storage 13, an input device 14, an output device 15, a photographing device 16, and a communication device 17.
The processor 11 includes a central processing unit (CPU) including a control unit, a calculation unit, a register, and the like, and controls the entire computer.
The processor 11 reads a program, data, and the like from the storage 13 and the communication device 17 into the memory 12, and executes various processes according to these.
The memory 12 is a computer-readable recording medium, such as a ROM, EPROM, EEPROM, or RAM.

The storage 13 is a computer-readable recording medium, such as a hard disk drive or a flash memory.
The storage 13 functions as a storage unit of the present invention.
The input device 14 is an input device (for example, a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives an input from the outside.
The output device 15 is an output device (for example, a monitor, a display, a display panel, a speaker, an LED lamp, etc.) that outputs to the outside.
The wearable terminal is provided with a projection device as the output device 15, and the translation information is displayed on the lens 151 by projecting the translation information from the projection device (see FIGS. 1 and 6). ..

The photographing device 16 is a camera and supplies the photographed video information to the processor 11.
The photographing device 16 is provided so as to be able to photograph an image in front of the user wearing the wearable terminal.
In the present embodiment, the image of the face of the speaker (at least the image including the lip) in front of the user can be captured while the user is wearing the wearable terminal.
The photographing device 16 may be an external camera.
Further, the photographed information of the speaker can be taken in from the outside and stored in the storage 13.
The communication device 17 is hardware (transmission / reception device) for performing communication between computers via a wired and / or wireless communication line, and is, for example, a network device, a network controller, a network card, a communication module, or the like.
In the glasses-type wearable terminal of the present embodiment, the respective parts are provided, for example, in a manner built in or externally attached to the temple part.

In the translation device 1, the processor 11 executes a program (translation program of the present invention) to control each part, thereby realizing the functions described below.
For example, a translation program is executed in response to an operation of the input device 14 such as a button, and shooting by the photographing device 16 is started.
FIG. 3 is a block diagram showing a software configuration of the translation device 1.
As shown in FIG. 3, the translation device 1 includes a video input unit 21, a translation model 22, an output unit 23, and a translation model generation unit 24.

The video input unit 21 inputs video information of the speaker's face shot by the shooting device 16.
Specifically, an image of the speaker's lip region, that is, an image capable of identifying the movement of the speaker's lip is extracted from the image of the speaker's face taken by the photographing device 16 and input to the translation model 22. ..
The "video" input to the translation model 22 is a moving image showing the movement of the speaker's lips, and is a plurality of still images having time series elements.

The translation model 22 is a program generated by the translation model generation unit 24.
In the generation of the translation model 22, as a preprocessing, a process of storing a large number of data sets (teacher data) used for machine learning in the storage 13 which is a storage unit is performed.
The data set is video information that can identify the movement of the lips when the speaker utters a word, and other languages that cannot be read from the movement of the lips, and is a word of another language corresponding to the word (that is, that is). , A data structure in which information in another language having the same meaning as the above-mentioned word) is associated with the information.
The term "unreadable" means "unreadable" based on common general technical knowledge, but is not limited to this, and includes "difficult to read" and "extremely difficult to read".
The "same meaning" is not limited to the exact same meaning, but includes "almost the same meaning" and "approximately the same meaning".
The "other language" is a language different from the language actually spoken by the speaker. For example, when the speaker speaks a word in English, it means a language other than English (for example, Japanese).
In other words, "another language" corresponds to "objective language" in the relationship between "original language (language before translation)" and "objective language (language after translation)" in translation.
A "word" includes not only a phrase or sentence (sentence) that combines a plurality of words, but also the word itself.

FIG. 4 is an example of a data set (teacher data).
As shown in FIG. 4, the data set of the present embodiment includes (a) video information (input data) that can identify the movement of the lips when an English speaker speaks a word, and (b) Japan of that word. It is composed of character information (output data) of words.
That is, the data set shown in FIG. 4 is a data set for the purpose of translating from English to Japanese.
For example, No. The data set of 1 is (a) video information obtained by preliminarily capturing the movement of the lips when an English speaker says "I have a apple." In English, and (b) the Japanese text information "I am." I have an apple. "
No. The data set of 2 is (a) video information obtained by preliminarily capturing the movement of the lips when an English speaker says "I have a pen." In English, and (b) the Japanese text information "I am." I have a pen. "
In such a data set, for example, a simultaneous interpreter is made to perform English translation while showing the video information (including the speaker's voice) of (a), and the translation result is used as the character information of (b) (a). ) Is linked with the video information.
As a result, it can be expected that the intuitive translation function and translation technology of the simultaneous interpreter will be transplanted.
In addition to this, for example, a known application program capable of automatically translating a language spoken by a speaker into another language by voice recognition is used, and the translation result of the program is used as the information in (b). You can also.
A plurality (many) patterns of such a data set are stored for one word, and various words are stored.
Although FIG. 4 is an example of a data set related to a phrase or a sentence, a data set related to a word can also be included.

FIG. 4 is a data set for translating English, which is the original language (language before translation), into Japanese, which is the target language (language after translation).
Therefore, various translations can be supported by changing the constituent data of the data set according to the original language and the target language.
For example, when the purpose is to translate from Japanese (original language) to English (target language), (a) is video information that can identify the movement of the lips when uttering Japanese words, and (b) is the video information. A data set for English textual information of words may be prepared.
In addition, when the purpose is to translate from English (original language) to French (target language), (a) is used as video information that can identify the movement of the lips when the English word is spoken, and (b) is the video information of that word. A data set for French character information may be prepared.

The translation model generation unit 24 generates the trained translation model 22 by machine learning the data set stored in the storage 13 as teacher data.
Machine learning shall be performed using a known neural network such as deep learning.
Specifically, in the neural network, by inputting the data of FIG. 4 (a) to the input layer, the data of FIG. 4 (b) is output from the output layer with a high probability (for example, 90% or more). To learn.
That is, when video information that can identify the movement of the lips when the speaker utters a word is input to the input layer, the probability of the information of the word in another language corresponding to the word is derived with high probability in the output layer. The trained translation model 22 is generated by executing the optimization process such as the weighting coefficient in each neuron so as to be performed.
The object of machine learning is the movement of the lips represented by a plurality of still images (videos) having a time series element, and since it is a word having a time series element represented by the movement of the lips, it is a known CNN. (Convolutional neural network), RNN (recurrent neural network), LSTM, etc. are used.

The translation model 22 is a function of the translation unit, and when video information that can identify the movement of the speaker's lips is input, the information of words in another language corresponding to the words is output as translation information.
Therefore, by inputting the video information into the translation model 22 while shooting the video of the speaker with the shooting device 16, when the speaker utters any words, the movement of the speaker's lips is used as the basis. So-called simultaneous interpretation, which outputs words in other languages as translation information, becomes possible.
FIG. 5 is a schematic diagram of the translation model 22 generated by the translation model generation unit 24.
As shown in FIG. 5, the translation model 22 is composed of a multi-layer neural network having an input layer, an intermediate layer, and an output layer.
FIG. 5 shows that when an English speaker inputs video information that can identify the movement of the lips when he / she says “I have a pen.” To the input layer, “I have an apple” in the output layer. "Is output (classified).
That is, when the speaker utters a word, the translation model 22 responds to the word by inputting video information that can identify the movement of the lips when the speaker utters the word into the translation model 22. Words in other languages are output as translation information.
In addition, in "when the speaker utters a word, the movement of the lips when the speaker utters the word", if the speaker utters a word, the lip movement becomes such a movement of the lip. Including the movement of.
Therefore, it is possible to deal with the so-called lip-synching in which the speaker moves only the mouth without speaking a word.
Further, the translation model 22 can be made to input the video information stored in the storage 13 and the video information received from the outside by the communication device 17.
Therefore, not only simultaneous interpretation but also translation based on the movement of the speaker identified from the past video information is possible.

The output unit 23 causes the output device 15 to output the translation information output from the translation model 22.
In the wearable terminal of the present embodiment, translation information can be displayed on the lens 151 by projection by a projection device which is an example of the output device 15 (see FIG. 1).
FIG. 6 shows an example in which "I have an apple" is displayed on the lens 151 as translation information.
In this case, the user can see the translation information displayed on the lens 151 while wearing the wearable terminal.
Therefore, for example, a wearable terminal can be used for simultaneous interpretation, and smooth conversation with a speaker can be supported.
When the translation device 1 is a personal computer, a smartphone, or a tablet terminal, the translation information can be displayed on a display or a monitor.
The output unit 23 can also store the translation information in the storage 13.
As a result, for example, the minutes of a meeting or an interview can be automatically recorded.
The output unit 23 can also transmit the translation information to the outside by the communication device 17.

Next, the translation method of the present invention will be described.
FIG. 7 is a flowchart showing a processing procedure according to the translation method of the present invention.
As shown in FIG. 7, first, video information that can identify the movement of the speaker's lips is input (S1). This is a step corresponding to the previous step of the present invention.
Next, the information of words in other languages associated with the movement of the lips is output as translation information (S2). This is a step corresponding to the subsequent step of the present invention.
Specifically, by inputting video information that can identify the movement of the speaker's lips into the translation model 22, the translation model 22 uses the information of words in other languages corresponding to the movement of the lips as translation information. Output.

That is, in the translation method of the present invention, when a speaker utters a word, a pre-process of inputting video information capable of identifying the movement of the speaker's lips and the movement of the lips input by the pre-process are input. It has two major steps, a post-process of translating the words spoken by the speaker into the words of the other language based on the identifiable video information.
That is, in the prior art, the language of the original language read in the intermediate process is obtained through the intermediate process of reading the words of the original language uttered by the speaker from the video information that can identify the movement of the lips input in the previous process. Since it is assumed that the language is translated into a language of another language, the translation method of the present invention does not go through such an intermediate process, that is, directly from the movement of the lips related to the original language input in the previous process. Is output as translation information.
According to such a translation method of the present invention, unnecessary steps can be omitted, so that the translation process can be speeded up.

Here, the "correlation" between the input data and the output data will be described.
In short, the correlation between (a) "movement of the lips" (input data) and (b) "words of other languages" (output data) will be described (see FIG. 4).
Of these, the "words of other languages" in (b) are Japanese words that cannot be read from the movement of the lips (a) when an English speaker speaks a word in English, for example. Since it is a Japanese word having the same meaning as an English word that can be read from (a), (a) and (b) have a correlation.
Also, even if it is possible to read the words in the original language from the movement of the speaker's lips, it is possible to read the words in another language directly from the movement of the speaker's lips (that is, the speaker actually speaks). This correlation is new and characteristic because the idea of reading a non-existent language (other language) is beyond the common technical wisdom.
The present invention uses such characteristicly correlated input data and output data to directly translate the words spoken by the speaker into another language from the movement of the lips.
That is, the present invention conventionally has a problem of delay caused by unnecessary processing or process such as recognizing the word of "original language" from the movement of the speaker's lip in the translation processing, and registration processing. , It solves the problem of storage capacity etc. by including unnecessary "original language" data in the registered information.
According to the present invention as described above, it is possible to speed up the translation process and the like.

In addition, conventionally, a plurality of methods such as first recognizing a word of "original language" from "movement of lips" and secondly translating the recognized "word of original language" into "word of another language". Since the translation process is performed via the process or process, if there is a misrecognition or mistranslation in either the first process or the second process, the final translation accuracy is lowered.
For this reason, conventionally, it is necessary to register appropriate data for each registered data corresponding to a plurality of processes, and if each registered data is not appropriate, there is a problem that the accuracy of translation is lowered.
On the other hand, according to the present invention, that is, the "correlation" according to the present invention, such a conventional problem does not occur.

(Other embodiments)
The translation device 1 can also be realized by combining a plurality of devices.
FIG. 8 is a diagram showing an example of a translation device 1 composed of a plurality of devices. (A) is an example of a translation device 1 composed of a wearable terminal and a smartphone, and (b) is an example of a translation device 1 composed of a server and various devices.
As shown in FIG. 8, the translation device 1 can be realized by connecting a plurality of devices so that data communication is possible via a predetermined communication line.
The translation device 1 shown in FIG. 8A has a configuration in which a wearable terminal and a smartphone or the like can communicate with each other via Bluetooth (registered trademark).
According to this configuration, for example, a translation program and a function (software configuration) executed by the program can be provided on the smartphone side, and on the wearable terminal side, it is as simple as displaying a photograph of the speaker and the translation result. Can be configured.
FIG. 8B shows a configuration in which a server in cloud computing and various terminal devices can communicate with each other via an Internet line or a mobile communication line (5G or the like).
According to this configuration, for example, a translation program and a function (software configuration) executed by the program can be provided on the server side, and the terminal device side is as simple as displaying a photograph of a speaker and a translation result. Can be configured.
In particular, by using high-speed, large-capacity, low-delay 5G, while realizing a simple configuration on the terminal device side, stress-free and smooth translation processing can be performed as if it were being executed by the terminal device alone. It can be carried out.

As described above, the translation program of the present invention is a computer in another language that cannot be read from the movement of the lips when the speaker speaks the words when the speaker speaks the words. It functions as a translation unit that outputs information on words in another language associated with the information on the words as translation information.
Specifically, the computer uses video information that can identify the movement of the lips when the speaker utters a word, and other languages that cannot be read from the movement of the lips and that correspond to the words. A storage unit that stores the information of the words in association with each other, and teacher data stored in the storage unit that inputs the video information that can identify the movement of the lips and outputs the information of the words of the other language. It functions as a translation model generation unit 24 that generates a trained translation model 22 by learning the above, and the translation unit is an image capable of identifying the movement of the speaker's lips when the speaker speaks. By inputting the information into the translation model 22, the information of the words in the other language output from the translation model 22 is output as the translation information.
Further, the translation device 1 of the present invention is a computer, and the processor 11 can make the translation unit (translation model 22), the storage unit, and the translation model generation unit 24 function by executing a translation program. ..
Further, in the translation method of the present invention, when a speaker utters a word, a pre-step of inputting video information capable of identifying the movement of the speaker's lip and a pre-step of inputting the lip movement input by the pre-step. The post-process has a post-process of translating the words uttered by the speaker into the words of the other language based on the video information that can identify the above, and the post-process is the movement of the lips input by the pre-step. The words spoken by the speaker are translated into words in the other language without going through the intermediate process of reading the language spoken by the speaker from the identifiable video information.

According to the present invention, since the word is directly translated into a word of another language based on the movement of the lip when the speaker utters the word, the delay in the translation process can be eliminated and the speed can be increased.
Specifically, the translation process can be speeded up by simplifying the configuration, processing, and process as compared with the conventional case.
That is, according to the present invention, immediate translation processing becomes possible.
Therefore, for example, according to the present invention, a stress-free simultaneous interpretation service can be realized.
Specifically, when actually meeting and interacting with a foreigner, the dialogue can be facilitated.
In particular, by wearing a wearable terminal as the translation device 1, for example, a meeting or a meeting with a foreigner can be smoothly held.
Further, even when a personal computer or a smartphone is used as the translation device 1, smooth communication can be achieved at an international conference or the like via a Web conference.

The translation program, translation apparatus, and translation method of the present invention have been described above by showing preferred embodiments, but the translation apparatus and the like of the present invention are not limited to the above-described embodiments, and the present invention is not limited to the above-described embodiments. Needless to say, various changes can be implemented within the scope.
For example, the movement of the speaker's lip is specified from the input video, the movement of the lip similar to the movement of the specified lip is extracted from the data set, and another language associated with the extracted movement of the lip is used. Words can also be output as translation information.
In addition, voice information may be output as translation information. In this case, for example, it can be realized by using a known voice reading program capable of converting character information into voice information.

The present invention can be used for automatic translation into other languages, especially simultaneous interpretation.

1 Wearable terminal (translation device)
16 Imaging device 22 Translation model 24 Translation model generator

Claims (5)

In a translation program that can translate a word spoken by a speaker into a word in another language that has the same meaning as the word.
Computer,
A storage unit that stores video information that can identify the movement of the lips when the speaker speaks words and information on words in other languages corresponding to the words in association with each other.
A translation model that generates a learned translation model by learning teacher data stored in the storage unit, which inputs video information that can identify the movement of the lips and outputs information of words in the other language. Generator and
It functions as a translation unit that translates the words spoken by the speaker into the words of the other language output from the translation model by inputting video information that can identify the movement of the speaker's lips into the translation model. A translation program characterized by letting you do it. In a translation program that can translate a word spoken by a speaker into a word in another language that has the same meaning as the word.
Computer,
A storage unit that stores video information that can identify the movement of the lips when the speaker speaks a word and information of a word in another language corresponding to the word in association with each other.
A translation unit that extracts words spoken by a speaker from the memory unit and translates the movements of the lips similar to the movements of the speaker's lips into words in the other language associated with the extracted movements of the lips. A translation program characterized by functioning as . In a translation device that can translate a word spoken by a speaker into a word in another language that has the same meaning as the word.
A storage unit that stores video information that can identify the movement of the lips when the speaker speaks a word and information of a word in another language corresponding to the word in association with each other.
A translation model that generates a learned translation model by learning teacher data stored in the storage unit, which inputs video information that can identify the movement of the lips and outputs information of words in the other language. Generation part and
A translation unit that translates the words spoken by the speaker into the words of the other language output from the translation model by inputting video information that can identify the movement of the speaker's lips into the translation model. A translation device characterized by being equipped . In a translation method that can translate a word spoken by a speaker into a word in another language that has the same meaning as the word.
A process of associating and memorizing video information that can identify the movement of the lips when the speaker utters a word and information of a word in another language corresponding to the word.
A step of generating a trained translation model by learning teacher data stored in the memorizing step, in which video information capable of identifying the movement of the lips is input and information of words in the other language is output.
It has a step of translating a word uttered by a speaker into a word in another language output from the translation model by inputting video information capable of identifying the movement of the speaker's lip into the translation model. A translation method characterized by that. In a wearable terminal capable of operating the program according to claim 1 or 2.
A shooting unit that allows the user to shoot a predetermined range while wearing the wearable terminal,
It is provided with an output unit capable of outputting predetermined information while the user is wearing the wearable terminal.
The translation department
With the user wearing the wearable terminal, the words spoken by the speaker can be translated into the words of the other language based on the movement of the lips when the speaker photographed by the photographing unit speaks the words. ,
The output unit
It is possible to output information on words in other languages translated by the translation unit while the user is wearing the wearable terminal.
A wearable device that features that.

Images