JP6401488B2 - Foreign language conversation understanding support device, foreign language conversation understanding support method and program - Google Patents

Description

  The present invention relates to a technology that supports understanding of conversation in a foreign language.

  2. Description of the Related Art Conventionally, a technique for recognizing the content of a conversation in a foreign language and displaying a character string obtained as a result of the recognition and its translation on a screen is known. For example, in Patent Document 1, when a Speak-in button displayed on a touch panel is pressed by a user, the user's voice is captured until the button is pressed again, and voice recognition of the captured voice is performed. A technique for displaying the result and the translation result of the result on the screen is described.

JP 2009-205579 A

  The technology described in Patent Document 1 is a flow of performing an operation for capturing a voice spoken by the user before the user utters, and performing an operation for instructing the start of speech recognition and translation after the utterance. Is assumed. Such a technique is effective when an operation for capturing speech in advance can be performed in accordance with the timing of the utterance by the speaker. However, if the user cannot specify the timing of the conversation partner's utterance as in a general conversation, the operation for capturing the voice in advance according to the timing of the conversation partner's speech cannot be performed. As a result, the voice of the conversation partner cannot be recognized from the beginning and translated.

  The present invention has been made in view of such circumstances, and an object thereof is to support understanding of conversations in a foreign language.

  In order to solve the above problems, the present invention provides a storage unit that stores a plurality of sets in which speech content data indicating content spoken by a conversation partner and time information indicating time are associated with each other, and the storage unit An input unit that receives from the user an instruction to process the utterance content data stored in the user, time information indicating a time after a time that is a first time later than the time at which the instruction is received by the input unit, and the storage A foreign language comprising: an extraction unit for extracting utterance content data associated with the unit; and an output unit for outputting information related to a result of speech recognition processing or translation processing for the utterance content data extracted by the extraction unit A conversation understanding support device is provided.

  In a preferred aspect, the extraction unit includes utterance content data correlated in the storage unit with time information indicating a time after a time that is a predetermined time before the time when the instruction is received by the input unit. In addition, utterance content data input by the user or meeting predetermined conditions may be extracted.

  In a further preferred aspect, the input unit further accepts input of characters by the user, and the utterance content data matching the condition is an utterance representing a word including the characters whose input has been accepted by the input unit. It may be content data.

  In another more preferable aspect, the input unit further accepts input of information for specifying a phonetic symbol by the user, and utterance content data matching the condition is accepted by the input unit. It may be utterance content data representing a word including a phonetic symbol specified by the given information in the phonetic symbol.

  In another more preferable aspect, the utterance content data matching the condition may be utterance content data representing a word whose part of speech is a predetermined part of speech.

  Further, the present invention is executed by a foreign language conversation understanding support device including a storage unit that stores a plurality of sets in which speech content data indicating content spoken by a conversation partner is associated with time information indicating time. A foreign language conversation understanding support method comprising: a step of accepting a processing instruction for utterance content data stored in the storage unit from a user; and a time that is a first time later than a time when the instruction is accepted Extracting the utterance content data associated with the time information indicating the time and the storage unit, and outputting information related to the result of speech recognition processing or translation processing on the extracted utterance content data; A foreign language conversation understanding support method is provided.

  Further, the present invention is stored in the storage unit in a computer including a storage unit that stores a plurality of sets in which speech content data indicating content uttered by a conversation partner and time information indicating time are associated with each other. A step of accepting an instruction to process the received utterance content data from the user, and utterance associated in the storage unit with time information indicating a time after a time that is a first time later than the time when the instruction was accepted There is provided a program for executing a step of extracting content data and a step of outputting information related to a result of speech recognition processing or translation processing on the extracted utterance content data.

  According to the present invention, it is possible to support understanding of a conversation in a foreign language.

3 is a functional block diagram of the foreign language conversation understanding support device 1. FIG. It is a figure which shows an example of a data structure of word DB122. It is a flowchart which shows an example of a word memory | storage process. It is a flowchart which shows an example of a word display process. It is a flowchart which shows an example of an extraction process. FIG. 6 is a diagram illustrating an example of a screen displayed on the touch panel 13. It is a functional block diagram of foreign language conversation understanding support device 1A. It is a figure which shows an example of a data structure of word DB122A. It is a figure which shows an example of a data structure of phonetic symbol dictionary DB124. It is a figure which shows an example of a data structure of reading dictionary DB125. FIG. 6 is a diagram illustrating an example of a screen displayed on the touch panel 13.

1. Embodiment 1-1. Configuration FIG. 1 is a functional block diagram of a foreign language conversation understanding support device 1 according to an embodiment of the present invention. This foreign language conversation understanding support device 1 is a computer that enables conversation using voice and video with users of other devices connected via a network. In particular, the foreign language conversation understanding support device 1 is a computer that supports understanding of a conversation when the conversation is performed in a foreign language for the user of the device.

  The foreign language conversation understanding support device 1 is specifically a personal computer, for example. More specifically, it is a mobile terminal such as a smartphone or a tablet terminal, or a stationary computer. Generally speaking, the foreign language conversation understanding support device 1 is, as shown in FIG. 1, a processing unit 11, a storage unit 12, a touch panel 13, a camera 14, a microphone 15, a speaker 16, and a communication unit. 17.

  The processing unit 11 is an arithmetic processing device such as a CPU (Central Processing Unit). As shown in FIG. 1, the processing unit 11 includes an encoder 111, a decoder 112, a word recognition unit 113, a stored information update unit 114, a clock unit 115, an extraction unit 116, a translation unit 117, The display controller 118 has a function. These functions are realized by the processing unit 11 executing a program stored in the storage unit 12. These functions will be described later.

  The storage unit 12 is a storage device such as an EEPROM (Electronically Erasable and Programmable ROM) or a flash memory. The storage unit 12 stores a program executed by the processing unit 11. In addition, as illustrated in FIG. 1, the storage unit 12 stores a learning model DB (Database) 121, a word DB 122, and a translation dictionary DB 123.

  The learning model DB 121 is a database that stores a learning model necessary for speech recognition processing executed by the word recognition unit 113. Specifically, the learning model DB 121 stores, for example, an acoustic model, a language model, and a dictionary. The learning model DB 121 may store a plurality of learning models so as to correspond to a plurality of languages.

The word DB 122 is a database that stores word data generated as a result of speech recognition processing executed by the word recognition unit 113 in association with time information. FIG. 2 is a diagram showing an example of the data structure of the word DB 122. As shown in FIG. As shown in FIG. 2, each record constituting the word DB 122 is composed of fields of “time” and “word”. The word field stores word data generated as a result of the speech recognition processing executed by the word recognition unit 113, and the time field stores time information indicating the time at which the word data is stored. . This word data is an example of “utterance content data” according to the present invention.
In the present embodiment, articles, prepositions, and personal pronouns are not registered in the word DB 122, but may be registered.

  The translation dictionary DB 123 is a database that stores translation dictionary data necessary for translation processing executed by the translation unit 117. The translation dictionary DB 123 may store a plurality of translation dictionary data so as to correspond to a plurality of languages.

  The touch panel 13 includes an input unit 131 that is an input device such as a touch sensor, and a display unit 132 that is a display device such as a liquid crystal display. When receiving an instruction for translation from the user, the input unit 131 notifies the extraction unit 116 of the fact. The display unit 132 displays an image represented by the image data acquired from the decoder 112. The display unit 132 is an example of the “output unit” according to the present invention.

  The camera 14 is a digital camera including an imaging element such as a CCD (Charge Coupled Device) and a lens, for example. The camera 14 outputs the generated image data to the processing unit 11. Note that the camera 14 may shoot a moving image as well as a still image.

The microphone 15 collects the voice of the user of the foreign language conversation understanding support device 1 and outputs the voice data to the processing unit 11.
The speaker 16 outputs sound represented by the sound data input from the processing unit 11 (specifically, the decoder 112).

The communication unit 17 is a network card, for example. The communication unit 17 performs data communication with an external device via a network. Specifically, when the communication unit 17 receives a packet group including encoded audio data or image data via a network, the communication unit 17 reconstructs the packet group and encodes the encoded audio data or image data. Is transferred to the processing unit 11. In addition, when the communication unit 17 acquires the encoded audio data or image data from the processing unit 11, the communication unit 17 packetizes the data and transmits the packetized data to an external device via the network.
Next, each function of the processing unit 11 will be described.

When the encoder 111 acquires image data from the camera 14 or acquires audio data from the microphone 15, the encoder 111 encodes the data and passes it to the communication unit 17.
The decoder 112 decodes (in other words, decodes) the encoded audio data acquired from the communication unit 17 and passes it to the word recognition unit 113 and the speaker 16. In addition, the decoder 112 decodes the encoded image data acquired from the communication unit 17 and passes it to the display control unit 118.

The word recognition unit 113 performs speech recognition processing on the speech data acquired from the decoder 112 with reference to the learning model DB 121 stored in the storage unit 12, and updates the stored word data as a result of the processing. Passed to the unit 114. A known method may be used for the speech recognition processing method.
When the storage information update unit 114 acquires word data from the word recognition unit 113, the storage information update unit 114 acquires time information from the clock unit 115 and associates the word data with the time information and stores them in the word DB 122.

The clock unit 115 receives a request from the storage information update unit 114 or the extraction unit 116, and returns time information indicating the current time as a response.
When the input unit 131 of the touch panel 13 receives a translation instruction from the user and receives a notification to that effect, the extraction unit 116 executes an extraction process of extracting word data that matches a certain condition from the word DB 122. In this extraction process, the extraction unit 116 executes the following process.

First, the extraction unit 116 acquires time information from the clock unit 115. Next, the retroactive time creating unit 1161 of the extracting unit 116 creates retroactive time information indicating a time that is a predetermined time after the time indicated by the acquired time information. This predetermined time is an example of the “first time” according to the present invention. This predetermined time is set by a user, for example.
Next, the extraction unit 116 associates, in the word DB 122, time information indicating a time after the time indicated by the created retroactive time information and before the time indicated by the time information acquired from the clock unit 115. Get word data. Next, the extraction unit 116 passes the word list data indicated by the acquired word data to the translation unit 117.

  When the translation unit 117 acquires the data of the word list from the extraction unit 116, the translation unit 117 searches the translation dictionary DB 123 for each word in the list to obtain a translation destination word, and the translation source word (that is, the word in the list). The data of the list of word pairs in which the translation destination word and the translation destination word are associated is transferred to the display control unit 118.

The display control unit 118 displays a list of word pairs indicated by the data acquired from the translation unit 117 at a predetermined position on the screen displayed by the display unit 132 of the touch panel 13.
The above is the description of each function of the processing unit 11.

1-2. Operation Next, the operation of the foreign language conversation understanding support device 1 will be described. Specifically, a word storage process for storing the word data generated as a result of speech recognition of the content uttered by the conversation partner in the word DB 122, and extracting the word data from the word DB 122 according to an instruction from the user. A word display process for displaying the word indicated by the word data and the translation destination word on the touch panel 13 will be described.

  When the foreign language conversation understanding support device 1 is activated prior to the execution of these processes, the learning model and the translation dictionary are used as the learning model and the translation used, respectively, according to the speech recognition target language and the translation destination language. Selected as a dictionary. The learning model and the translation dictionary selected here may be designated in advance by the user using the input unit 131 of the touch panel 13. When the foreign language conversation understanding support device 1 is activated, the input unit 131 of the touch panel 13 is in a state where it can accept a translation instruction from the user.

  In the following description, an example in which the language of the translation source is English and the language of the translation destination is Japanese will be described, but the combination of languages is not limited to this example, and may be a combination of other languages.

1-2-1. Word Storage Process FIG. 3 is a flowchart illustrating an example of the word storage process. The foreign language conversation understanding support device 1 continues this word storage processing until an instruction to end the processing by the user is received by the input unit 131 of the touch panel 13 or the power of the device is turned off.

  First, in step S <b> 11, when the communication unit 17 receives encoded audio data, the communication unit 17 delivers the audio data to the decoder 112. The decoder 112 decodes the delivered voice data, and delivers the decoded voice data to the word recognition unit 113 (step S12).

  The word recognition unit 113 performs speech recognition processing on the received speech data with reference to the learning model DB 121 stored in the storage unit 12, and stores the word data generated as a result of the processing as a storage information update unit It passes to 114 (step S13). When the storage information update unit 114 receives the word data, the storage information update unit 114 acquires the time information from the clock unit 115 and associates the word data with the time information and stores them in the word DB 122 (step S14). For example, when the word data indicating the word “hello” and the time information indicating the time “10:05:30” are acquired, these data are associated with each other in the word DB 122 as shown in FIG. Remember.

1-2-2. Word Display Process FIG. 4 is a flowchart illustrating an example of the word display process.
First, when the input unit 131 of the touch panel 13 receives an instruction for translation from the user, the input unit 131 notifies the extraction unit 116 of that (step S21). Upon receiving the notification from the input unit 131, the extraction unit 116 executes an extraction process for extracting word data that matches a certain condition from the word DB 122 (step S22).

FIG. 5 is a flowchart showing an example of this extraction process.
In this extraction process, the extraction unit 116 first receives a notification from the input unit 131, issues a request for time information to the clock unit 115, and acquires the time information from the clock unit 115 (step S31). Next, the retroactive time creating unit 1161 of the extracting unit 116 creates retroactive time information indicating a time that is a predetermined time after the time indicated by the time information acquired from the clock unit 115 (step S32). For example, when the time information acquired from the clock unit 115 indicates the time “10:05:34” and the predetermined time is set to “3 seconds”, the time “10:05:31” is set. The retroactive time information shown is created.

  Next, the extraction unit 116 associates, in the word DB 122, time information indicating a time after the time indicated by the created retroactive time information and before the time indicated by the time information acquired from the clock unit 115. The acquired word data is acquired (step S33). For example, when the created retroactive time information indicates the time “10:05:31” and the time information acquired from the clock unit 115 indicates the time “10:05:34”, the word DB 122 illustrated in FIG. In the example, each word data indicating the words “live”, “Tokyo”, “work”, “computer”, and “company” is acquired.

Next, the extraction unit 116 transfers the data of the word list indicated by the acquired word data to the translation unit 117 (step S34).
The above is the description of the extraction process.

  When the translation unit 117 acquires the data of the word list from the extraction unit 116, the translation unit 117 searches the translation dictionary DB 123 for each word in the list to obtain a translation destination word, and the translation source word (that is, the word in the list). Data of the list of word pairs in which the translation destination words are associated with each other is transferred to the display control unit 118 (step S23). The display control unit 118 displays a list of word pairs indicated by the transferred data at a predetermined position on the screen displayed by the display unit 132 of the touch panel 13 (step S24).

FIG. 6 is a diagram illustrating an example of a screen displayed on the display unit 132 of the touch panel 13 as a result of step S24. As shown in FIG. 6, this screen has a video display area A1 for displaying the video of the conversation partner and a list display area A2 for displaying the word pair list.
The word pair list displayed in the list display area A2 may be hidden in response to an operation on the input unit 131 of the touch panel 13 by the user, or may be automatically hidden after a certain time.

  The position on the screen where the word pair list is displayed is not limited to the illustrated example, and may be any position that does not hinder the progress of the conversation. For example, when the application for progressing conversation being executed is a TV conference application as in the illustrated example, it may be an area where the video of the conversation partner is not displayed.

  According to the foreign language conversation understanding support device 1 according to the present embodiment described above, the content uttered by the conversation partner is continuously voice-recognized, the word is stored, and a translation instruction from the user is accepted. Then, a word associated with a time after a predetermined time after that time is specified, and the word and a translation destination word are displayed on the screen. Therefore, when there is a part that cannot be understood in the conversation, the user can understand the contents of the part by giving a translation instruction after that point.

Further, according to the foreign language conversation understanding support device 1, the user can instruct translation by an operation such as a tap operation on the screen. Such an operation is a relatively simple operation and is not an operation using voice, so that it does not hinder a conversation performed by voice. Moreover, since such an operation is also a relatively inconspicuous operation, it is difficult for the conversation partner to realize that the translated word is being examined.
Further, according to the foreign language conversation understanding support device 1, the speech-recognized word and the translated word are displayed on the screen as character information, not as speech information. Therefore, it does not disturb the conversation conducted by voice.

2. Modifications The above embodiment may be modified as follows. Further, the following modifications may be combined with each other.

2-1. Modification 1
In the extraction process according to the above-described embodiment, the extraction unit 116 may extract word data based on further information in addition to the time information, and may support the specification of necessary information by the user. . The following two methods are conceivable as specific methods.
(1) In addition to time information, word data is extracted based on other information received from the user at the time of a translation instruction.
(2) In addition to the time information, word data is extracted based on other information preset by the user.

2-1-1. When narrowing down based on other information received from the user As the method (1), specifically, the following two methods are conceivable.
(1-1) Narrowing is performed based on a search character input by a user.
(1-2) Narrowing down based on phonetic symbols (or information for specifying phonetic symbols) input by the user.

2-1-1-1. In the case of this example, the input unit 131 of the touch panel 13 accepts input of a search character in addition to a translation instruction by the user in step S21 of the word display process, and the instruction May be notified to the extraction unit 116, and character data indicating the input search character may be transferred to the extraction unit 116. Here, the search character may be one character or a plurality of characters.

When the extraction unit 116 receives the notification and acquires the character data, the extraction unit 116 obtains the time information after the time indicated by the retroactive time information and the time information acquired from the clock unit 115 in step S33 of the extraction process. Word data that is associated with time information indicating a time before the indicated time in the word DB 122 and that represents a word that starts (or includes) a search character indicated by the acquired character data is acquired. You may do it.
For example, the created retroactive time information indicates the time “10:05:31”, the time information acquired from the clock unit 115 indicates the time “10:05:34”, and the acquired character data indicates the character “li”. In the case of showing, in the example of the word DB 122 shown in FIG. 2, word data indicating the word “live” is acquired.

  When the search character is input to the input unit 131, the display unit 132 of the touch panel 13 may display a software keyboard on the screen, and the search character may be input using the software keyboard. Alternatively, the display unit 132 of the touch panel 13 may display an area for handwriting input on the screen, and a character input in the area may be recognized as a search character.

  As another example, the search characters may be input using a tangible keyboard (not shown) connected to the foreign language conversation understanding support device 1. Alternatively, voice recognition processing may be performed on the user's voice data collected by the microphone 15, and the character specified as a result may be used as a search character.

2-1-1-2. FIG. 7 is a functional block diagram of the foreign language conversation understanding support device 1A according to the present example. The foreign language conversation understanding support device 1A shown in the figure is the foreign language conversation understanding support device according to the above embodiment in that the storage unit 12 stores the word DB 122A, the phonetic symbol dictionary DB 124, and the reading dictionary DB 125. 1 and different.

  FIG. 8 is a diagram illustrating an example of a data configuration of the word DB 122A. As shown in FIG. 8, each record of the word DB 122A is different from the word DB 122 according to the above embodiment in that it further has a “phonetic symbol” field.

  FIG. 9 is a diagram illustrating an example of a data configuration of the phonetic symbol dictionary DB 124. The phonetic symbol dictionary DB 124 is a database that stores word data of a translation source language and its phonetic symbol data in association with each other. As shown in FIG. 9, each record constituting the phonetic symbol dictionary DB 124 includes fields of “word of translation source language” and “phonetic symbol”.

  FIG. 10 is a diagram illustrating an example of a data configuration of the reading dictionary DB 125. The reading dictionary DB 125 is a database that stores character (specifically, phonetic character) data of a translation destination language and phonetic symbol data of a translation source language in association with each other. As shown in FIG. 10, each record constituting the reading dictionary DB 125 includes fields of “translation target language characters” and “translation source language phonetic symbols”. A plurality of phonetic symbol data may be associated with one character data.

  In the foreign language conversation understanding support device 1A, when the storage information update unit 114 receives word data from the word recognition unit 113 in step S14 of the word storage process, the storage information update unit 114 associates the word data with the pronunciation symbol dictionary DB 124. The phonetic symbol data may be acquired. Then, time information may be acquired from the clock unit 115, and the word data, phonetic symbol data, and time information may be associated with each other and stored in the word DB 122A.

  In step S21 of the word display process, the input unit 131 of the touch panel 13 accepts input of characters (specifically, phonetic characters) for specifying phonetic symbols in addition to instructions for translation by the user. The extraction unit 116 may be notified that the instruction has been received, and character data indicating the input character may be transferred to the extraction unit 116.

When the extraction unit 116 receives the notification and acquires character data, the extraction unit 116 first acquires phonetic symbol data associated with the character data and the reading dictionary DB 125 in step S33 of the extraction process. Also good.
For example, when the acquired character data indicates the character “RI”, each phonetic symbol data indicating phonetic symbols “l” and “r” is acquired in the example of the reading dictionary DB 125 shown in FIG.

The extraction unit 116 associates in the word DB 122 with time information indicating a time after the time indicated by the retroactive time information and before the time indicated by the time information acquired from the clock unit 115. You may make it acquire the word data matched with the phonetic symbol data which is word data and shows the phonetic symbol group which begins with the phonetic symbol shown by the acquired phonetic symbol data in the word DB122.
For example, the created retroactive time information indicates the time “10:05:31”, the time information acquired from the clock unit 115 indicates the time “10:05:34”, and the acquired phonetic symbol data is the phonetic symbol “ When “l” and “r” are indicated, word data indicating the word “live” is acquired in the example of the word DB 122A shown in FIG.

  Note that the word data acquired by the extraction unit 116 is limited to word data associated with phonetic symbol data indicating a phonetic symbol group starting with a phonetic symbol indicated by the acquired phonetic symbol data as in the above example. Instead, it may be word data associated with phonetic symbol data indicating a phonetic symbol group including a phonetic symbol indicated by the acquired phonetic symbol data.

  According to this example, even if the user does not understand the spelling of the word whose meaning could not be understood, the user can display the translation of the word based on the sound that he has heard.

In this example, a character for specifying a phonetic symbol may be input using a software keyboard or a handwriting input area displayed on the display unit 132 of the touch panel 13. Alternatively, the input may be performed using a tangible keyboard (not shown) connected to the foreign language conversation understanding support device 1A.
FIG. 11 is a diagram illustrating an example of a screen displayed on the display unit 132 of the touch panel 13 when a character for specifying a phonetic symbol is input using a handwriting input area. As shown in FIG. 11, the screen further has a handwriting input area A3 as compared to the screen shown in FIG.

  As another example, a character for specifying a phonetic symbol may be input using the microphone 15. In this case, the microphone 15 may deliver the collected voice data of the user to a phonetic symbol conversion unit (not shown). The phonetic symbol conversion unit performs voice recognition processing on the acquired voice data with reference to a learning model (not shown) for outputting phonetic symbol data based on the voice data, and is generated as a result. The phonetic symbol data may be transferred to the extraction unit 116. The extraction unit 116 may narrow down word data using the acquired phonetic symbol data.

2-1-2. When narrowing down based on other information preset by the user As the above method (2), specifically, the following two methods are conceivable.
(2-1) Narrow down based on the type of part of speech of the word.
(2-2) Narrowing is performed based on the level of the word (specifically, difficulty level).

2-2-1-1. When narrowing down based on the type of part of speech of a word In the case of this example, when the word recognition unit 113 acquires voice data from the decoder 112 in step S13 of the word storage process, the word recognition unit 113 stores the voice data. A speech recognition process is performed with reference to the learning model DB 121 stored in the unit 12, and word data generated as a result of the process and part of speech data indicating the part of speech of the word represented by the word data are stored information update unit It may be transferred to 114.

  The storage information updating unit 114 that acquires part-of-speech data in addition to word data first evaluates the correspondence between the type of part-of-speech and importance stored in the storage unit 12 in step S14 of the word storage process. You may make it identify the importance of the part of speech shown with the said part of speech data with reference to information (not shown). Then, the stored information update unit 114 stores the acquired word data in the word DB 122 in association with the time information acquired from the clock unit 115 only when the identified importance is equal to or greater than a predetermined threshold. It may be. Here, the importance level and the threshold value of the part of speech may be set in advance by the user.

  According to this example, since only the part-of-speech word data having an importance level equal to or higher than a predetermined threshold is stored in the word DB 122, only such word data is extracted by the extraction unit 116 in the extraction process. Become.

  In this example, the stored information updating unit 114 refers to information on a list of parts of speech to be stored (not shown) instead of the above-described evaluation criterion information, and the word of speech included in the list. Only data may be stored in the word DB 122 in association with time information.

2-1-2-2. In the case of this example, the storage information update unit 114 first stores word data in the storage unit 12 after acquiring word data from the word recognition unit 113 in step S14 of the above word storage process. The level of the word represented by the word data may be specified with reference to a level dictionary (not shown) that associates the word data with the level information. The stored information update unit 114 stores the acquired word data in the word DB 122 in association with the time information acquired from the clock unit 115 only when the specified level is equal to or higher than the language level of the user. May be. Here, the language level of the user may be input in advance using the input unit 131 of the touch panel 13.

  According to this example, since only word data of a level higher than the user's language level is stored in the word DB 122, only such word data is extracted by the extraction unit 116 in the extraction process.

2-2. Modification 2
In the word storage processing according to the above embodiment, the encoded voice data received by the communication unit 17 is decoded by the decoder 112 and transferred to the word recognition unit 113 (see steps S11 and S12). When the user's conversation partner is present in front of the foreign language conversation understanding support device 1, the voice data of the conversation partner collected by the microphone 15 is transferred to the word recognition unit 113 as a target for speech recognition processing. May be.

2-3. Modification 3
In the word storage processing according to the above-described embodiment, the software for speech recognition processing may be properly used according to the language that is the target of speech recognition. Further, in the word display processing according to the above-described embodiment, the software for translation processing may be properly used according to the language of the translation source and the language of the translation destination. The voice recognition software and translation processing software to be used may be set in advance by the user.

2-4. Modification 4
In the word storage processing according to the above embodiment, the word data and the time information are associated with each other and stored in the word DB 122 (see step S14). However, instead of the word data, the sentence data including a plurality of words and The time information may be stored in the word DB 122 in association with the time information. Accordingly, in the word display process according to the above-described embodiment, the extraction unit 116 extracts sentence data instead of word data, and sends the sentence list data represented by the extracted sentence data to the translation unit 117. You may make it deliver (refer step S22). The translation unit 117 searches the translation dictionary DB 123 for each sentence in the acquired list data to obtain a translation destination sentence, and associates the translation source sentence (that is, the sentence in the list) with the translation destination sentence. The attached sentence pair list data may be transferred to the display control unit 118 (see step S23).

2-5. Modification 5
In the above embodiment, pairs of word data and time information stored in the word DB 122 may be sequentially deleted from the oldest. For example, in step S14 of the word storage process described above, the storage information update unit 114 may store a set of word data and time information in a ring buffer secured in the storage unit 12. In this case, the storage information update unit 114 manages the address information of the buffer that was last written, and the next time writing is performed on the buffer at the next address of the address. Also good.

As another example, when the stored information update unit 114 newly stores a set of word data and time information in the word DB 122, the stored information update unit 114 starts from a time that is a predetermined time later than the time indicated by the time information. Alternatively, the word data associated with the time information indicating the previous time may be deleted from the word DB 122 together with the time information.
Alternatively, the stored information updating unit 114 acquires time information from the clock unit 115 at a constant cycle, and time information indicating a time before a time that is a predetermined time before the time indicated by the time information. The associated word data may be deleted from the word DB 122 together with the time information.

2-6. Modification 6
In the word storage processing according to the above embodiment, the speech data is decoded by the decoder 112, and the word data generated by performing the speech recognition processing on the speech data by the word recognition unit 113 is the time information. Correspondingly stored in the word DB 122 (see steps S12 to S14), instead of the word data, the audio data decoded by the decoder 112 and the time information may be stored in the word DB 122. . Here, the audio data associated with the time information may be audio data cut out for each silence period of the predetermined period in the decoded audio data (see, for example, JP-A-2001-154691). The audio data according to this modification is an example of “utterance content data” according to the present invention.

  In this case, in the word display process according to the above-described embodiment, the extraction unit 116 extracts the voice data from the word DB 122 instead of the word data (see step S22), and then passes this voice data to the word recognition unit 113. You may do it. The word recognition unit 113 performs a speech recognition process on the acquired speech data with reference to the learning model DB 121 stored in the storage unit 12, and creates a list of words indicated by the word data generated as a result of the process. Data may be transferred to the translation unit 117. The translation unit 117 searches the translation dictionary DB 123 for each word in the acquired word list to obtain a translation destination word, and displays word pair list data that associates the translation source word with the translation destination word. You may make it deliver to the control part 118 (step S23).

According to this modification, since it is only necessary to perform the speech recognition process on the speech data indicating the word displayed on the screen, the processing load on the processing unit 11 can be reduced.
In the above example, the decoded voice data and time information are stored in the word DB 122. However, the encoded voice data and time information are stored in the word DB 122 to extract the voice data. It may be decoded later (see step S22), and then the speech recognition process may be performed. According to this example, since unnecessary decoding processing is omitted, the processing load on the processing unit 11 can be further reduced in that respect.

2-7. Modification 7
In the above embodiment, the speech recognition process is performed in the foreign language conversation understanding support device 1, but it may be performed in another device (for example, a speech recognition server) connected to this device via a network. Good. In this case, in the word storage process, the communication unit 17 transfers the received encoded voice data to a voice recognition server (not shown), and the word data generated as a result of the voice recognition process in the voice recognition server. And the word data may be transferred to the stored information updating unit 114. Here, since the communication unit 17 only needs to transfer the already encoded audio data to the speech recognition server as it is, there is no need to perform a new encoding process, and thus a new processing load is generated on the processing unit 11. There is nothing.

2-8. Modification 8
In the word display processing according to the above embodiment, the word data extracted by the extraction unit 116 is translated by the translation unit 117, and as a result, not only the translation source word but also the translation destination word are displayed on the screen. (See steps S23 and S24). However, only the translation source word may be displayed on the screen. That is, step S23 may be omitted in the word display process according to the above embodiment. This is because, depending on the user, as long as the words of the translation source language appearing in the conversation can be identified, the contents of the conversation may be understood even if the words of the translation destination language are not shown.

2-9. Modification 9
In the extraction process according to the above-described embodiment, retroactive time information indicating a time that is a predetermined time backward from the time indicated by the time information acquired from the clock unit 115 is created (step S32). The given time may be a variable time that changes according to conditions. For example, the variable time may change according to the pressing force or pressing time of the user's finger when a translation instruction is input by the user. Specifically, it may be controlled so that the variable time becomes longer as the pressing force becomes stronger or the pressing time becomes longer. This variable time is an example of the “first time” according to the present invention.

2-10. Modification 10
In the word display process according to the above embodiment, the display control unit 118 displays information related to the translation source or translation destination words included in the list on the screen in addition to or instead of the word pair list. May be. Here, the information related to the translation source or translation destination word is, for example, information hit as a result of searching on the Internet using the word as a search key.

2-11. Modification 11
The program for realizing the function group of the foreign language conversation understanding support device 1 or 1A according to the above-described embodiment and modification group may be provided via a recording medium readable by the computer device. Here, the recording medium is, for example, a magnetic recording medium such as a magnetic tape or a magnetic disk, an optical recording medium such as an optical disk, a magneto-optical recording medium, or a semiconductor memory. In addition, this program may be provided via a network such as the Internet.

DESCRIPTION OF SYMBOLS 1 ... Foreign language conversation understanding support apparatus, 11 ... Processing part, 12 ... Memory | storage part, 13 ... Touch panel, 14 ... Camera, 15 ... Microphone, 16 ... Speaker, 17 ... Communication part, 111 ... Encoder, 112 ... Decoder, 113 ... Word recognizing unit 114... Stored information updating unit 115 115 clock unit 116 extracting unit 117 translating unit 118 display control unit 121 learning model DB 122 word DB 123 translation dictionary DB 124 ... phonetic symbol dictionary DB, 125 ... reading dictionary DB, 131 ... input unit, 132 ... display unit, 1161 ... retroactive time creation unit

Claims (4)

Of speech contents data indicating the content uttered by the other party of the conversation, only speech contents data representing the words of the inputted or predetermined difficulty by the user, in association time information and corresponds indicating a time more A storage unit for storing;
An input unit processing instructions for speech contents data stored in the storage unit receives from the user,
An extraction unit that extracts time information indicating a time after a time that is a first time later than a time when the instruction is received by the input unit and utterance content data that is associated with the storage unit;
An output unit that outputs information related to a result of speech recognition processing or translation processing for the utterance content data extracted by the extraction unit;
The input unit is a touch sensor;
The foreign language conversation understanding support device, wherein the first time changes according to a force or time when the input unit is pressed by the user. A word recognition unit for generating word data by performing voice recognition processing on voice data indicating contents uttered by the conversation partner;
A storage information updating unit that stores only word data representing the difficulty level word in the generated word data in the storage unit in association with time information indicating time as the utterance content data; ,
The foreign language conversation according to claim 1, wherein the output unit outputs information related to the utterance content data extracted by the extraction unit or information related to a result of translation processing on the utterance content data. Understanding support device. Of speech contents data indicating the content uttered by the other party of the conversation, only speech contents data representing the words of the inputted or predetermined difficulty by the user, in association time information and corresponds indicating a time more A foreign language conversation understanding support method executed by a foreign language conversation understanding support device including a storage unit for storing,
A step of processing instructions for the speech contents data stored in the storage unit receives from the user,
Extracting utterance content data associated with time information indicating a time after a time that is a first time later than the time when the instruction is received and the storage unit;
Outputting information related to a result of speech recognition processing or translation processing on the extracted utterance content data, and
The foreign language conversation understanding support device includes a touch sensor,
The foreign language conversation understanding support method, wherein the first time changes according to a force or time when the touch sensor is pressed by the user. Of speech contents data indicating the content uttered by the other party of the conversation, only speech contents data representing the words of the inputted or predetermined difficulty by the user, in association time information and corresponds indicating a time more In a computer having a storage unit for storing,
A step of processing instructions for the speech contents data stored in the storage unit receives from the user,
Extracting utterance content data associated with time information indicating a time after a time that is a first time later than the time when the instruction is received and the storage unit;
Outputting information related to a result of speech recognition processing or translation processing on the extracted utterance content data, and a program for executing
The computer includes a touch sensor,
The program according to claim 1, wherein the first time changes according to a force or time when the touch sensor is pressed by the user.

Images