JP7323210B2 - VOICE RECOGNITION DISPLAY DEVICE, VOICE RECOGNITION DISPLAY METHOD AND PROGRAM - Google Patents

Description

The present invention relates to a speech recognition display device, a speech recognition display method, and a program.

Japanese Patent Application Laid-Open No. 2002-200001 discloses a speech recognition system that divides a voice file into text at predetermined time intervals in order to shorten the text processing time. Further, in Patent Document 1, in audio data for generating an audio file, when the silence time between utterances is equal to or greater than a predetermined threshold, the audio data is segmented, and the speaker It is described to determine

Patent Literature 2 discloses an information processing system that divides voice data, performs voice recognition, and displays the resultant text data as text on a screen. Further, in Patent Document 2, the output timing of divided voice data corresponding to the text data is delayed from the output timing of the text data by a period until the voice recognition processing of the voice data is determined. As a result, the text data can be manually corrected while listening to the voice data corresponding to the contents of the output text data.

JP 2020-60735 A JP 2019-185005 A

In recent years, voice recognition has begun to be introduced in emergency call systems such as fire brigade 119 calls and police 110 calls. In an emergency call system that requires such judgments in seconds, other dispatchers listen to (monitor) the call voice of the caller and the receptionist, and instruct emergency vehicles such as fire trucks and ambulances to be dispatched. In that case, by converting the content of the call between the caller and the receptionist into text in real time, the dispatcher can instantly grasp the content of the report and issue the above-mentioned dispatch instruction.

A real-time speech recognition system is required to shorten the time required for text conversion so as not to impair real-time performance. Methods for dividing speech data to be converted into text in order to shorten the processing time for text conversion include a method of dividing at a predetermined time and a method of dividing at silent intervals in the voice data.

In the method of dividing the audio data at predetermined times, the audio data is divided at predetermined fixed times. In this case, the audio data is divided in the middle of the sound portion (sentence or word), and the sentence or word is not converted into text correctly. It may not make sense as a single sentence.

In addition, when detecting a silent interval in the input voice data and dividing the voice data into significant units (sentence units), a threshold is set at the voice level at which to judge silence, and the voice level of the input voice data is Silence/speech is determined depending on whether or not the threshold value is exceeded, and voice division is realized by silent intervals.

However, it is difficult to determine the optimal sound level for judging silence because the sound level for judging silence differs depending on the surrounding environmental sounds (noise, etc.) from which the sound is collected, and environmental sounds fluctuate in real time. is. For example, if the audio level for judging silence is set lower than the optimum value, if noise exceeding this level is mixed into the audio, it will be judged as active even though it should actually be silent, and the audio data will not be separated at the correct position. Gone.

Also, in such a speech recognition system that divides speech by silent intervals, the speech data before the silent interval is converted into text only after the silent interval is detected in the speech data. For this reason, if the text is long, it is not converted into text until the end of the text, and real-time performance in seconds required by the emergency call system cannot be ensured.

An object of the present disclosure is to provide a speech recognition display device, a speech recognition method, and a program capable of improving speech recognition accuracy and ensuring real-time performance in view of the above-described problems.

A voice recognition display device according to an aspect of the present invention includes a voice acquisition unit that acquires voice data, and a voice division unit that divides the voice data along a time axis at predetermined division times to generate divided voice data. a voice buffer for storing the divided voice data in order from the top, a voice recognition unit for converting the divided voice data into divided voice text data, and whether the divided voice text data is empty or not, each time the division time elapses. a text determination unit for determining whether or not the divided voice text data is not empty, a buffer control unit for generating combined voice data by combining the divided voice data stored in the voice buffer in order from the beginning; and a text display unit for displaying combined voice text data obtained by performing voice recognition of the combined voice data each time a delimitation time elapses.

A speech recognition method according to an aspect of the present invention acquires speech data, divides the speech data along the time axis at predetermined division times, generates divided speech data, and generates divided speech data each time the division time passes. and storing the divided voice data in a voice buffer in order from the beginning, converting the divided voice data into divided voice text data, determining whether the divided voice text data is empty, and determining whether the divided voice text data is empty. Otherwise, combined voice data is generated by combining the divided voice data stored in the voice buffer in order from the beginning, and combined voice text data obtained by voice recognition of the combined voice data is generated each time the division time elapses. indicate.

A program according to an aspect of the present invention includes a process of acquiring audio data, a process of dividing the audio data along a time axis at a predetermined delimitation time to generate divided audio data, and a process of generating divided audio data. each time, a process of storing the divided voice data in the voice buffer in order from the beginning, a process of converting the divided voice data into divided voice text data, and a process of determining whether the divided voice text data is empty. a process of generating combined voice data by combining the divided voice data stored in the voice buffer in order from the head when the divided voice text data is not empty; and a process of displaying the combined speech text data obtained by speech recognition of the data.

An object of the present invention is to provide a speech recognition display device, a speech recognition method, and a program capable of improving speech recognition accuracy and ensuring real-time performance.

1 is a block diagram showing a schematic configuration of a speech recognition display device according to an embodiment; FIG. 1 is a block diagram showing the configuration of a speech recognition display device according to Embodiment 1; FIG. It is an example of the call content and call voice waveform in the IP phone. FIG. 2 is a flowchart for explaining a speech recognition method according to Embodiment 1; FIG. FIG. 10 is a diagram showing divided audio data stored in an audio buffer after time Td1 has elapsed; FIG. 10 is a diagram showing divided audio data stored in an audio buffer after time Td2 has elapsed; FIG. 10 is a diagram showing divided audio data stored in an audio buffer after time Td6 has elapsed; FIG. 10 is a diagram showing divided audio data stored in an audio buffer after time Td7 has elapsed; FIG. 10 is a diagram showing text displayed on the text display section after time Td1 has elapsed; FIG. 10 is a diagram showing text displayed on the text display portion after time Td2 has elapsed; FIG. 10 is a diagram showing text displayed on the text display portion after time Td6 has elapsed; FIG. 10 is a diagram showing text displayed on the text display section after time Td7 has elapsed; FIG. 10 is a diagram showing combined audio data in which divided audio data with indexes 0 and 1 are combined; 2 is a block diagram showing the configuration of a speech recognition display device according to Embodiment 2; FIG. FIG. 10 is a flow diagram for explaining a speech recognition method according to Embodiment 2; FIG. 10 is a diagram showing combined speech text data saved in a text buffer after time Td1 has elapsed; FIG. 10 is a diagram showing combined speech text data saved in a text buffer after time Td2 has elapsed; FIG. 10 is a diagram showing combined speech text data saved in a text buffer after time Td6 has elapsed; FIG. 10 is a diagram showing combined speech text data saved in a text buffer after time Td7 has elapsed; FIG. 10 is a diagram showing text displayed on the text display section after time Td1 has elapsed; FIG. 10 is a diagram showing text displayed on the text display portion after time Td2 has elapsed; FIG. 10 is a diagram showing text displayed on the text display portion after time Td6 has elapsed; FIG. 10 is a diagram showing text displayed on the text display section after time Td7 has elapsed; FIG. 10 is a diagram showing an example of call content and a call voice waveform that are voice-recognized by the voice recognition display device according to the second embodiment; FIG. 11 is a block diagram showing the configuration of a speech recognition display device according to Embodiment 3; FIG. 12 is a block diagram showing the configuration of a speech recognition display device according to Embodiment 4; FIG. 10 is a diagram showing divided audio data stored in the audio buffer X after the lapse of three division times; FIG. 10 is a diagram showing divided audio data stored in the audio buffer Y after the lapse of three division times; FIG. 10 is a diagram showing divided speech text data stored in the text buffer X after the first delimitation time has elapsed; FIG. 10 is a diagram showing divided speech text data stored in the text buffer Y after the lapse of the first delimitation time; FIG. 10 is a diagram showing divided speech text data stored in the text buffer X after the second delimitation time has elapsed; FIG. 10 is a diagram showing divided speech text data stored in the text buffer Y after the lapse of the second delimitation time; FIG. 10 is a diagram showing divided speech text data stored in the text buffer X after the third delimitation time has elapsed; FIG. 10 is a diagram showing divided speech text data stored in the text buffer Y after the lapse of the third delimitation time; FIG. 10 is a diagram showing a saved state of a merged text buffer before merging; FIG. 10 is a diagram showing merged text data stored in a merged text buffer for the first time of merging; FIG. 10 is a diagram showing merged text data saved in a merged text buffer for the second time of merging; FIG. 12 is a diagram showing merged text data saved in a merged text buffer after the third merging; FIG. 12 is a diagram showing merged text data saved in a merged text buffer after the fourth merging; FIG. 12 is a diagram showing merged text data saved in a merged text buffer after the fifth merge. It is a figure which shows the display state of the text display part before display. FIG. 10 is a diagram showing text displayed in the text display portion for the first time of display; FIG. 10 is a diagram showing text displayed in the text display portion for the second time of display; It is a figure which shows the text displayed on the text display part of the display of the 3rd time. It is a figure which shows the text displayed on the text display part of the display of the 4th time. FIG. 13 is a diagram showing text displayed in the text display portion for the fifth time of display; FIG. 11 is a block diagram showing the configuration of a speech recognition display device according to Embodiment 5; 22A and 22B are diagrams showing an example of utterance content and an utterance voice waveform collected by the microphone of FIG. 21; FIG. FIG. 10 is a diagram showing divided audio data stored in an audio buffer after time Td3 has elapsed; FIG. 10 is a diagram showing divided audio data stored in an audio buffer after time Td7 has elapsed; FIG. 10 is a diagram showing combined speech text data saved in a text buffer after time Td1 has elapsed; FIG. 10 is a diagram showing combined speech text data saved in a text buffer after time Td2 has elapsed; FIG. 10 is a diagram showing combined speech text data saved in a text buffer after time Td3 has elapsed; FIG. 10 is a diagram showing combined speech text data saved in a text buffer after time Td4 has elapsed; FIG. 10 is a diagram showing combined speech text data saved in a text buffer after time Td5 has elapsed; FIG. 10 is a diagram showing combined speech text data saved in a text buffer after time Td6 has elapsed; FIG. 10 is a diagram showing combined speech text data saved in a text buffer after time Td7 has elapsed; FIG. 10 is a diagram showing an initial display state of a text display section; FIG. 10 is a diagram showing text displayed on the text display section after time Td1 has elapsed; FIG. 10 is a diagram showing text displayed on the text display portion after time Td2 has elapsed; FIG. 10 is a diagram showing text displayed on the text display section after time Td3 has elapsed; FIG. 10 is a diagram showing text displayed on the text display section after time Td4 has elapsed; FIG. 10 is a diagram showing text displayed on the text display section after time Td5 has elapsed; FIG. 10 is a diagram showing text displayed on the text display portion after time Td6 has elapsed; FIG. 10 is a diagram showing text displayed on the text display section after time Td7 has elapsed;

Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. However, the present invention is not limited to the following embodiments. Also, for clarity of explanation, the following description and drawings are simplified as appropriate. In each drawing, the same components are denoted by the same reference numerals, and the description thereof is omitted as appropriate.

FIG. 1 is a block diagram showing a schematic configuration of a speech recognition display device 1 according to an embodiment. The speech recognition display device 1 is a device having a function of acquiring speech data and a function of displaying text data generated from the speech data. As shown in FIG. 1, the speech recognition display device 1 includes a speech acquisition unit 10, a time-segmented speech division unit 11, a speech buffer 12, a speech recognition unit 13, a text display unit 14, a text determination unit 21, and a buffer control unit 31. Prepare.

The voice acquisition unit 10 acquires voice data and transmits it to the time-segmented voice division unit 11 . The time-segmented audio dividing unit 11 divides the received audio data along the time axis at predetermined time intervals to generate divided audio data. The divided voice data are sequentially stored in the voice buffer 12 from the top and input to the voice recognition section 13 every time the division time elapses. The voice recognition unit 13 converts the divided voice data into divided voice text data and transmits the data to the text determination unit 21 . The text determination unit 21 determines whether or not the divided voice text data is empty.

A determination result by the text determination unit 21 is input to the buffer control unit 31 . If the divided voice text data is not empty, the buffer control unit 31 generates combined voice data by combining the divided voice data stored in the voice buffer 12 in order from the beginning. The text display unit 14 displays combined voice text data obtained by performing voice recognition of the combined voice data each time the delimitation time elapses. This makes it possible to improve speech recognition accuracy and ensure real-time performance.

Embodiment 1.
FIG. 2 is a block diagram showing the configuration of the speech recognition display device 1A according to the first embodiment. In the example shown in FIG. 2, the voice recognition display device 1A recognizes the call voice of the IP telephone 40 in real time and displays it as text. FIG. 2 shows a split speech/text control section 20 including the text determination section 21 of FIG. 1 and a combined speech/text control section 30 including a buffer control section 31 .

The speech recognition display device 1A includes a time-segmented speech division unit 11, a speech buffer 12, a speech recognition unit 13, a text display unit 14, a recognition DB 15, a divided speech/text control unit 20, and a combined speech/text control unit 30. The divided speech/text control unit 20 includes a text determination unit 21 , a divided speech transmission unit 22 and a divided speech text reception unit 23 . The combined voice/text control unit 30 includes a buffer control unit 31 , a combined voice transmission unit 32 , and a combined voice/text reception unit 33 .

The voice acquisition unit 10 acquires a digital voice signal output from the IP telephone 40 and outputs it as voice data to the time-delimited voice division unit 11 . The time-segmented audio division unit 11 divides the audio data received from the audio acquisition unit 10 by preset division times to generate a plurality of divided audio data. The time-segmented audio dividing unit 11 sequentially stores the divided audio data in the audio buffer 12 from the top every time the interval time elapses. Also, the time-segmented audio dividing unit 11 transmits the divided audio data to the divided audio transmission unit 22 of the divided audio/text control unit 20 .

FIG. 3 shows an example of call contents and call voice waveforms on the IP telephone 40 . Here, it is assumed that the voice start time is 0 and the predetermined interval time is Td. The audio data is divided into a plurality of divided audio data every division time Td. As shown in FIG. 3, the time when the first interval time Td elapses is Td1, and thereafter, Td2→Td3→Td4→Td5→Td6→Td7 each time the interval time Td elapses. Divided audio data at times 0 to Td1, Td1 to Td2, Td2 to Td3, Td3 to Td4, Td4 to Td5, Td5 to Td6, Td6 to Td7 are divided into divided audio data d1, d2, d3, d4, d5, d6, d7.

The time-delimited audio dividing unit 11 saves the divided audio data at the end of the queue (audio queue) of the audio buffer 12 every time the interval time Td elapses. The divided audio transmission unit 22 transmits the divided audio data received from the time-segmented audio dividing unit 11 to the audio recognition unit 13 . The speech recognition unit 13 refers to the recognition DB 15 , converts the divided speech data into divided speech text data, and transmits the divided speech text data to the divided speech text reception unit 23 .

The recognition DB 15 stores acoustic models, language models, dictionaries, etc., which are used when performing speech recognition processing. The speech recognition unit 13 discriminates phonemes by using, for example, an acoustic model such as a Hidden Markov Model for time-series patterns of feature amounts obtained by acoustically analyzing speech data. In addition, the speech recognition unit 13 generates text data by selecting the most appropriate word from the accumulated words for the determined phoneme using a dictionary and a language model such as an N-gram. do.

The divided speech text receiving unit 23 transmits the divided speech text data received from the speech recognition unit 13 to the text determination unit 21 . The text determination unit 21 determines whether the divided voice text data is “empty” and inputs the determination result to the buffer control unit 31 .

If the determination result is “not empty”, the buffer control unit 31 generates combined audio data by combining the divided audio data stored in the audio buffer 12 in order from the beginning, and transmits the combined audio data to the combined audio transmission unit 32 . On the other hand, if the determination result is "empty", the buffer control unit 31 deletes the divided audio data stored in the audio buffer 12 to make the audio buffer 12 empty.

The combined voice transmission unit 32 transmits the combined voice data received from the buffer control unit 31 to the voice recognition unit 13 . The speech recognition unit 13 refers to the recognition DB 15 , converts the combined voice data into combined voice text data, and transmits the combined voice text data to the combined voice text reception unit 33 . Here, one speech recognition unit 13 is configured to perform speech recognition processing for both the divided speech data and the combined speech data, but each speech data may be processed by a different speech recognition unit. .

The combined voice text receiving unit 33 transmits the received combined voice text data to the text display unit 14 . The text display unit 14 displays the received combined speech text data. For example, the text display unit 14 can sequentially display the combined speech text data updated at each delimitation time Td line by line.

Here, the speech recognition method according to Embodiment 1 will be described with reference to FIGS. 4, 5A to 5D, 6A to 6D, and FIG. FIG. 4 is a flowchart for explaining the speech recognition method according to the first embodiment. 5A to 5D are diagrams showing divided audio data stored in the audio buffer after each time. 6A to 6D are diagrams showing the text displayed on the text display section after each time has elapsed. The text data generated by speech recognition of the divided voice data d1, d2, d3, d4, d5, d6 and d7 are assumed to be divided voice text data TX1, TX2, TX3, TX4, TX5, TX6 and TX7, respectively. .

First, the voice acquisition unit 10 acquires voice data from the digital voice signal of the IP telephone 40 (step S10). The audio acquisition unit 10 sequentially transmits the audio data from time 0 to the time-segmented audio division unit 11 . Then, the time-segmented audio dividing unit 11 divides the audio data along the time axis at a predetermined interval time Td to generate divided audio data, and each time the interval time elapses, the divided audio data is transferred to the audio buffer. 12 and transmitted to the speech recognition unit 13 (step S11).

Specifically, the time-separated audio dividing unit 11 stores the divided audio data d1 from time 0 to Td1 in the audio buffer 12 when the first delimiting time Td has passed (time Td1). At this time, since the audio buffer 12 is empty, the divided audio data d1 is stored in the audio queue index 0 of the audio buffer 12 as shown in FIG. 5A.

Then, the speech recognition unit 13 refers to the recognition DB 15 and converts the divided speech data d1 into divided speech text data TX1 (step S12). The content of the divided voice text data TX1 is "a passerby collapses". The divided speech text data TX1 is input to the text judgment section 21 via the division speech text receiving section 23. FIG. Then, the text determination unit 21 determines whether or not the divided voice text data TX1 is empty (step S13).

A determination result by the text determination unit 21 is input to the buffer control unit 31 . As described above, the segmented speech text data TX1 is not empty. If the divided speech text data is not empty (step S13 NO), the buffer control unit 31 combines the divided speech data stored in the speech buffer 12 in order from the beginning to generate combined speech data, and sends the data to the speech recognition unit 13. Send (step S14). At this time, since only index 0 is stored in the audio queue, the divided audio data with index 0 is input to the audio recognition section 13 via the combined audio transmission section 32 .

Then, the speech recognition unit 13 refers to the recognition DB 15 and converts the combined voice data into combined voice text data. This combined speech text data is input to the text display section 14 via the combined speech text receiving section 33 . The text display unit 14 displays the received combined speech text data (step S15). The content of the combined speech text data at this time is "a passerby collapses". The text display unit 14 displays the received combined speech text data line by line. As shown in FIG. 6A, the text display unit 14 displays the text "passerby collapses".

Returning to step S11, when the next delimitation time Td elapses (time Td2), the time delimitation audio division unit 11 stores the division audio data d2 of the times Td1 to Td2 at the end of the audio queue of the audio buffer 12. At this time, as shown in FIG. 5B, the divided audio data d2 is stored in the index 1 of the audio queue. At the same time, the time-segmented audio dividing unit 11 transmits the divided audio data d2 of the times Td1 to Td2 to the divided audio transmitting unit 22 .

The voice recognition unit 13 receives the divided voice data d2 from the divided voice transmission unit 22 . Then, the speech recognition unit 13 refers to the recognition DB 15 and converts the divided speech data d2 into divided speech text data TX2 (step S12). The content of the divided voice text data TX2 is "It's heartbreaking". The divided speech text data TX2 is input to the text judgment section 21 via the division speech text receiving section 23. FIG. Then, the text determination unit 21 determines whether or not the divided voice text data TX2 is empty (step S13).

A determination result by the text determination unit 21 is input to the buffer control unit 31 . As described above, the split speech text data TX2 is not empty. If the divided speech text data is not empty (step S13 NO), the buffer control unit 31 combines the divided speech data stored in the speech buffer 12 in order from the beginning to generate combined speech data, and sends the data to the speech recognition unit 13. Send (step S14). At this time, since the divided audio data stored in the audio queue have indexes 0 and 1, the divided audio data with indexes 0 and 1 are combined and input to the audio recognition section 13 via the combined audio transmission section 32. . FIG. 7 shows combined speech data input to the speech recognition unit 13 at this time.

Then, the speech recognition unit 13 refers to the recognition DB 15 and converts the combined voice data into combined voice text data. This combined speech text data is input to the text display section 14 via the combined speech text receiving section 33 . The text display unit 14 displays the received combined speech text data (step S15). The content of the combined speech text data at this time is "The passerby is lying down and it hurts my heart". As shown in FIG. 6B, the text display unit 14 displays the text "passerby is lying down and my chest hurts".

Similarly, steps S11 to S15 are repeated until time Td6 has passed (that is, for divided audio data d3 to d6 at times Td2 to Td3, Td3 to Td4, Td4 to Td5, and Td5 to Td6). FIG. 5C shows the divided audio data stored in the audio buffer 12 after time Td6 has elapsed. As shown in FIG. 5C, the audio buffer 12 stores divided audio data d1 to d6 at indexes 0 to 5, respectively. FIG. 6C shows the text displayed on the text display section 14 after voice recognition of combined voice data obtained by combining the divided voice data d1 to d6 of FIG. 5C in order from the beginning.

Next, the operation after time Td7 has elapsed will be described. It is assumed that the divided audio data is silent between times Td6 and Td7. In step S11, when the next delimitation time Td elapses (time Td7), the time delimitation audio dividing unit 11 stores the demultiplexed audio data d7 of times Td6 to Td7 at the end of the audio queue of the audio buffer 12. FIG.

As shown in FIG. 5D, the divided audio data d7 is stored in index 6 of the audio queue. At the same time, the time-segmented audio dividing section 11 transmits the divided audio data d7 of the times Td6 to Td7 to the divided audio transmitting section 22. FIG.

The voice recognition unit 13 receives the divided voice data d7 from the divided voice transmission unit 22. FIG. Then, the speech recognition unit 13 refers to the recognition DB 15 and converts the divided speech data d7 into the divided speech text data TX7 (step S12). As described above, since the divided voice data is silent, the content of the divided voice text data TX2 is empty. The divided speech text data TX7 is input to the text judgment section 21 via the division speech text receiving section 23. FIG. Then, the text determination unit 21 determines whether or not the divided voice text data TX7 is empty (step S13).

A determination result by the text determination unit 21 is input to the buffer control unit 31 . As described above, since the divided voice text data TX7 is empty (step S13 YES), the buffer control unit 31 deletes the divided voice data stored in the voice buffer 12 (step S16) and clears the voice queue. do.

The text displayed on the text display section 14 at this time is shown in FIG. 6D. Since conversion of all the voice data to text has been completed at the time when time Td6 has elapsed, there is no difference between the contents of the texts displayed in FIGS. 6C and 6D. In this way, each time the delimiting time Td elapses, the gradually combined text is displayed line by line on the text display section 14 .

In Comparative Example 1, in which the audio data is forcibly divided by a predetermined fixed time, the text displayed on the text display section is simply combined text obtained by recognizing the audio data for each fixed time. The text obtained by speech recognition of speech data for each fixed time cannot be analyzed based on the relationship with the words before and after, and it is a text of speech data that may be separated in the middle of a word. There is a possibility that simply combining such texts may result in content that cannot be understood as sentences.

On the other hand, according to Embodiment 1, it is possible to re-recognize the combined voice data obtained by combining all the divided voice data up to that time each time the division time is reached. Therefore, the speech recognition unit 13 can perform analysis based on the relationship between words before and after the words, so that words can be converted into text with improved recognition accuracy without being broken in the middle of the words.

Further, in Comparative Example 2 in which audio is divided by silent intervals, when the audio data in FIG. The divided voice data is not generated, and the text is displayed by recognizing the divided voice data for the first time after Td5'. On the other hand, according to Embodiment 1, the speech data up to that point can be combined and converted into text at each delimitation time Td. As a result, it is possible to display the text so that sentences are gradually constructed at each delimiting time Td before Td5' elapses, and speech recognition with improved real-time performance compared to Comparative Example 2 is possible.

Embodiment 2.
FIG. 8 is a block diagram showing the configuration of a speech recognition display device 1B according to the second embodiment. In the second embodiment, as in the first embodiment, voice recognition is performed in real time on the call content (call voice waveform) at the IP telephone 40, and the text is displayed. As described above, in Embodiment 1, the combined speech text data updated at each delimitation time Td are sequentially displayed line by line. In the second embodiment, in order to make the display of the text display section 14 easier to see, the text displayed on the text display section 14 is arranged in units of sentences.

Embodiment 2 differs from Embodiment 1 in that the combined voice/text control unit 30 further includes a text buffer 34, and the buffer control unit 31 controls the voice buffer 12 and the text buffer 34. is. In the following, differences from the first embodiment will be described in detail, and duplication of description will be omitted as appropriate.

Upon receiving the combined voice text data at each interval Td, the combined voice text reception unit 33 stores the combined voice text data received from the voice recognition unit 13 in the index with the smallest number among the empty indexes of the text buffer 34 . Note that the "empty index" is an index that does not have a linefeed code at the end of the line. In other words, the empty index stores either no combined voice text data, or stores combined voice text data that has not been determined as one sentence without a line feed code at the end of the line.

If the combined speech text data already exists in the index, the combined voice text reception unit 33 overwrites the existing data with new data. That is, the combined voice-text data is updated to new combined voice-text data each time the interval time elapses.

If the determination result received from the text determination unit 21 is "empty", the buffer control unit 31 determines the combined speech text data stored in the text buffer 34 as one sentence. The buffer control unit 31, for example, assigns a line feed code [EOL] to the end of the line of the combined voice text data stored in the text buffer 34, thereby determining the combined voice text data as one sentence.

The index in which the combined speech text data determined as one sentence is stored is the used index. In this case, next time the combined speech text data is received by the combined voice text receiving unit 33, the combined voice text data will be stored in the empty index immediately after the used index stored last time.

The text display unit 14 reads out the combined speech text data from the text buffer 34 and displays the text in order from the top index. For example, the text display unit 14 can read the combined voice text data stored in the text buffer 34 and display the text at each read time Tr shorter than the delimiter time Td.

Here, a speech recognition method according to Embodiment 2 will be described with reference to FIGS. 9 and 10A to 10D and 11A to 11D. FIG. 9 is a flowchart for explaining the speech recognition method according to the second embodiment. 10A-10D are diagrams showing the combined speech text data saved in the text buffer after each time. 11A to 11D are diagrams showing text displayed on the text display portion after each time has elapsed.

In addition, in FIG. 9, the same steps as in FIG. 4 are given the same reference numerals. It is assumed that the contents of the call, the voice waveform of the call, the method of delimiting the voice data, and the transition of the contents of the voice queue in the voice buffer 12 in the IP telephone 40 are the same as those in the first embodiment (FIGS. 3, 5A to 5D). ).

As shown in FIG. 9, the processes of steps S10 to S14 are executed in the same manner as in the first embodiment when the first delimiting time Td has passed (time Td1). Then, the combined voice text receiving unit 33 stores the combined voice text data converted by the voice recognition unit 13 in the text buffer 34 (step S17). The combined voice text receiving unit 33 stores the combined voice text data of "Passerby falls down" in the index with the smallest number among the indexes in the text buffer 34 which do not have a linefeed code at the end of the line.

The contents of the text file stored in the text buffer 34 at this time are shown in FIG. 10A. The text file is empty when writing the combined speech text data of "passerby falls". Therefore, the first line of this text file does not store the combined speech text data and corresponds to the lowest numbered free index. As shown in FIG. 10A, the combined speech-text data of "passerby collapses" is written to the first line of the text file. At this time, the line (first line) without a line feed code at the end of the text file becomes the last line of the text file.

The text display unit 14 then reads the text file from the text buffer 34 and displays the combined speech text data (step S18). For example, the text display unit 14 can read the text file in a readout time Tr shorter than the delimiter time Td and display the text. At this time, as shown in FIG. 11A, the text display unit 14 displays the text "passerby collapses".

Returning to step S11, when the next delimitation time Td has elapsed (time Td2), steps S11 to S14 are executed again. As shown in FIG. 10A, the first line of the text file stores the combined speech text data of "Passerby falls down", but no line feed code is added. For this reason, the first line of the text file corresponds to the lowest numbered index in which a line feed code is not added at the end of the line and combined speech text data which is not determined as one sentence is stored.

In step S17, the first line of the text file is overwritten with the combined speech text data "The passerby is lying down and my chest hurts." As shown in FIG. 10B, the combined voice-text data "Passerby is lying down and my chest hurts" is written to the first line of the text file. Then, in step S18, as shown in FIG. 11B, the text display unit 14 displays the text "passerby is lying down and my chest hurts".

Similarly, steps S11 to S14, S17, and S18 are repeatedly executed until time Td6 has elapsed. FIG. 10C shows the combined speech text data stored in the text buffer 34 after time Td6. As shown in FIG. 10C, in the text buffer 34, in the first line of the text file, there is combined speech text data of "A passerby is lying down and complaining of chest pain, and he says that there is no family hospital." is stored. Then, as shown in FIG. 11C, the text display section 14 displays similar text.

Next, the operation after time Td7 has elapsed will be described. Between times Td6 and Td7, the divided audio data is silent. In step S13, since the divided voice text data TX7 is empty (YES), the process proceeds to step S19. In step S19, the buffer control unit 31 deletes the divided audio data stored in the audio buffer 12, empties the audio queue, and adds a line feed code [EOL] to the end of the combined audio text data. The text file stored in the text buffer 34 at this time is shown in FIG. 10D.

Also, the text displayed by the text display unit 14 at this time is shown in FIG. 11D. Since conversion of all audio data to text has been completed by the time Td6 has passed, there is no difference between the contents of the texts displayed in FIGS. 11C and 11D. Thus, in the second embodiment, the texts that are gradually combined are displayed on the text display section 14 each time the delimiting time Td elapses.

Thus, in the second embodiment, a text file is used, and the combined speech text data is overwritten and updated up to the break of the sentence. As a result, the speech recognition text is displayed in units of sentences on the text display section 14, making it easier to see than in the first embodiment.

As described above, in Comparative Example 2 in which voice is divided according to silent intervals, there are cases where voice data is not divided into sentences depending on the voice level determined to be silent. Therefore, an example of improving such a problem by using the speech recognition display device 1B described in the second embodiment will be described. Here, speech recognition is performed on the utterance content and speech waveform in FIG. FIG. 12 shows the speech waveform of the utterance content of the caller who called 119, "I can see a convenience store nearby. There are no other injured people." In FIG. 12, the vertical axis represents the audio level (amplitude) of the audio waveform, and the horizontal axis represents time.

In Comparative Example 2, in which voice is divided according to silent intervals, the voice levels determined to be silent are assumed to be -A1 to A1 in FIG. If all of the audio waveforms fall within -A1 to A1, it is determined that there is no sound, and the audio data is divided at that time. Between the utterances of the caller, ``I can see a convenience store nearby,'' and ``There are no other injured people.'' The sound level during the time Tn when sound is heard and no speech is not within -A1 to A1.

For this reason, in Comparative Example 2, it is not possible to determine the time Tn during which no speech is made as a silent section, and it is not possible to separate the voice data. Therefore, in Comparative Example 2, "I can see a convenience store nearby. There are no other injured people." in the entire voice data is converted into text at once.

Therefore, in the speech recognition display device 1B, the interval time Td is set to Td<1/2Tn. As a result, the voice data of the no-speech time is always stored in the voice queue. Since the speech recognition unit 13 does not convert the noise sound into text, when the speech data of the non-speech section is stored by the operation of the buffer control unit 31 described above, a line feed code [EOL] is added to the end of the line of the combined speech text data. Granted. In other words, the entire voice data "I can see a convenience store nearby. There are no other injured people." It will be converted into text and displayed on the text display section 14 .

As described above, according to the second embodiment, it is possible to divide the speech into text units without being influenced by the surrounding environmental sounds. It is possible to solve the problem that there are cases where the audio data is not divided into units.

Embodiment 3.
In the second embodiment, the divided voice data is input from the divided voice transmission unit 22, and the combined voice data is input from the combined voice transmission unit 32 to the voice recognition unit 13 separately. The combined voice-text data is separately received by the combined voice-text receiving unit 33 . In order to simplify this configuration, in the third embodiment, by setting a flag, the function of transmitting the divided audio data and the combined audio data is combined into one, and the divided audio data and the combined audio data are received. Combine features into one.

FIG. 13 is a block diagram showing the configuration of a speech recognition display device 1C according to the third embodiment. In the example shown in FIG. 13, it is assumed that the speech of the analog telephone 41 is recognized in real time and displayed as text. As shown in FIG. 13, the speech recognition display device 1C includes a time segmented speech division unit 11, a speech buffer 12, a speech recognition unit 13, a text display unit 14, a recognition DB 15, a text determination unit 21, a buffer control unit 31, a text buffer 34 , a shared memory 50 , a voice transmitter 51 and a text receiver 52 .

Embodiment 3 differs from Embodiment 2 in that it is possible to set a flag indicating whether voice transmission unit 51 is transmitting divided voice data or combined voice data, and text reception unit 52 , and selectively executes whether to transmit the divided speech data to the text determination unit 21 or to write the combined speech data to the text buffer 34 . In the following, differences from the first embodiment will be described in detail, and duplication of description will be omitted as appropriate.

The voice acquisition unit 10 converts an analog voice signal output from the analog telephone 41 into digital voice data by an analog-digital conversion unit (AD conversion unit) 10A included in the voice acquisition unit 10, and converts it into time-delimited voice data. Output to the dividing unit 11 . A time-segmented audio dividing unit 11 generates divided audio data by dividing audio data by a preset division time Td, and stores the divided audio data in an audio buffer 12 in order from the beginning every time the division time Td elapses. do. Also, the time-segmented audio dividing unit 11 transmits the divided audio data to the audio transmitting unit 51 .

The audio transmission unit 51 receives the combined audio data from the buffer control unit 31 in addition to the divided audio data. The audio transmission unit 51 transmits the divided audio data received from the time-delimited audio dividing unit 11 and the combined audio data received from the buffer control unit 31 to the audio recognition unit 13 . At this time, the audio transmission unit 51 sets a FALSE flag in the shared memory 50 when the divided audio data is received from the time-segmented audio dividing unit 11 in the shared memory 50, and receives the combined audio data from the buffer control unit 31. If so, set the TRUE flag.

The speech recognition unit 13 refers to the recognition DB 15 , converts the divided speech data into divided speech text data, converts the combined speech data into combined voice text data, and transmits the data to the text reception unit 52 . The shared memory 50 can be accessed from the voice transmission section 51 and the text reception section 52 . The text receiving unit 52 refers to the flag set in the shared memory 50, and inputs the divided speech text data to the text determining unit 21 when the flag is FALSE. Also, when the flag is TRUE, the text receiving unit 52 writes the combined speech text data to the index with the smallest number among the free indexes in the text buffer 34 where no linefeed code is added at the end of the line. At this time, if the combined speech text data already exists in the index, the existing data is overwritten with the new data.

As described above, the text determination unit 21 determines whether the divided voice text data is “empty” and inputs the determination result to the buffer control unit 31 . If the determination result is “not empty”, the buffer control unit 31 generates combined audio data by combining the divided audio data stored in the audio buffer 12 in order from the beginning, and transmits the combined audio data to the audio transmission unit 51 . On the other hand, if the determination result is "empty", the buffer control unit 31 deletes the divided audio data stored in the audio buffer 12, empties the audio queue, and inserts a line feed at the end of the line of the combined audio text data. Add a code [EOL]. The text display unit 14 reads the text file in the text buffer 34 at a read time Tr (Tr<Td) and displays the text.

A speech recognition method according to Embodiment 3 will be described below. In the example shown here, in the voice acquisition unit 10, the analog voice input from the analog telephone 41 is sequentially AD-converted from time 0, and the voice data after AD conversion is sent to the time-delimited voice division unit 11. Sequentially entered. The waveform of the voice data, how to delimit the voice data, the transition of the contents of the voice queue in the voice buffer 12, the transition of the contents of the text file of the text in the text buffer 34, and the text displayed on the text display section 14 are described in the implementation. It shall be similar to form 2 (FIGS. 3, 5A-5D, 10A-10D, 11A-11D).

The time-segmented audio dividing unit 11 stores the divided audio data d1 from time 0 to Td1 at the end of the audio queue of the audio buffer 12 when the time Td1 has passed. At this time, since the audio queue is empty, the divided audio data d1 is stored at index 0 of the audio queue. The content of the audio queue at this time is the same as in FIG. 5A. At the same time, the time-segmented audio dividing section 11 transmits the divided audio data d1 to the audio transmitting section 51 .

Since the audio transmission unit 51 has received the divided audio data d1 from the time-delimited audio division unit 11, it sets the FALSE flag in the shared memory 50. FIG. Also, the voice transmission unit 51 transmits the divided voice data d1 to the voice recognition unit 13 . The voice recognition unit 13 refers to the recognition DB 15 to convert the divided voice data d1 into divided voice text data TX1 and transmits the data to the text receiving unit 52 . The divided voice text data TX1 at this time is "a passerby collapses".

The text receiving unit 52 refers to the flag stored in the shared memory 50, and since the flag is FALSE, the text receiving unit 52 transmits the divided speech text data TX1 of "passerby collapses" to the text determining unit 21. FIG. The text determination unit 21 determines whether or not the divided voice text data TX1 from the text reception unit 52 is empty, and transmits the determination result to the buffer control unit 31 . Since the divided voice text data TX1 is not empty, the buffer control unit 31 combines the divided voice data stored in the voice buffer 12 in order from the beginning.

At this time, since the divided audio data stored in the audio buffer 12 is only the data of index 0, the data of index 0 is transmitted to the audio transmission unit 51 as combined audio data. Since the audio transmission unit 51 has received the combined audio data from the buffer control unit 31 , it sets the TRUE flag in the shared memory 50 . Also, the voice transmission unit 51 transmits the combined voice data to the voice recognition unit 13 .

The voice recognition unit 13 refers to the recognition DB 15 to convert the combined voice data into combined voice text data and transmits the combined voice text data to the text receiving unit 52 . The combined speech text data at this time is "a passerby collapses".

The text receiving unit 52 refers to the flag stored in the shared memory 50 , and writes the combined voice text data of “the passerby collapses” to the text buffer 34 because the flag is TRUE. As described above, the combined voice-text data of "passerby collapses" is written in the first line of the text file without a linefeed code at the end of the line (FIG. 10A). Then, the text display unit 14 reads the text file stored in the text buffer 34 at the read time Tr (Tr<Td) and displays the text (FIG. 11A).

When the next time interval Td elapses (time Td2), the time-interval audio dividing unit 11 stores the divided audio data d2 of times Td1 to Td2 at the end of the audio buffer 12. FIG. As shown in FIG. 5B, the divided audio data d2 is stored in index 1 of the audio queue. At the same time, the time-segmented audio dividing section 11 transmits the divided audio data d2 to the audio transmitting section 51 .

Since the audio transmission unit 51 has received the divided audio data d2 from the time-delimited audio division unit 11, it sets the FALSE flag in the shared memory 50. FIG. Also, the voice transmission unit 51 transmits the divided voice data d2 to the voice recognition unit 13 . The voice recognition unit 13 refers to the recognition DB 15 to convert the divided voice data d2 into divided voice text data TX2 and transmits the data to the text receiving unit 52 . The divided voice text data TX2 at this time is "It's heartbreaking".

The text receiving unit 52 refers to the flag stored in the shared memory 50, and since the flag is FALSE, it transmits the divided speech text data TX2 of "It hurts my chest" to the text determining unit 21. The text determination section 21 determines whether or not the divided voice text data TX2 from the text reception section 52 is empty, and transmits the determination result to the buffer control section 31 . Since the divided voice text data TX2 is not empty, the buffer control unit 31 combines the divided voice data stored in the voice buffer 12 in order from the beginning. The combined voice data at this time is the same as in FIG.

At this time, since the divided audio data stored in the audio buffer 12 are the data of indexes 0 and 1, the data of indexes 0 and 1 are transmitted to the audio transmission unit 51 as combined audio data. Since the audio transmission unit 51 has received the combined audio data from the buffer control unit 31 , it sets the TRUE flag in the shared memory 50 . Also, the voice transmission unit 51 transmits the combined voice data to the voice recognition unit 13 .

The voice recognition unit 13 refers to the recognition DB 15 to convert the combined voice data into combined voice text data and transmits the combined voice text data to the text receiving unit 52 . The combined voice-text data at this time is "My chest hurts because the passerby is lying down".

The text receiving unit 52 refers to the flag stored in the shared memory 50, and since the flag is TRUE, writes the combined voice text data of "passerby is lying down and my chest hurts" in the text buffer 34.例文帳に追加As shown in FIG. 10A, the first line of the text file stores the combined voice data of "passerby falls", but no line feed code is added. For this reason, the first line of the text file is overwritten with the combined speech text data "A passerby is lying down and my chest hurts" (FIG. 10B). The text display unit 14 then reads the text file stored in the text buffer 34 and displays the text (FIG. 11B).

Thereafter, similar processing is repeatedly executed until time Td6 has elapsed. FIG. 10C shows the combined speech text data stored in the text buffer 34 after time Td6. FIG. 11C shows the text displayed on the text display section 14 at this time.

Next, the operation after time Td7 has elapsed will be described. Between times Td6 and Td7, the divided audio data is silent. After the time Td7 has passed, the time-segmented audio dividing section 11 stores the divided audio data d7 of the times Td6 to Td7 at the end of the audio buffer 12. FIG. As shown in FIG. 5D, the divided audio data d7 is stored in index 6 of the audio queue. At the same time, the time-segmented audio dividing section 11 transmits the divided audio data d7 to the audio transmitting section 51 .

Since the audio transmission unit 51 has received the divided audio data d7 from the time-segmented audio dividing unit 11, it sets the FALSE flag in the shared memory 50. FIG. Also, the voice transmission unit 51 transmits the divided voice data d7 to the voice recognition unit 13 . The voice recognition unit 13 refers to the recognition DB 15 to convert the divided voice data d7 into divided voice text data and transmits the data to the text receiving unit 52 . The divided speech text data at this time is "- (empty)".

The text receiving unit 52 refers to the flag stored in the shared memory 50 , and since the flag is FALSE, transmits the divided speech text data of “− (empty)” to the text determining unit 21 . The text determination unit 21 determines whether or not the divided voice text data TX7 from the text reception unit 52 is empty, and transmits the determination result to the buffer control unit 31. FIG. Since this divided voice text data TX7 is empty, the buffer control unit 31 deletes the divided voice data stored in the voice buffer 12, empties the voice queue, and inserts a linefeed code at the end of the line of the combined voice text data. [EOL] is given.

The combined speech text data stored in the text buffer 34 at this time is the same as in FIG. 10D. FIG. 11D shows the text displayed on the text display section 14 at this time. As described above, in the third embodiment, as in the second embodiment, the combined text is gradually displayed on the text display section 14 each time the delimiting time Td elapses.

As described above, according to the third embodiment, as in the second embodiment, the text display unit 14 can display the speech recognition text in units of sentences. In addition, since the function of transmitting the divided speech data and the combined speech data to the speech recognition unit 13 and the function of receiving the divided speech text data and the combined speech text data from the speech recognition unit 13 can be combined into one component, It is possible to simplify the configuration of the speech recognition display device.

Embodiment 4.
In the fourth embodiment, two speech recognition display devices 1B of the second embodiment are used to display the contents of the speeches of the two speakers in chronological order. FIG. 14 is a diagram showing the configuration of a speech recognition display device 1D according to the fourth embodiment. In FIG. 14, the same reference numerals are assigned to the same components as in the second embodiment. In addition, in order to distinguish between the voice recognition display devices 1B used by two speakers X and Y, each element is denoted by X or Y.

In the example shown in FIG. 14, the speech recognition display device 1B used by two speakers (speaker X, speaker Y) uses one speech recognition unit 13 in common. 13 may be provided separately. In addition, although the contents uttered by two speakers are collectively displayed on one text display section 14, the text display sections 14 are separately provided for speaker X and speaker Y to display the same content. may

In Embodiment 4, the utterance contents of speakers X and Y picked up by two microphones 42 (microphone X and microphone Y) are displayed in chronological order. As shown in FIG. 14, the speech recognition display device 1D according to the fourth embodiment includes two speech recognition display devices 1B described in the second embodiment.

The speech recognition display device 1D further includes a text merger 60 and a merged text buffer 61 . The text merge unit 60 merges the text file X stored in the text buffer X and the text file Y stored in the text buffer Y. FIG. The merged text buffer 61 stores merged text data merged by the text merger 60 . The text display unit 14 reads out the merged text buffer 61 and displays the merged text data.

In the fourth embodiment, unlike the second embodiment, the text files X and Y stored in the text buffers X and Y divide the text data into several fields (items) and delimit the information of each item. It is a text file in CSV (character-separated values) format using commas and tabs as characters (delimiters). It is assumed that text files X and Y have TIME_FIELD (first division) and TEXT_FIELD (second division).

Merged text data obtained by merging text files X and Y is also a text file in CSV format. Assume that the merged text data has SPEAKER_FIELD (1st division), TIME_FIELD (2nd division), and TEXT_FIELD (3rd division).

15A and 15B are diagrams showing divided audio data saved in audio buffers X and Y after three division times have passed. FIGS. 16A and 16B, FIGS. 17A and 17B, and FIGS. 18A and 18B are diagrams showing the divided speech text data stored in the text buffers X and Y after the first to third delimiting times have passed, respectively. be. 19A to 19F show the saved state of the merged text buffer from before merging to the fourth merging. 20A to 20F are diagrams showing display states of the text display portion from before display to the fifth display.

The time-delimited audio division units X and Y store the divided audio data at the end of the queues (audio queues X and Y) of the audio buffers X and Y each time the division time elapses. After the three division times have elapsed, the audio cues X and Y become those shown in FIGS. 15A and 15B. In the fourth embodiment, when storing divided audio data in the audio cues X and Y, the time-segmented audio dividing units X and Y associate the storage time of the divided audio data with each index of the audio cues A and B. save.

If the judgment result received from the text judging units X and Y is "not empty", the buffer control units X and Y combine the divided audio data stored in the audio queues X and Y from the beginning, and transmit the combined audio transmitting unit. Send to X and Y respectively. This operation is similar to that of the second embodiment. In addition to this, the buffer control units X and Y store the storage time linked to the divided audio data at the head index of the audio queues X and Y to the last line of the text files X and Y (where the line feed code [EOL] is Write to the TIME_FIELD of the unsigned row).

When the determination result received from the text determination units X and Y is "empty", the buffer control units X and Y read the divided audio data stored in the audio buffers X and Y as in the second embodiment. Empty each audio queue X, Y.

The operation of the speech recognition display device 1D shown in FIG. 14 will be described below along the time axis. Referring to FIGS. 15A and 15B, when the first and second division times have elapsed in the time division audio division units X and Y, each divided audio data is not empty. Therefore, the determination results received by the buffer control units X and Y from the text determination units X and Y are both "not empty". Therefore, after the first delimitation time has elapsed, the text file X is as shown in FIG. 16A and the text file Y is as shown in FIG. 16B. After the second delimitation time has elapsed, the text file X is shown in FIG. 17A and the text file Y is shown in FIG. 17B.

After that, when the third division time has elapsed in the time division audio division units X and Y, the division audio data is empty. Therefore, the determination results received by the buffer control units X and Y from the text determination units X and Y are both "empty". At this time, the buffer controllers X and Y add a line feed code [EOL] to the end of the last line of the text files X and Y, respectively. Therefore, after the third delimitation time has passed, the text file X is as shown in FIG. 18A and the text file Y is as shown in FIG. 18B.

The text merging section 60 reads out the text file X and the text file Y and updates the merged text file stored in the merged text buffer 61 every time the merged time Tm (Tm<Td) elapses. The text merging section 60 first reads the text file X stored in the text buffer X. As shown in FIG. Then, the text merging unit 60 uses the TIME_FIELD time of the last line of the text file X (the line to which the line feed code [EOL] is not added) as a key to merge the TIME_FIELD of the merged text file stored in the merged text buffer 61 . and the SPEAKER_FIELD is "speaker X".

If the corresponding line exists, the TEXT_FIELD of that line is overwritten with the contents of the TEXT_FIELD of the text file X. If the corresponding line does not exist, the TIME_FIELD time of the last line of the text file X is referenced, and the TIME_FIELD times of the merged text files are sorted in ascending order (that is, from oldest to newest as the index number increases). The contents of the text file X are written in the corresponding line of the merged text file so that

Specifically, in the corresponding line of the merged text file, "speaker X" is written in SPEAKER_FIELD, the time in TIME_FIELD of the last line of text file X in TIME_FIELD, and the content of TEXT_FIELD in the last line of text file X in TEXT_FIELD. As for the text file Y, the contents of the text file Y are written in the corresponding line of the merged text file by the same operation as the text file X.

As an example, the operation flow of the text merging section 60 will be described with the delimiting time Td of 500 msec, the merging time Tm of 400 msec, and the first merging time of the text merging section 60 operating at 13:00:15.400.

First, before the operation of the text merging section 60 starts (13:00:15.000) (before merging), the merged text file saved in the merged text buffer 61 is empty as shown in FIG. 19A. At the time of the first merging (13:00:15.400) after 400 msec of merging time has passed, the text file X is as shown in FIG. 16A and the text file Y is empty. At this time, the operation of the text merging section 60 results in a merged text file as shown in FIG. 19B.

After that, the merged text file is shown in FIG. 19C at the time of the second merge (13:00:15.800), FIG. 19E at the time of 16.600), and FIG. 19F at the time of the fifth merge (13:00:17.000).

The text display unit 14 reads and displays the merged text file at a predetermined read time Tr (Tr<Tm). As shown in FIG. 20A, in the fourth embodiment, the text display section 14 has two areas, a display area for speaker X (left side) and a display area for speaker Y (right side). The text display unit 14 reads out the merged text file, excluding the FIELD line, one line at a time from the top line, and displays them in order from the top line on the text display means. At this time, the SPEAKER_FIELD of the merged text file displays the merged text data of "speaker X" in the display area for speaker X (left side). Also, SPEAKER_FIELD displays the combined text data of "speaker Y" in the display area for speaker Y (right side).

As described above, the merge time Tm is assumed to be 400 msec, and the readout time Tr is assumed to be 300 msec, for example. The flow of operation of the text display section 14 will be described assuming that the first display time at which the text display section 14 operates is 13:00:15.600.

First, before the operation of the text display section 14 starts (13:00:15.000) (before display), the text displayed on the text display section 14 is empty as shown in FIG. 20A. At the time of the first display (13:00:15.600) when the readout time of 300 ms has passed, the merged text file is shown in FIG. 19B, so the text displayed on the text display section 14 is as shown in FIG. 20B.

After that, at the time of the second display (13:00:15.900), the merged text file is shown in FIG. 19C, so the text displayed on the text display section 14 is as shown in FIG. 20C. Then, at the third display (13:00:16.200), the merged text file is as shown in FIG. 19D, so the text displayed on the text display section 14 is as shown in FIG. 20D.

At the time of the fourth display (13:00:16.500), the merged text file remains as shown in FIG. 19D. It looks like FIG. 20E. At the time of the fifth display (13:00:16.800), the merged text file is shown in FIG. 19E, so the text displayed on the text display section 14 is shown in FIG. 20F.

As described above, in the fourth embodiment, it is possible to display the conversation contents of two speakers like a chat while ensuring the same real-time performance as in the second embodiment. Here, an example using two microphones is shown, but for example, it is also possible to display the transmitted voice on the telephone as one voice data and the received voice as the other voice data and display them in a chat format. is. As a result, it is possible to convert the voice of the caller and the caller into text in real time in a chat format in an emergency call system, etc. dispatch of an ambulance, etc.) can be carried out quickly.

Embodiment 5.
FIG. 21 is a block diagram showing the configuration of a speech recognition display device 1E according to the fifth embodiment. As shown in FIG. 21, the speech recognition display device 1E further includes an overall speech buffer 16 and a speech reproduction section 17 in addition to the configuration of the speech recognition display device 1B of the second embodiment.

In the example shown in FIG. 21, the speech recognition display device 1E converts the speech content collected by the microphone 42 into text in real time, and automatically converts the text and the corresponding voice each time the text conversion of one sentence is completed. or read out the corresponding voice in units of sentences by user operation. Hereinafter, differences from the second embodiment will be described in detail, and duplication of description will be omitted as appropriate.

The audio acquisition unit 10 transmits the acquired audio data to the time-segmented audio dividing unit 11 and stores all of the audio data in the entire audio buffer 16 . In Embodiment 5, when storing divided audio data in the audio buffer 12, the time-segmented audio dividing unit 11 increments the division number counter by 1 and stores the divided audio data in association with each index of the audio cue. Note that the division number counter is a counter that measures the number of divisions of the audio data, and its initial value is zero.

Also, unlike the second embodiment, the text file stored in the text buffer 34 divides the text data into several fields (items) and uses commas and tabs as delimiters (delimiters) to separate the information of each item. It is a text file in CSV format. It is assumed that this text file has COUNT_FIELD (first division) and TEXT_FIELD (second division).

As in the second embodiment, when the determination result input from the text determination unit 21 is "not empty", the buffer control unit 31 combines the divided audio data stored in the audio queue from the beginning, and transmits the combined audio. 32. At this time, the buffer control unit 31 sets the division count counter value linked to the divided audio data at the head index of the audio queue to the last line of the text file stored in the text buffer 34 (added with a line feed code [EOL]). Write to the COUNT_FIELD of the unfilled row).

When the determination result input from the text determination unit 21 is "empty", the buffer control unit 31 deletes the divided audio data stored in the audio buffer 12, and empty.

The text display unit 14 displays the above-described text, and includes the division count counter value of the text file read out to the audio reproduction unit 17 automatically or by user operation each time text conversion of one sentence is completed. Outputs an audio playback instruction. Note that the text display unit 14 may be configured to accept user operations using a touch panel in which a display device and an input device are integrated. Upon receiving the audio reproduction instruction, the audio reproduction unit 17 reads the audio data stored in the entire audio buffer 16 and reproduces the audio data from the reproduction start position based on the division count counter value.

The operation of the speech recognition display device 1E shown in FIG. 21 will be described below along the time axis. FIG. 22 is a diagram showing an example of the utterance content and the utterance voice waveform collected by the microphone 42 of FIG. In FIG. 22, it is assumed that the voice start time is 0 and the predetermined interval time is Td. As shown in FIG. 22, the time when the first interval time Td elapses is defined as Td1, and thereafter, Td2→Td3→Td4→Td5→Td6→Td7 in order each time the interval time Td elapses. The time-segmented audio dividing unit 11 divides the audio data into times Td1 to Td7.

FIG. 23A is a diagram showing the divided audio data stored in the audio buffer 12 after the three division times Td of the first sentence have elapsed (after the time Td3 has elapsed). FIG. 23B is a diagram showing the divided audio data stored in the audio buffer 12 after the third division time Td of the second sentence has elapsed (after time Td7 has elapsed).

FIGS. 24A to 24G are diagrams showing combined speech text data saved in the text buffer after times Td1 to Td7, respectively. 25A shows the initial display state of the text display section 14. FIG. 25B to 25G are diagrams showing text displayed on the text display section 14 after times Td1 to Td7 have passed, respectively.

Referring to FIG. 23A, in the first audio division at time Td1 and the second audio division at time Td2, each divided audio data is not empty, so the determination result by the text determination unit 21 is "not empty". Therefore, the text file after time Td1 has passed is shown in FIG. 24A, and the text file after time Td2 has passed is shown in FIG. 24B.

After that, in the third audio division at time Td3, since the divided audio data is empty, the judgment result by the text judging section 21 is "empty". Therefore, the buffer control unit 31 adds a line feed code [EOL] to the end of the last line (first line) of the text file. Therefore, the text file after Td3 has passed is shown in FIG. 24C.

Next, referring to FIG. 23B, in the 4th, 5th, and 6th audio divisions at times Td4, Td5, and Td6, the respective divided audio data are not empty, so the judgment result by the text judgment unit 21 is "not empty." becomes. Therefore, the text file after time Td4 has passed is shown in FIG. 24D, the text file after time Td5 has passed is shown in FIG. 24E, and the text file after time Td6 has passed is shown in FIG. 24F.

After that, in the seventh audio division at time Td7, since the divided audio data is empty, the judgment result by the text judging section 21 is "empty". Therefore, the buffer control unit 31 adds a line feed code [EOL] to the end of the last line (second line) of the text file. Therefore, the text file after Td7 has passed is shown in FIG. 24G.

The text display unit 14 reads and displays the text file at a predetermined read time Tr (Tr<Td). Also, the text display unit 14 can display a [playback] button on the right side of the display area. For example, the text display unit 14 can hide the [playback] button in the initial display, and display the [playback] button on the right side of the line when some text is displayed. The audio reproducing unit 17 can reproduce the audio data in response to the user's pressing of the [Playback] button.

At this time, each [Playback] button is associated with a division number counter value corresponding to the text displayed in the line. When the user presses the [playback] button, the text display unit 14 transmits a voice playback instruction including the division number counter value corresponding to the text of the line to the voice playback unit 17 . In addition, when the text display unit 14 detects a line with a line feed code [EOL] at the timing when the text file is read, the value of the COUNT_FIELD (divided number counter value) of the line is sent to the voice playback unit 17 along with voice playback. May output instructions.

Here, the flow of operation of the text display unit 14 will be described. FIG. 25A shows an initial display state of the text display section 14. As shown in FIG. After time Td1 has passed (first audio division), the text file saved in the text buffer 34 becomes the one shown in FIG. 24A. In the text display section 14, the text is displayed on the first line as shown in FIG. 25B. At this time, since the text is displayed on the line for the first time, a [Playback] button is displayed on the right side of the line.

After time Td2 has passed (second audio division), the text file saved in the text buffer 34 becomes the one shown in FIG. 24B. In the text display section 14, the text on the first line is updated as shown in FIG. 25C. At this time, since text has already been displayed in the line, the [Playback] button remains displayed.

After time Td3 has passed (the third audio division), the text file saved in the text buffer 34 becomes the text file shown in FIG. 24C. The difference between FIG. 24B and FIG. 24C is that the line feed code [EOL] is added to the end of the first line. Therefore, as shown in FIG. do not.

When the text display unit 14 detects the line feed code [EOL] in the first line of the text file, the text display unit 14 displays the division number count value "1" written in the COUNT_FIELD of the first line of the text file to the voice reproduction unit 17. , the audio playback instruction may be automatically sent.

After time Td4 has passed (fourth audio division), the text file saved in the text buffer 34 becomes the one shown in FIG. 24D. In the text display section 14, the second line of text is displayed as shown in FIG. 25E. At this time, since the text is displayed on the line for the first time, a [Playback] button is displayed on the right side of the line.

When times Td5 and Td6 have passed (fifth and sixth audio divisions), the text files are shown in FIGS. 24E and 24F, respectively. In the text display section 14, the text on the second line is updated as shown in FIGS. 25E and 25F. At this time, since text has already been displayed in the line, the [Playback] button remains displayed.

After time Td7 has passed (seventh audio division), the text file saved in the text buffer 34 becomes the one shown in FIG. 24G. The only difference between FIG. 24F and FIG. 24G is that the line feed code [EOL] is added at the end of the second line, so the text on the second line of the text display section 14 changes as shown in FIG. 25H. do not.

When the text display unit 14 detects the line feed code [EOL] in the second line of the text file, the text display unit 14 displays the division number count value "4" written in the COUNT_FIELD of the second line of the text file to the voice reproduction unit 17. , the audio playback instruction may be automatically sent.

When the audio reproduction unit 17 receives the audio reproduction instruction including the division number counter value from the text display unit 14, the audio reproduction unit 17 acquires the audio data from the entire audio buffer 16, and reproduces the audio from the reproduction start position calculated by the following formula. .
Playback start position (time) = (division number counter value - 1) x Td

As a result, each time text is displayed line by line on the text display unit 14, the voice corresponding to the displayed text is automatically read out.

It should be noted that when the [playback] button is displayed on the text display section 14, the voice corresponding to the text can be played back at any time. That is, without waiting for the end of the sentence (automatic playback), pressing the [Playback] button makes it possible to play back the text and the corresponding voice at the time of pressing. Furthermore, even if the contents of the utterance increase, by pressing the [Playback] button corresponding to the text, it is possible to go back in time and reproduce the voice corresponding to the text.

As described above, according to Embodiment 5, voice recognition is performed on the contents of utterances in real time, and each time a text is displayed, it is possible to automatically reproduce the voice corresponding to the sentence in units of sentences. Also, it is possible to manually reproduce the voice corresponding to the sentence in units of sentences or in the middle of the sentence. As a result, for example, after converting voice data acquired in interviews or the like into text by voice recognition, the effect is demonstrated when an operator compares the text with the voice and manually corrects the text.

It should be noted that each element described in the drawing as a functional block that performs the various processes described above can be configured by a CPU, a memory, and other lines in terms of hardware. In addition, the present invention can also realize arbitrary processing by causing a CPU (Central Processing Unit) to execute a computer program. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and are not limited to either one.

A program includes instructions (or software code) that, when read into a computer, cause the computer to perform one or more of the functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or tangible storage medium. By way of example, and not limitation, computer readable media or tangible storage media may include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drives (SSD) or other memory technology, CDs -ROM, digital versatile disc (DVD), Blu-ray disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disc storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example, and not limitation, transitory computer readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

It should be noted that the present invention is not limited to the above embodiments, and can be modified as appropriate without departing from the scope of the invention.

REFERENCE SIGNS LIST 1 voice recognition display device 10 voice acquisition unit 11 time-delimited voice division unit 12 voice buffer 13 voice recognition unit 14 text display unit 15 recognition DB
16 Whole Speech Buffer 17 Speech Playback Unit 20 Split Speech/Text Control Unit 21 Text Judging Unit 22 Split Speech Sending Unit 23 Split Speech Text Receiving Unit 30 Combined Speech/Text Control Unit 31 Buffer Control Unit 32 Combined Speech Sending Unit 33 Combined Speech Text Receiver 34 Text Buffer 40 IP Telephone 41 Analog Telephone 42 Microphone 50 Shared Memory 51 Voice Transmitter 52 Text Receiver 53 Voice Combiner 60 Text Merger 61 Merged Text Buffer

Claims (10)

an audio acquisition unit that acquires audio data;
an audio dividing unit that divides the audio data along the time axis at predetermined division times to generate divided audio data;
an audio buffer for storing the divided audio data in order from the beginning every time the delimitation time elapses;
a speech recognition unit that converts the divided speech data into divided speech text data;
a text determination unit that determines whether the divided voice text data is empty;
a buffer control unit for generating combined voice data by combining the divided voice data stored in the voice buffer in order from the beginning when the divided voice text data is not empty;
a text display unit for displaying combined speech text data resulting from voice recognition of the combined speech data each time the delimitation time elapses;
A speech recognition display device comprising: The buffer control unit deletes the divided audio data stored in the audio buffer when the divided audio text data is empty.
The speech recognition display device according to claim 1. further comprising a text buffer for storing the combined speech text data;
said combined speech text data is stored in the lowest numbered index among free indexes of said text buffer, updated each time said delimiting time elapses;
the text display unit reads the text buffer and displays the combined speech text data;
3. The speech recognition display device according to claim 1 or 2. When the divided voice text data is empty, the buffer control unit adds a line feed code to the end of the line of the combined voice text data stored in the text buffer, and uses the index in which the combined voice text data is stored. as an index,
The speech recognition display device according to claim 3. if there is a silent section in the audio data, the interval time is less than 1/2 of the silent section;
5. The speech recognition display device according to claim 3 or 4. an audio transmission unit that transmits the divided audio data and the combined audio data to the audio recognition unit and sets a flag indicating which of the divided audio data and the combined audio data is being transmitted;
A text receiving unit that receives the divided speech text data and the combined speech text data, refers to the flag, and transmits the divided speech text data to the text determination unit and the combined speech text data to the text buffer. and,
further comprising
5. The speech recognition display device according to claim 3 or 4. a first speech recognition display device comprising the speech recognition display device configuration according to claim 3 or 4, for converting first speech data into first combined speech text data;
a second speech recognition and display device having the same configuration as the first speech recognition and display device, for converting second speech data different from the first speech data into second combined speech text data;
a text merging unit that merges the first combined voice-text data and the second combined voice-text data to generate merged text data;
with
the text display unit displays the merged text data;
Voice recognition display device. an entire audio buffer storing all of the audio data;
an audio reproduction unit that reproduces the audio data stored in the entire audio buffer from a reproduction start position based on a division number counter value linked to the divided audio data;
further comprising
5. The speech recognition display device according to claim 3 or 4. get audio data,
dividing the audio data along the time axis at predetermined time intervals to generate divided audio data;
storing the divided audio data in an audio buffer in order from the beginning each time the delimitation time elapses;
converting the divided voice data into divided voice text data;
determining whether the divided speech text data is empty;
if the divided voice text data is not empty, generating combined voice data by combining the divided voice data stored in the voice buffer in order from the head;
displaying combined voice text data obtained by performing voice recognition of the combined voice data each time the delimitation time elapses;
Speech recognition display method. a process of acquiring audio data;
a process of dividing the audio data along the time axis at predetermined division times to generate divided audio data;
a process of storing the divided audio data in an audio buffer in order from the top every time the delimitation time elapses;
a process of converting the divided speech data into divided speech text data;
a process of determining whether the divided voice text data is empty;
a process of generating combined voice data by combining the divided voice data stored in the voice buffer in order from the beginning when the divided voice text data is not empty;
a process of displaying combined voice text data obtained by performing voice recognition of the combined voice data each time the delimitation time elapses;
A program that makes a computer run

Images