JP6464465B2 - Conversation support device, method for controlling conversation support device, and program for conversation support device - Google Patents

Description

  The present invention relates to a conversation support apparatus, a method for controlling the conversation support apparatus, and a program for the conversation support apparatus.

  Patent Document 1 discloses a hearing aid device that visually assists hearing by visually recognizing the content of a conversation and displaying characters. In the hearing aid device described in Patent Document 1, the voice recognition unit recognizes the voice collected by the microphone, and displays the character corresponding to the recognized content based on the voice recognition result on the display means. Moreover, in the hearing aid apparatus described in Patent Document 1, a speaker uses a transmitter and a listener uses a receiver. The transmitter includes a microphone, a voice recognition circuit, a transmission unit, and the like, and the transmission unit transmits character information corresponding to the content recognized based on the voice recognition result to the receiver. The receiver includes a receiver, a CPU (Central Processing Unit), a display, and the like, and displays character on the display when character information is received from the transmitter.

JP-A-9-206329

  However, in the technique described in Patent Document 1, since it is assumed that a speaker or a listener uses a hearing aid device, when a plurality of sounds are mixed and input to a microphone, There was a problem that it was difficult to recognize speech.

  The present invention has been made in view of the above-described problems, and even when there are a plurality of speakers, the conversation support device capable of recognizing each voice and supporting the hearing, and the control of the conversation support device It is an object to provide a method and a program for a conversation support apparatus.

(1) In order to achieve the above object, a conversation support apparatus according to an aspect of the present invention recognizes a voice input unit that inputs voice signals of two or more users, and a voice signal input to the voice input unit. A speech recognition unit, a sound source estimation unit for estimating a sound source direction of a speech signal input to the speech input unit, a display unit on which a recognition result recognized by the speech recognition unit is displayed, and for each user When the corresponding display area is set as the image display area of the display unit and voice is detected from a direction different from that of the speaker already localized, it is determined that a new speaker is participating in the conference. And an image processing unit that displays the utterance content between the text display frames of the already recognized speakers.

(2) Moreover, in the conversation support apparatus according to an aspect of the present invention, the image processing unit includes a display color and a pattern of a display area corresponding to each user, an icon displayed in the display area, and the display area At least one of the avatars displayed on the screen may be displayed differently for each user.

(3) In the conversation support device according to the aspect of the present invention, the image processing unit corresponds to the user of the display unit an image based on the sound source direction estimated by the sound source estimation unit. You may make it display on the said display area.

(4) Moreover, the conversation assistance apparatus which concerns on 1 aspect of this invention is equipped with the sound source separation part which isolate | separates the audio | voice signal input into the said audio | voice input part for every said user, The said image processing part is the said sound source separation. Of the audio signals for each user separated by the unit, the recognition result other than the user corresponding to the display region may be displayed in the display region corresponding to the user of the display unit. Good.

(5) Moreover, the conversation assistance apparatus which concerns on 1 aspect of this invention is provided with the position estimation part which estimates the said user's position, The said image processing part is the said user's position estimated by the said position estimation part The display area corresponding to each user may be set or rearranged in the image display area of the display unit at a position corresponding to the user.

(6) Moreover, the conversation assistance apparatus which concerns on 1 aspect of this invention WHEREIN: The said position estimation part may estimate the said user's position using the audio | voice signal input into the said audio | voice input part.

(7) Moreover, the conversation assistance apparatus which concerns on 1 aspect of this invention is provided with the translation part which translates the recognition result recognized by the said speech recognition part, The said image process part is the translation result translated by the said translation part May be displayed in the display area corresponding to each user of the display unit.

(8) Moreover, the conversation assistance apparatus which concerns on 1 aspect of this invention is provided with the language information detection part which detects the language which the said user utters, and the said translation part other than the said user corresponding to the said display area. The recognition result may be translated into a language detected by the language information detection unit.

(9) Moreover, the conversation assistance apparatus which concerns on 1 aspect of this invention is provided with the communication part which communicates with another conversation assistance apparatus, The said voice input part is said other conversation assistance apparatus which the said communication part received The voice signal received from the voice input unit may be input, and the voice recognition unit may recognize a voice signal other than the user corresponding to the display area among the voice signals input from the voice input unit. .

(10) Moreover, the conversation assistance apparatus which concerns on 1 aspect of this invention is provided with the input part which selects a part of image displayed on the said display part, The said image process part is the image selected by the said input part. When a part of the recognition result is a recognition result, another recognition candidate corresponding to the selected recognition is displayed on the display unit, and the recognition result is corrected to the candidate selected by the input unit among the recognition candidates. The corrected recognition result may be transmitted to the other conversation support apparatus via the communication unit.

(11) In order to achieve the above object, a method for controlling a conversation support apparatus according to an aspect of the present invention includes: a voice input unit that inputs a voice signal of two or more users; A voice recognition procedure for recognizing a voice signal input by the voice input procedure, a sound source estimation unit for estimating a sound source direction of the voice signal input by the voice input procedure, and an image processing unit, The display area corresponding to each user is set as the image display area of the display unit on which the recognition result recognized by the voice recognition procedure is displayed, and the voice is heard from a direction different from that of the speaker already localized. If detected, an image processing procedure for determining that a new speaker is participating in the conference and displaying the utterance content between the text display frames of the adjacent already recognized speakers, Including.

(12) In order to achieve the above object, a program for a conversation support apparatus according to an aspect of the present invention includes a voice input procedure for inputting two or more user voice signals to a computer of the conversation support apparatus;
A speech recognition procedure for recognizing a speech signal input by the speech input procedure, a sound source estimation procedure for estimating a sound source direction of the speech signal input by the speech input procedure, and a display area corresponding to each user, When the recognition result recognized by the voice recognition procedure is set in the image display area of the display unit where the recognition result is displayed and the voice is detected from a direction different from the speaker already localized, the new speaker It is determined that the user is participating in the conference, and an image processing procedure for displaying the utterance content between the adjacent text recognition frames of the already recognized speakers is executed.

According to the configuration of (1), (11), or (12) described above, even if there are a plurality of speakers, it is possible to recognize each voice and support hearing. In addition, according to the configuration of (1), (11), or (12) described above, the user can visually determine which speaker has a new speaker added when the number of participants increases. Can know.
According to the configuration of the aspect (1), (11), or (12) of the present invention, it is easy to see the result recognized by the user, so that the convenience for the user can be improved.
According to the aspect (3) of the present invention, the user can easily determine his / her display area.
According to the aspect (4) of the present invention, it is possible to accurately estimate the direction for each speaker and to separate the utterances for each speaker. In addition, other speakers can visually confirm the other party's utterance on the conversation support device with high accuracy, and thus can support the hearing of the speaker.
According to the aspects (5) and (6) of the present invention, since the display position is arranged at the closest position of each speaker, the character data (recognition result) in which the utterance contents of other speakers are recognized is obtained. It is easier for the speaker to see.

According to aspects (7) and (8) of the present invention, the translation result translated by the translation unit is displayed in the display area corresponding to each user of the display unit (image display unit 15). Since the person can visually confirm the utterance of the other party on the conversation support device, it is possible to support the hearing of the speaker.
According to the aspect (9) of the present invention, voice recognition can be performed using a plurality of conversation support apparatuses.
According to the aspect (10) of the present invention, the user's utterance content can be correctly presented to other users.

It is a block diagram showing the structure of the conversation assistance apparatus which concerns on 1st Embodiment. It is a figure explaining the example in case the microphone which concerns on 1st Embodiment is integrated in the main body. It is a figure explaining the example in case the microphone which concerns on 1st Embodiment is integrated in the cover. It is a figure explaining the example in case the speaker which concerns on 1st Embodiment uses a close-talking microphone. It is a figure explaining an example of the menu image which concerns on 1st Embodiment. It is a figure explaining the example of the image of the screen pattern displayed on an image display part, when there are two speakers concerning 1st Embodiment. It is a figure explaining the example of the image of the screen pattern displayed on an image display part, when there are three speakers which concern on 1st Embodiment. It is a figure explaining the example of the image of the screen pattern displayed on an image display part, when there are four speakers concerning 1st Embodiment. It is a flowchart of the process sequence which the conversation assistance apparatus which concerns on 1st Embodiment performs. It is a figure for demonstrating experimental environment. It is an image displayed on the image display part before starting a conversation. This is an image displayed on the image display unit after the first speaker speaks “Good evening”. FIG. 13 is an image displayed on the image display unit after the second speaker utters “Good evening” after FIG. 12. This is an image displayed on the image display unit after the first speaker speaks four times and the second speaker speaks three times. It is a figure explaining the example of the image displayed on an image display part when there are three speakers. It is a block diagram showing the structure of the conversation assistance apparatus which concerns on 2nd Embodiment. It is a figure explaining the example of the combination of each part corresponding to the microphone array which concerns on 2nd Embodiment. It is a figure explaining an example of the sound source localization which concerns on 2nd Embodiment. It is a flowchart of the process sequence which the conversation assistance apparatus which concerns on 2nd Embodiment performs. It is a figure explaining a process when the number of speakers concerning a 2nd embodiment changes. It is a block diagram showing the structure of the conversation assistance apparatus which concerns on 3rd Embodiment. It is a figure explaining an example of arrangement | positioning of the some conversation assistance apparatus which concerns on 3rd Embodiment. It is a figure explaining an example of the image displayed on the image display part of each conversation assistance apparatus concerning 3rd Embodiment. It is a block diagram showing the structure of the conversation assistance apparatus which concerns on 3rd Embodiment. It is a block diagram showing the structure of the conversation assistance apparatus which concerns on 4th Embodiment. It is a figure explaining an example of the image displayed on the image display part of the conversation assistance apparatus which concerns on 4th Embodiment.

First, the outline of the present invention will be described.
In the present invention, of the utterances included in the audio signal collected by the microphone, information indicating the utterances uttered by the speaker is displayed in the display area of the other person. In the present invention, when there are a plurality of speakers, the display area of the display unit is divided into areas according to the number of speakers, each speaker is associated with each divided area, and the utterance is indicated in each associated area. Display information.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram showing the configuration of the conversation support apparatus 1 according to this embodiment. As shown in FIG. 1, the conversation support apparatus 1 includes a sound collection unit 11 (speech input unit), an acoustic signal acquisition unit 12 (speech input unit), a speech recognition unit 13 (speech recognition unit, position estimation unit), and image processing. Unit 14, image display unit 15 (display unit), and input unit 16. The image processing unit 14 includes an image pattern generation unit 141, a display image generation unit 142, and an image composition unit 143.
The conversation support device 1 is, for example, a tablet terminal, a mobile phone, a portable game machine, a terminal having an image display unit on the surface of a table, or the like. In the following embodiments, the conversation support device 1 will be described as a tablet type terminal (hereinafter also referred to as a tablet terminal).

  The input unit 16 is a touch panel sensor provided on the image display unit 15, and outputs coordinate information on the screen touched by the user to the image processing unit 14. Note that the input unit 16 may be an external input device by wired or wireless connection. Examples of the external input device include a keyboard and a mouse.

  The sound collection unit 11 records acoustic signals of N channels (N is an integer greater than 1, for example, 8), and transmits the recorded N channel acoustic signals to the acoustic signal acquisition unit 12. The sound collection unit 11 includes, for example, N microphones 101-1 to 101-N that receive sound waves having components in a frequency band (for example, 200 Hz to 4 kHz). The sound collection unit 11 may transmit the recorded N-channel acoustic signals wirelessly or by wire. When N is larger than 1, it is only necessary that the acoustic signals are synchronized between the channels at the time of transmission. In the following description, when the microphones 101-1 to 101-N are not specified, they are simply referred to as the microphone 101. The microphone 101 of the sound collection unit 11 may be incorporated in the conversation support device 1 or attached to the conversation support device 1 as described later, or a close-talking microphone used by a speaker. It may be.

  The acoustic signal acquisition unit 12 acquires N acoustic signals recorded by the N microphones 101 of the sound collection unit 11. The acoustic signal acquisition unit 12 generates an input signal in the frequency domain by performing Fourier transform on the acquired N acoustic signals for each frame in the time domain. The acoustic signal acquisition unit 12 outputs the N acoustic signals subjected to Fourier transform to the speech recognition unit 13. Note that the N acoustic signals may include information that can identify the microphones 101-1 to 101-N or information indicating the direction in which the microphones 101-1 to 101-N are attached. Note that the orientations of the microphones 101-1 to 101 -N are acquired by an orientation sensor (not shown) provided in the conversation support device 1, and the positional relationship between information based on the acquired orientation and the microphone 101 incorporated in the conversation support device 1. Based on the above, information indicating the direction in which the microphones 101-1 to 101-N are attached may be estimated.

The speech recognition unit 13 performs speech recognition processing on the acoustic signal input from the acoustic signal acquisition unit 12 to recognize the utterance content (for example, text indicating a word or sentence). In the case of an acoustic signal from a plurality of speakers, the voice recognition unit 13 determines a speaker and recognizes the utterance content for each determined speaker. For example, when the voice recognition unit 13 has the direction of the microphone 101 that has acquired the acoustic signal having the highest signal level among the N-channel acoustic signals input from the acoustic signal acquisition unit 12 as the speaker direction, for example, You may make it estimate by the MUSIC (Multiple Signal Classification) method. Then, the speech recognition unit 13 outputs information indicating the speaker, information indicating the direction of the speaker, and recognition data to the image processing unit 14.
The speech recognition unit 13 includes, for example, a hidden Markov model (HMM: Hidden Markov Model) that is an acoustic model and a word dictionary.

  For example, the voice recognition unit 13 calculates an acoustic feature amount at predetermined time intervals (for example, 10 ms) for the acoustic signal. The acoustic features are, for example, a set of 34th-order Mel-Frequency Cepstrum Coefficients (Static MSLS: Mel-Scale Log Spectrum), Delta MSLS, and one delta power. A feature vector, a static Mel-scale log spectrum (MSLS), a delta MSLS, a set of one delta power, and the like. The speech recognition unit 13 determines phonemes from the calculated acoustic feature quantities using an acoustic model, and recognizes words using a word dictionary from a phoneme string composed of the determined phonemes.

  In addition, since the conversation assistance apparatus 1 of this embodiment is an apparatus which assists auditory sense, it is difficult for the speaker who is a user to hear the other party's utterance. Therefore, for example, when there are two speakers and the first speaker Sp1 speaks, first, character data based on the speech of the first speaker Sp1 is displayed on the image display unit 15. Next, the second speaker Sp2 utters a response to the character data displayed on the image display unit 15. Thereafter, the first speaker Sp1 utters a response to the character data displayed on the image display unit 15. As described above, when two speakers do not speak at the same time, as shown in FIG. 1, the sound localization process and the sound separation process are not performed on the sound signal input from the sound signal acquisition unit 12. In addition, it is possible to determine which speaker is speaking from the feature value of the acoustic signal and the acoustic signal of the microphone 101 having the largest signal level, and to recognize the utterance content.

  The image pattern generation unit 141 generates a menu image to be described later based on the coordinate information on the screen input from the input unit 16, and outputs the generated menu image to the image composition unit 143. The image pattern generation unit 141 generates a screen pattern image according to the content selected by the user on the menu screen based on the coordinate information on the screen input from the input unit 16, and generates the generated screen pattern image. The image is output to the image composition unit 143. The screen pattern image is a display image corresponding to the number of speakers, as will be described later.

  The display image generation unit 142 generates character data corresponding to the recognition data for each speaker input from the speech recognition unit 13, and outputs the generated character data for each speaker to the image composition unit 143. The display image generation unit 142 generates an image indicating the direction of each speaker and the direction of each speaker based on the information indicating the direction of the speaker input from the voice recognition unit 13, and generates the image for each generated speaker. An image indicating the direction is output to the image composition unit 143.

The image composition unit 143 displays the menu image generated by the image pattern generation unit 141 on the image display unit 15. In the display image generated by the image pattern generation unit 141, the image synthesis unit 143 synthesizes the image so that the character data for each speaker input from the display image generation unit 142 is displayed in a display area other than the speaker who spoke. To do. In addition, the image composition unit 143 synthesizes the images so as to display the image indicating the direction of each speaker input from the display image generation unit 142 in the display area of each speaker. The image composition unit 143 displays the synthesized image on the image display unit 15.
Note that the image synthesis unit 143 may synthesize an image so that character data for each speaker is displayed in the display area of the speaker who has spoken in the display image.

  The image display unit 15 displays the image output by the image composition unit 143. The image display unit 15 is, for example, a liquid crystal display device, an organic EL (electroluminescence) display device, an electronic ink display device, or the like.

  Next, the microphone 101 of the sound collection unit 11 will be described with reference to FIGS. FIG. 2 is a diagram illustrating an example in which the microphone 101 according to the present embodiment is incorporated in the main body 201. FIG. 3 is a diagram illustrating an example in which the microphone 101 according to the present embodiment is incorporated in the cover 202. FIG. 4 is a diagram for explaining an example when a speaker according to the present embodiment uses a close-talking microphone. The number and arrangement of the microphones 101 shown in FIGS. 2 to 4 are examples, and the present invention is not limited to this. 2 to 4, an image denoted by reference numeral 201 is an image showing the main body of the conversation support apparatus 1. The shape of the main body 201 is not limited to a vertically long rectangle, and may be a square, a horizontally long rectangle, a circle, an ellipse, or a polygon.

  In the example shown in FIG. 2, eight microphones 101 are incorporated in the periphery of the main body 201. Among the eight microphones 101, the microphones 101-1 to 101-4 are attached along the right long side toward the paper surface, and the remaining microphones 101-5 to 101-8 are the left long toward the paper surface. It is attached along the side.

  In the example shown in FIG. 3, eight microphones 101 are incorporated in the periphery of a cover 202 that can be removed from the main body 201. The connection between each microphone 101 and the main body 201 may be wired or wireless. As in FIG. 2, microphones 101-1 to 101-4 are mounted along the right long side facing the paper surface, and the remaining microphones 101-5 to 101-8 are the left long side facing the paper surface. Attached along. Although the example of incorporating the microphone 101 into the cover 202 is connected, as another example, the microphone 101 may be incorporated into a bumper or the like that protects the main body 201.

  In the example illustrated in FIG. 4, four speakers use close-talking microphones 101-1 to 101-4, respectively. The connection between each microphone 101 and the main body 201 may be wired or wireless. Further, the position of the microphone 101 is a position corresponding to the speaker, and when there are two speakers, it is desirable that the speakers be arranged on the left and right or top and bottom of the main body 201 toward the page. In addition, when there are four speakers, it is desirable that the speakers be arranged in the upper right, lower right, lower left, and upper left of the main body 201 as shown in FIG.

Next, a menu screen selected by the input unit 16 will be described. FIG. 5 is a diagram illustrating an example of the menu image 301 according to the present embodiment.
In FIG. 5, the menu image 301 is displayed on the image display unit 15 when the user selects or switches the display screen. The menu image 301 includes a selection menu area 311 for a speaker, selection menu areas 312 to 315 for languages in which the first speaker (first speaker) to the fourth speaker (fourth speaker) speak, and a screen. Rotation selection menu area 316. Note that the menu image 301 shown in FIG. 5 is an example, and all the menu images may be displayed as one menu as shown in FIG. 5, or the items may be divided into a plurality of menu images for each item. It may be.

  The speaker selection menu area 311 is selected by any one of the speakers by operating the input unit 16 in accordance with the number of speakers having a conversation using the conversation support device 1. In the speaker selection menu area 311, for example, “Two speakers (green)” indicates that there are two speakers and the display color corresponding to the second speaker is green. In the example shown in FIG. 5, an example is shown in which the display color corresponding to the first speaker to the fourth speaker is fixed, but the display color is not the same for a plurality of speakers. You may make it selectable in the visually distinguishable range. In this case, for example, the first speaker may change the display color by operating “one speaker (red)” with the input unit 16. For example, after red is selected by the first speaker, the image pattern generation unit 141 selects a color that is visually identifiable even when adjacent to the red, and a selection menu for speakers corresponding to other speakers. You may make it display on the area | region 311. FIG.

The language selection menu area 312 for the first speaker speaks a menu image for selecting the language used by the first speaker by operating the input unit 16. Similarly, in the language selection menu areas 313 to 315 for the second to fourth speakers, the language used by the second to fourth speakers is selected by operating the input unit 16. It is a menu image. In the example shown in FIG. 5, as an example of the language used by the speaker, an example of selecting from four languages of Japanese, English, French, and Chinese is shown, but the number of languages is not limited to this. . Further, in the example shown in FIG. 5, a selection menu area 311 for a speaker, a selection menu area 312 for languages in which the first speaker (first speaker) to the fourth speaker (fourth speaker) speak, respectively. Although an example in which all 315 is displayed in Japanese is shown, the present invention is not limited to this. For example, “speaker first: language Japanese” is displayed in Japanese in the language selection menu area 312 spoken by the first speaker, and “speaker first: language Japanese” is displayed.
“English” may be displayed in English, “Speaker First: Language French” may be displayed in French, and “Speaker First: Language Chinese” may be displayed in Chinese.

The screen rotation selection menu area 316 includes an instruction to fix the image displayed on the screen so as not to rotate, an instruction to rotate the screen orientation by 90 degrees, and an instruction to reverse the screen orientation (rotate 180 degrees). The instruction to fix the image displayed on the screen so as not to rotate has, for example, a function of rotating the screen direction according to a detection output of a sensor (not shown) included in the conversation support device 1 that detects rotation of the main body. This is an instruction to fix the display screen so that it does not rotate even if it is. Further, an instruction to rotate the screen orientation by 90 degrees and an instruction to reverse the screen orientation (rotate by 180 degrees) indicate that the image displayed on the image display unit 15 by the speaker depends on the number and arrangement of the speakers. This is an instruction to rotate or reverse the display direction of the image so that it can be seen most easily.
For example, when a projector (not shown) is connected to the conversation support apparatus 1, an image displayed on the image display unit 15 is displayed on the screen via the projector. In this case, if the conversation support apparatus 1 is rotated, the image displayed on the screen is also rotated, which may make it difficult for the speaker to understand the display area to be viewed by the speaker. In order to prevent this, the image displayed on the screen is fixed so as not to rotate.

Next, an example of a screen pattern image generated by the image pattern generation unit 141 will be described with reference to FIGS. FIG. 6 is a diagram illustrating an example of an image of a screen pattern displayed on the image display unit 15 when there are two speakers according to the present embodiment. FIG. 7 is a diagram illustrating an example of an image of a screen pattern displayed on the image display unit 15 when there are three speakers according to the present embodiment.
FIG. 8 is a diagram for explaining an example of an image of a screen pattern displayed on the image display unit 15 when there are four speakers according to the present embodiment. The arrangements and color schemes shown in FIGS. 6 to 8 are examples, and the present invention is not limited to these. 6 to 8, images 301 </ b> A to 301 </ b> C are images of screen patterns displayed on the image display unit 15. The color corresponding to each speaker is red for the first speaker, green for the second speaker, blue for the third speaker, and yellow for the fourth speaker as in the menu image shown in FIG. is there.
In addition, although the example shown below showed the example which divided the display area corresponding to each speaker with a different color, it is not restricted to this. The display area corresponding to each speaker may be distinguished by, for example, a different pattern, an icon corresponding to each speaker, an avatar that is an anthropomorphic image corresponding to each speaker, or the like. In this case, since the speaker can identify the display region corresponding to the speaker without identifying the region or the like by color, a monochrome image display device or an electronic ink display device may be used for the image display unit 15. And power consumption can be reduced.

As shown in FIG. 6, when there are two speakers, the image 301 </ b> A is divided up and down toward the page, and for example, a display area for information to be presented to the first speaker is assigned to the upper area. A display area for information to be presented to the second speaker is assigned to this area.
In the first display image 321A of the information display area presented to the first speaker, as described later, the first character presentation image 322A of the character display area in which the text of the utterance content spoken by the second speaker is displayed. It has. The color of the first character presentation image 322A is, for example, white. Further, the first character presentation image 322A includes an orientation image 323A indicating the direction of the first speaker as will be described later. The example shown in FIG. 6 is an example in which the first speaker is in front of the upper side with respect to the page.
In the second presentation image 331A in the display area of information presented to the second speaker, the second character presentation image 332A in the character display area in which the text of the utterance content spoken by the first speaker is displayed, and the second An orientation image 333A indicating the direction of the speaker is included. The example shown in FIG. 6 is an example in which the second speaker is at the lower right with respect to the page. The color of the second character presentation image 332A is, for example, white.

As shown in FIG. 7, when there are three speakers, the image 301B is divided into three parts. For example, a display area for information presented to the first speaker is assigned to the upper left area, and the speaker is assigned to the lower left area. A display area for information presented to the second person is assigned, and a display area for information presented to the third person is assigned to the right area.
In addition, in each of the first presentation image 321B to the third presentation image 341B in the display area corresponding to the first speaker to the third speaker, a character display in which texts of other uttered utterance contents not including itself are displayed. The first character presentation image 322B to the third character presentation image 342B of the area and the orientation images 323B to 324B indicating the direction of the user are included. The color of the first character presentation image 322B to the third character presentation image 342B is, for example, white.
As an example, in the third character presentation image 342B of the third presentation image 341B in the display area of the information presented to the third speaker, the text of the utterance content uttered by the first speaker and the second speaker is displayed. The The orientation image 343B represents the direction of the third speaker.

As shown in FIG. 8, when there are four speakers, the image 301C is divided into four parts. For example, a display area for information presented to the first speaker is assigned to the upper left area, and the speaker is assigned to the lower left area. A display area for information to be presented to the second person is allocated. In addition, a display area for information presented to the third speaker is assigned to the lower right area, and a display area for information presented to the fourth speaker is assigned to the upper right area.
In addition, in each of the first presentation image 321C to the fourth presentation image 351C in the display area corresponding to the first speaker to the fourth speaker, a character display in which text of other uttered utterance contents not including itself is displayed. The first character presentation image 322C to the fourth character presentation image 352C of the region and the orientation images 323C to 353C indicating their orientation are included. The color of the first character presentation image 322C to the fourth character presentation image 352C is, for example, white.

As an example, in the fourth character presentation image 352C of the fourth presentation image 351C in the display area of information presented to the fourth speaker, the text of the utterance content uttered by the first to third speakers is displayed. The The orientation image 353C represents the direction of the fourth speaker.
For example, in FIG. 8, each speaker may input the initial speaker direction by operating the touch panel type input unit 16 provided in the image display unit 15. In this case, the conversation support apparatus 1 may continue to display the text of the utterance content on the input report corresponding to the initial speaker direction.

6 to 8, for example, the image pattern generation unit 141 has the same size of the image in the character display area in which texts of other uttered utterances not including itself are displayed. Each display area is divided so that Alternatively, the image pattern generation unit 141 detects that the device is tilted on a table or the like based on the detection result of a tilt detection sensor (not shown) provided in the conversation support device 1 and detects the detected tilt. The ratio of the image size of the character display area is calculated according to the angle of the image, and the size of each area to be divided is determined according to the size of the image of the character display area based on the calculated ratio. Good.
Note that which region corresponds to which speaker among the regions divided into 2 to 4 is stored in the image pattern generation unit 141 in advance. The image pattern generation unit 141 may switch which region corresponds to which speaker in accordance with an instruction input from the input unit 16. For example, in FIG. 8, when the positions of the second speaker and the fourth speaker are switched, for example, the second speaker displays the second presentation image 331 </ b> C as the fourth display image in the touch panel type input unit 16. You may make it replace the 2nd presentation image 331C and the 4th presentation image 351C by operating so that it may move to the area | region of the presentation image 351C. Thereby, even if the position of the speaker is changed in the middle, according to the present embodiment, it is possible to view the screen while maintaining the display of the conversation content up to that point, so that it is convenient for the speaker. Will improve.

Next, a processing procedure performed by the conversation support apparatus 1 will be described. FIG. 9 is a flowchart of a processing procedure performed by the conversation support apparatus 1 according to the present embodiment.
(Step S1) The speaker selects the number of users by operating the input unit 16 in the menu image 301 shown in FIG. Next, each speaker selects the language to be used by operating the input unit 16 in the menu image 301 shown in FIG. Next, the input unit 16 outputs the coordinate information on the screen selected by the speaker to the image processing unit 14.

(Step S2) The image pattern generation unit 141 generates an image of the screen pattern according to the content selected by the user on the menu screen based on the coordinate information on the screen input from the input unit 16, and the generated screen The pattern image is output to the image composition unit 143. Next, the image composition unit 143 displays the menu image generated by the image pattern generation unit 141 on the image display unit 15.
(Step S <b> 3) The acoustic signal acquisition unit 12 detects, for example, that the start of recognition is instructed by the input unit 16, or at the timing when Step S <b> 1 is performed, and the N microphones 101 of the sound collection unit 11. The acquisition of N acoustic signals recorded by is started. Next, the acoustic signal acquisition unit 12 outputs the N acoustic signals subjected to Fourier transform to the speech recognition unit 13.

(Step S4) The speech recognition unit 13 performs speech recognition processing for each speaker on the acoustic signal input from the acoustic signal acquisition unit 12 to recognize the utterance content. Next, the voice recognizing unit 13 estimates the direction of the speaker for each speaker based on, for example, the direction of the microphone 101 that has acquired the acoustic signal having the highest signal level when the speaker speaks. Next, the voice recognition unit 13 outputs information indicating the speaker, information indicating the direction of the speaker, and recognition data to the image processing unit 14.

(Step S <b> 5) The display image generation unit 142 generates character data corresponding to the recognition data for each speaker input from the speech recognition unit 13, and outputs the generated character data for each speaker to the image display unit 15. . The display image generation unit 142 generates an image of information indicating the direction of each speaker and the direction of each speaker based on the information indicating the direction of the speaker input from the voice recognition unit 13, and the generated speaker An image of information indicating each direction is output to the image composition unit 143.

(Step S6) In the display image generated by the image pattern generation unit 141, the image composition unit 143 displays the character data for each speaker input from the display image generation unit 142 in a display area other than the speaker who has spoken. Composite the image with Next, the image synthesizing unit 143 synthesizes the images so that the image of the information indicating the direction of each speaker input from the display image generating unit 142 is displayed in the display area of each speaker. Next, the image composition unit 143 displays the synthesized image on the image display unit 15.
Above, the process which the conversation assistance apparatus 1 performs is complete | finished.

<Explanation of experimental results>
Here, an example of an experimental result performed using the conversation support apparatus 1 according to the present embodiment will be described. FIG. 10 is a diagram for explaining the experimental environment.
As shown in FIG. 10, the conversation support apparatus 1 is placed on a table 401 at an angle.
In addition, the conversation support device 1 is placed so that one longitudinal direction is in contact with the table 401. The experiment was performed in a room having a predetermined area. Further, there are two speakers, a first speaker Sp1 and a second speaker Sp2, and the first speaker Sp1 and the second speaker Sp2 are seated on the chair 402.

  FIG. 11 shows an image 501 displayed on the image display unit 15 before the conversation is started. In the image 501, the first presentation image 521 in the upward direction toward the paper surface is a region presented to the first speaker Sp1, and the second presentation image 531 in the downward direction toward the paper surface is presented to the second speaker Sp2. It is an area to be done. On the image display unit 15 of the conversation support apparatus 1 in FIG. 10, the first presentation image 521 is displayed on the left side as viewed from the first speaker Sp1 and the second speaker Sp2, and the second presentation image 531 is displayed on the right side. The The figure shown in FIG. 11 is an image displayed on the image display unit 15 after two speakers are selected as the number of speakers by the first speaker Sp1 or the second speaker Sp2 (step S2). The first presentation image 521 includes a first character presentation image 522, and the second presentation image 531 includes a second character presentation image 532.

  FIG. 12 shows an image displayed on the image display unit 15 after the first speaker Sp1 utters “Good evening”. As shown in FIG. 12, in the second character presentation image 532, “Good evening”, which is an image 534 </ b> A indicating a character that recognizes the utterance of the first speaker Sp <b> 1, is displayed by the image processing unit 14. At this time, since the second speaker Sp2 has not yet spoken, the direction of the second speaker Sp2 is unknown. Therefore, as shown in FIG. 12, “Good evening” which is the image 534A is displayed by the image processing unit 14 in the initial direction. In addition, in the first character presentation image 522, the orientation image 523 indicating the direction of the first speaker Sp1 is displayed by the image processing unit 14. In the orientation image 523, the direction of the arrowhead is the direction of the first speaker Sp1.

FIG. 13 is an image displayed on the image display unit 15 after the second speaker Sp2 utters “Good evening” after FIG. At this time, since the second speaker Sp2 uttered, the voice recognition unit 13 estimates the direction of the second speaker Sp2. Then, in the second character presentation image 532, the azimuth image 533 indicating the direction of the second speaker Sp2 is displayed by the image processing unit 14. As a result, the image 534A displayed on the second character presentation image 532 is displayed by being rotated by the display image generation unit 142 according to the direction of the second speaker Sp2.
Furthermore, in the first character presentation image 522, “Good evening”, which is an image 524A indicating a character that recognizes the utterance of the second speaker Sp2, is displayed by the image processing unit 14 in a direction according to the direction of the first speaker Sp1. Is displayed.

FIG. 14 is an image displayed on the image display unit 15 after the first speaker Sp1 speaks four times and the second speaker Sp2 speaks three times.
In the first character presentation image 522, character images 524A to 524C in which the utterance of the second speaker Sp2 is recognized are displayed. Then, as shown in FIG. 14, the images 524A to 524C are sequentially displayed for the first speaker Sp1 from the back of the image display unit 15 toward the front. In the second character presentation image 532, character images 534A to 534D that recognize the utterance of the first speaker Sp1 are displayed. As shown in FIG. 14, the images 534A to 534D are sequentially displayed for the second speaker Sp2 from the back of the image display unit 15 toward the front. In FIG. 14, the utterance order is, for example, the order of image 534A-> image 524A-> image 534B-> image 524B-> image 534C-> image 524C-> image 534D.

  For example, the display image generation unit 142 determines whether or not the first character presentation image 522 is filled with an image corresponding to the recognized character, and the first character presentation image 522 corresponds to the recognized character. When it is determined that the image is filled with an image, the image corresponding to the old speech may be deleted, or the image may be scrolled. Then, when the first speaker Sp1 wants to see an image corresponding to the erased utterance, the image of the character displayed in the past of the first character presentation image 522 on the image display unit 15 is called up. By operating the input unit 16 of the touch panel provided on the display unit 15, the image processing unit 14 may perform processing so that past utterances can also be referred to.

In addition, in FIGS. 12-14, although the example with which the magnitude | size of the image displayed on the 1st character presentation image 522 is all equal was shown, it is not restricted to this. For example, an image obtained as a result of recognizing the latest utterance may be displayed large in the center of the first character presentation image 522, for example, and an image based on the past utterance may be displayed small. The same applies to the image displayed on the second character presentation image 532.
In FIGS. 12 to 14, the display image generation unit 142 may determine the character size so that the characters corresponding to each conversation fit in one line. Alternatively, the display image generation unit 142 may display an image corresponding to a character recognized with a predetermined character size over several lines. In this case, the voice recognition unit 13 may include the information indicating the phrase in the recognized recognition data and output it to the image processing unit 14. As a result, when the display image generation unit 142 determines that the recognized sentence does not fit in one line with a predetermined character size, the display image generation unit 142 uses the information indicating the phrase input from the speech recognition unit 13 in the sentence. You may make it wrap in the position of the break of a clause.
Further, for example, in the example shown in FIG. 14, the images 524A to 524C corresponding to the recognized characters may be displayed in a color corresponding to the second speaker Sp2. Similarly, the images 534A to 534D corresponding to the recognized characters may be displayed in a color corresponding to the first speaker Sp1.

Next, an example of an image displayed on the image display unit 15 when there are three speakers will be described. FIG. 15 is a diagram illustrating an example of an image 601 displayed on the image display unit 15 when there are three speakers.
The image 601 corresponds to the image 301B shown in FIG. 7, and the reference numerals 621, 622, 623, 631, 632, 633, 641, 642, and 643 are the reference numerals 321B, 322B, 323B, 331B, This corresponds to 332B, 333B, 341B, 342B, and 343B.

  In the example shown in FIG. 15, the first speaker utters "Hello". As a result, an image 634A corresponding to the character recognized in the second character presentation image 632 is displayed, and an image 644A corresponding to the character recognized in the third character presentation image 642 is displayed. The images 634A and 644A may be displayed in a color corresponding to the first speaker, for example, red. Alternatively, the display image generation unit 142 may add information indicating that the first speaker speaks to the image 634A and the image 644A. The information indicating that the first speaker speaks is a name, an avatar corresponding to the first speaker, an icon corresponding to the first speaker, a color mark corresponding to the first speaker (for example, a red circle), and the like. is there. Accordingly, it is possible to assist in visually recognizing which utterer has recognized the utterance result.

  Next, the second speaker speaks “Hey!”. Thereby, an image 625B corresponding to the recognized character in the first character presentation image 622 is displayed, and an image 645B corresponding to the recognized character in the third character presentation image 642 is displayed. Also in this case, the image 625B and the image 645B may be displayed in a color corresponding to the second speaker, for example, green.

  As described above, in the conversation support device 1 according to the present embodiment, the voice input unit (sound collection unit 11 and acoustic signal acquisition unit 12) that inputs two or more user's voice signals and the voice input unit are input. A voice recognition unit 13 for recognizing a voice signal, a display unit (image display unit 15) for displaying a recognition result recognized by the voice recognition unit, and a display area (image display unit) corresponding to each user. 15) image processing unit 14 set in the image display area (321A, 322A, 331A, 332A, 321B, 322B, 331B, 332B, 341B, 342B, 321C, 322C, 331C, 332C, 341C, 342C, 351C, 352C). And comprising.

  With this configuration, the conversation support apparatus 1 according to the present embodiment can support hearing by recognizing each voice even when there are a plurality of speakers. In addition, since the sound localization process and the sound separation process are not performed on the sound signal acquired by the sound signal acquisition unit 12, the amount of calculation of the device can be reduced, and the functional unit of the device can be reduced.

  In addition, the conversation support apparatus 1 according to the present embodiment includes a sound source estimation unit (acoustic signal acquisition unit 12) that estimates a user's sound source direction, and the image processing unit 14 has a sound source direction estimated by the sound source estimation unit. The recognition result recognized by the voice recognition unit is displayed in the display area corresponding to each user of the image display unit 15 at the display angle based on the display angle.

  With this configuration, the conversation support device 1 according to the present embodiment can display character data at an angle corresponding to the direction of the speaker as shown in FIGS. As a result, in the conversation support device 1 of the present embodiment, the result of the user's recognition can be easily seen, so that the convenience for the user can be improved.

  Further, in the conversation support device 1 according to the present embodiment, the image processing unit 14 supports an image based on the sound source direction estimated by the sound source estimation unit (acoustic signal acquisition unit 12) for each user of the image display unit 15. Display in the display area.

  With this configuration, the conversation support device 1 according to the present embodiment can display images 523, 533, 623, 633, and 643 indicating the direction of the speaker on the image display unit 15 as shown in FIGS. Therefore, the user can easily determine his / her display area.

  In the present embodiment, for example, when there are four speakers, the display area corresponding to the number of persons selected from the menu is displayed as shown in FIG. 8, but the present invention is not limited to this. For example, the conversation support apparatus 1 registers the voices of four speakers (first speaker Sp1 to fourth speaker Sp4) before the conversation starts. When the four speakers are each in a predetermined position, the speech recognition unit 13 of the conversation support device 1 uses the spoken voice when the four speakers sequentially speak. Is estimated. Then, the image processing unit 14 may determine or rearrange each image display position based on the position of each speaker estimated by the voice recognition unit 13.

For example, it is assumed that four speakers (first speaker Sp1 to fourth speaker Sp4) are at positions as shown in FIG. Here, it is assumed that each of the first speaker Sp1 to the fourth speaker Sp4 uses the microphones 101-1 to 101-4.
The voice recognition unit 13 performs voice recognition of each uttered speaker, arranges the display area of the first speaker Sp1 at the upper right of the fourth presentation image 351C in FIG. 8, and displays the display area of the second speaker Sp2. 8 is arranged at the lower right of the third presentation image 341C, the display area of the third speaker Sp3 is arranged at the lower left of the second presentation image 331C in FIG. 8, and the display area of the fourth speaker Sp4 is shown in FIG. Are arranged at the upper left of the first presentation image 321C. In this way, when a plurality of speakers do not speak at the same time and there is little noise in the environment where the conversation support apparatus 1 is used, sound source localization processing and sound source separation processing are performed as in the conversation support apparatus 1 of the present embodiment. Even if it does not perform, the process mentioned above can be performed.

As described above, the conversation support apparatus 1 according to the present embodiment includes the position estimation unit (voice recognition unit 13) that estimates the position of the user, and the image processing unit 14 uses the user estimated by the position estimation unit. The display area corresponding to each user is set or rearranged in the image display area of the display unit at a position corresponding to the position.
With this configuration, in the conversation support apparatus 1 of the present embodiment, the display position is arranged at the position closest to each speaker, so that character data (recognition result) in which the utterance contents of other speakers are recognized is spoken. Easier to see.

[Second Embodiment]
FIG. 16 is a block diagram showing the configuration of the conversation support apparatus 1A according to the present embodiment. As shown in FIG. 16, the conversation support apparatus 1A includes a sound collection unit 11, an acoustic signal acquisition unit 12, a speech recognition unit 13A, an image processing unit 14, an image display unit 15, an input unit 16, a sound source localization unit 21 (sound source estimation A sound source separation unit 22, a language information detection unit 23, and a translation unit 24. The image processing unit 14 includes an image pattern generation unit 141, a display image generation unit 142, and an image composition unit 143.
In addition, the same code | symbol is used for the function part which has the same function as the conversation assistance apparatus 1 demonstrated in FIG. 1, and description is abbreviate | omitted.

The sound source localization unit 21 estimates the azimuth angle of the sound source based on the input signal input from the acoustic signal acquisition unit 12, and sends the azimuth angle information indicating the estimated azimuth angle and the N-channel acoustic signal to the sound source separation unit 22. Output. The azimuth angle estimated by the sound source localization unit 21 is based on, for example, the direction from the center of gravity of the positions of the N microphones included in the sound collection unit 11 to one predetermined microphone among the N microphones. Direction in a horizontal plane. For example, the sound source localization unit 21 estimates an azimuth angle using a GSVD-MUSIC (Generalized Single Value Decomposition-Multiple Signal Classification; MUSIC using generalized singular value expansion) method.
For estimation of the azimuth angle, other sound source direction estimation methods such as a WDS-BF (Weighted Delay and Sum Beam Forming) method and a MUSIC method may be used.

The sound source separation unit 22 acquires the N-channel acoustic signal output from the sound source localization unit 21, and uses the acquired N-channel acoustic signal as a speaker using, for example, a GHDSS (Geometric High-order Decorrelation-based Source Separation) method. Separate into each acoustic signal. The GHDSS method will be described later. Alternatively, the sound source separation unit 22 may perform sound source separation processing using, for example, an independent component analysis (ICA) method. The sound source separation unit 22 outputs the separated acoustic signal for each speaker and the azimuth angle information input from the sound source localization unit 21 to the language information detection unit 23.
The sound source separation unit 22 may separate the acoustic signal for each speaker after separating the noise and the acoustic signal of the speaker using, for example, an indoor transfer function stored in the own unit. Good. The sound source separation unit 22 calculates an acoustic feature amount for each N-channel acoustic signal, for example, and separates into an acoustic signal for each speaker based on the calculated acoustic feature amount and azimuth angle information input from the sound source localization unit 21. You may make it do.

  The language information detection unit 23 detects the language for each speaker by a well-known method for each acoustic signal for each speaker input from the sound source separation unit 22. The language information detection unit 23 outputs the information indicating the detected language for each speaker, the acoustic signal for each speaker input from the sound source separation unit 22, and the azimuth information to the speech recognition unit 13A. The language information detection unit 23 refers to a language database, for example, and detects a language for each speaker based on the result of the reference. The language database may be provided in the conversation support apparatus 1A, or may be connected via a wired or wireless network.

  The voice recognizing unit 13 </ b> A applies the acoustic signal input from the acoustic signal acquiring unit 12 based on the information indicating the language for each speaker input from the language information detecting unit 23, the acoustic signal for each speaker, and the azimuth angle information. Speech recognition processing is performed for the speech content (for example, text indicating a word or sentence). The voice recognition unit 13A outputs the utterance content, the information indicating the speaker, the information indicating the speaker direction and the recognition data, and the information indicating the language for each speaker to the translation unit 24.

  The translation unit 24 translates the utterance content as necessary based on the utterance content input from the speech recognition unit 13A, information indicating the speaker, and information indicating the language of each speaker, and indicates the translated utterance content. The information is output to the image processing unit 14 in addition to or in place of the information input from the voice recognition unit 13A. Specifically, there are two speakers, the first speaker Sp1 and the second speaker Sp, the language used by the first speaker Sp1 is Japanese, and the language used by the second speaker Sp2 is English. This will be described with reference to FIG. In this case, the images 534A to 534D displayed on the second character presentation image 532 are displayed after being translated from Japanese spoken by the first speaker Sp1 into English which is the language used by the second speaker Sp2. The translation unit 24 translates the utterance content. Further, the images 524A to 524C displayed on the first character presentation image 522 are translated and displayed from English spoken by the second speaker Sp2 into Japanese which is the language used by the first speaker Sp1. The translation unit 24 translates the utterance content.

<GHDSS method>
Here, an outline of the GHDSS method used in the sound source separation unit 22 will be described. The GHDSS method is a technique in which a GC (sound source separation based on geometric constraint) method and a HDSS (High-order Dicorrelation-based Source Separation) method are integrated. The GHDSS method is a kind of blind deconvolution. In the GHDSS method, a separation matrix [V (ω)] is sequentially calculated, and an input speech vector [x (ω)] is multiplied by the calculated separation matrix [V (ω)] to obtain a sound source vector [u (Ω)] is a method for separating the sound signal into sound signals for each sound source. The separation matrix [V (ω)] is a pseudo-inverse matrix of a transfer function [H (ω)] having a transfer function as an element from each sound source to each microphone 101 included in the sound collection unit 11. . The input speech vector [x (ω)] is a vector having as an element the frequency domain coefficient of the acoustic signal of each channel. The sound source vector [u (ω)] is a vector having as an element the frequency domain coefficient of the acoustic signal emitted from each sound source.

In the GHDSS method, when calculating the separation matrix [V (ω)], the sound source vector [[ u (ω)] is estimated.
Here, the separation sharpness JSS is an index value representing the degree to which one sound source is erroneously separated as another sound source, and is represented by the following equation (1), for example.

In Expression (2), || ... || ² represents a Frobenius norm. * Indicates a conjugate transpose of a vector or matrix. Further, diag (...) indicates a diagonal matrix composed of ... diagonal elements.
The geometric constraint degree JGC is an index value representing the degree of error of the sound source vector [u (ω)], and is represented by the following equation (2), for example.

  In equation (2), [I] represents a unit matrix.

  When the microphone array is configured by the microphones 101-1 to 101-N included in the sound collection unit 11, the speaker inputs information indicating that he / she speaks to the conversation support apparatus 1A when he / she speaks. Or it may not be selected. In this case, the conversation support apparatus 1A can be separated into utterances for each speaker using a microphone array.

Next, an example of a combination using each unit illustrated in FIG. 16 will be described. FIG. 17 is a diagram for explaining an example of a combination of units corresponding to the microphone array according to the present embodiment.
In FIG. 17, the microphone array 1 is a microphone array in which an array of microphones 101 is incorporated in the conversation support apparatus 1A as shown in FIG. The microphone array 2 is a microphone array in which the microphone 101 is connected to the conversation support apparatus 1A by wire or wireless as shown in FIG. As shown in FIG. 4, the microphone array 3 is a microphone array in which each speaker uses, for example, a close-talking microphone 101 at the mouth, and the microphone 101 is connected to the conversation support apparatus 1A by wire or wirelessly. .

  As shown in the first line of FIG. 17, in the case of microphone arrays (also simply referred to as arrays) 1 to 3, when the localization and separation state of the acoustic signal is good depending on the position of the speaker, the conversation support apparatus 1 </ b> A is The sound source localization unit 21 and the sound source separation unit 22 may not be provided. In addition, the conversation support device 1A may not include the language information detection unit 23 and the translation unit 24 when it is not necessary to perform translation or when the language used by the speaker is the same. That is, the language information detection unit 23 and the translation unit 24 may be optional.

As shown in the second row of FIG. 17, in the case of the microphone arrays 1 and 2, the conversation support device 1 </ b> A includes the sound source separation unit 22 when the acoustic signal separation state is good according to the position of the speaker. It does not have to be. In addition, the conversation support device 1A may not include the language information detection unit 23 and the translation unit 24 when it is not necessary to perform translation or when the language used by the speaker is the same.
As shown in the third line of FIG. 17, in the case of the microphone arrays 1 and 2, the conversation support apparatus 1 </ b> A may include a sound source localization unit 21 and a sound source separation unit 22 according to the position of the speaker. In addition, the conversation support device 1A may not include the language information detection unit 23 and the translation unit 24 when it is not necessary to perform translation or when the language used by the speaker is the same.

FIG. 18 is a diagram illustrating an example of sound source localization according to the present embodiment.
As shown in FIG. 18, four speakers Sp1 to Sp4 surround the conversation support apparatus 1A. Then, the speaker Sp1 pre-selects the fourth presentation image 351C closest to him / her, and the speaker Sp2 pre-selects the first presentation image 321C closest to himself / herself. The speaker Sp3 pre-selects the second presentation image 331C closest to himself / herself, and the speaker Sp4 pre-selects the third presentation image 341C closest to himself / herself.

When the speaker Sp1 is not speaking, the conversation support apparatus 1A cannot estimate the direction in which the speaker Sp1 is present. For this reason, the conversation support apparatus 1A first performs sound source localization processing on the acquired acoustic signal in the direction of 360 degrees with respect to the surface of the image display unit 15 of the conversation support apparatus 1A. And when speaker Sp1 utters, sound source localization is performed based on this utterance. Since the speech direction of the speaker Sp1 can be estimated by this processing, the conversation support apparatus 1A thereafter sets the search range of the acoustic signal of the speaker Sp1 to the fourth presentation image 351C (display area) closest to the speaker Sp1. based on, it may be changed in a range of, for example, theta ₁ angle. As a result, the calculation amount of the sound source localization process can be reduced, and the accuracy of the sound source localization can be further improved. Similarly, the conversation support apparatus 1A, after the speaker Sp2 uttered, the search range of the audio signal of the speaker Sp2, based on the closest first presentation image 321C speaker Sp2, e.g. theta ₂ of the angular range You may make it change to. Conversation assistance device 1A, after the speaker Sp3 uttered, the search range of the audio signal of the speaker Sp3, based on the nearest second presentation image 331C speaker Sp3, change the range of, for example, theta ₃ angles way may, after the speaker Sp4 uttered, the search range of the audio signal of the speaker Sp4, based on the nearest third presentation image 341C speaker Sp4, changing the angle in the range of e.g. theta ₄ You may do it. Note that each angle of θ _{1 to} θ ₄ is, for example, 90 degrees.

  In the example described above, an example in which there are four speakers has been described with reference to FIG. 18, but the present invention is not limited to this. When there are two speakers, the search range of the acoustic signal for each speaker is set to an angle range of 360 degrees to 180 degrees, for example, based on the display area (321A, 331A) for each speaker as shown in FIG. It may be changed. Alternatively, when there are three speakers, the search range of the acoustic signal for each speaker may be changed from an angle range of 360 degrees to 120 degrees, for example. That is, the conversation support apparatus 1A may change the search range of each speaker based on the display area of each speaker. Thereby, since conversation support apparatus 1A can narrow a search range, it can improve the estimation accuracy of direction and can reduce the calculation amount of conversation support apparatus 1A.

  Further, for example, the voice recognition unit 13A may perform speaker identification. For example, before starting recognition, the voice is registered in advance in the conversation support apparatus 1A for each speaker. Thereby, for example, when the speaker Sp1 speaks, the speech recognition unit 13A may identify the acoustic signal of the speaker Sp1 from the acoustic signal separated by the sound source separation unit 22.

  Further, the language displayed in the image presented to each speaker may be based on a language selected in advance from a menu. For example, when the speaker Sp1 selects Japanese as the language to be used from the menu, the translation unit 24 translates the result of another speaker speaking in French, and the translated result is displayed on the first character presentation image 322C. You may do it. For this reason, even if another speaker speaks in French, English, or Chinese, the conversation support apparatus 1A displays all the other speaker's utterances in Japanese on the fourth character presentation image 352C in FIG. You may make it do.

A processing procedure performed by the conversation support apparatus 1A will be described. FIG. 19 is a flowchart of a processing procedure performed by the conversation support apparatus 1A according to the present embodiment.
(Steps S101 to S103) Steps S101 to S103 are performed in the same manner as steps S1 to S3 (see FIG. 9). In step S101, each speaker may select whether or not to translate the speech of another speaker on the menu image 301.

(Step S104) The sound source localization unit 21 estimates the azimuth angle of the sound source based on the input signal input from the acoustic signal acquisition unit 12, and uses the azimuth angle information indicating the estimated azimuth angle and the N-channel acoustic signal as the sound source. Output to the separation unit 22. The sound source localization unit 21 advances the process to step S105 after step S104 ends.
(Step S105) The sound source separation unit 22 acquires the N-channel acoustic signal output from the sound source localization unit 21, and separates the acquired N-channel acoustic signal into acoustic signals for each speaker using, for example, the GHDSS method. . Next, the sound source separation unit 22 outputs the separated acoustic signal for each speaker and the azimuth angle information input from the sound source localization unit 21 to the language information detection unit 23. The sound source separation unit 22 advances the process to step S106 after step S105 ends.

(Step S106) The language information detection unit 23 detects the language for each speaker by a well-known method for each acoustic signal for each speaker input from the sound source separation unit 22. The language information detection unit 23 outputs the information indicating the detected language for each speaker, the acoustic signal for each speaker input from the sound source separation unit 22, and the azimuth information to the speech recognition unit 13A. The language information detection unit 23 advances the process to step S107 after step S106 ends.

(Step S <b> 107) The speech recognition unit 13 </ b> A is input from the acoustic signal acquisition unit 12 based on the information indicating the language for each speaker input from the language information detection unit 23, the acoustic signal for each speaker, and the azimuth information. Speech recognition processing is performed on the received acoustic signal to recognize the utterance content. Next, the speech recognition unit 13A outputs to the translation unit 24 utterance contents, information indicating the speaker, information indicating the direction of the speaker and the recognition data, and information indicating the language of each speaker. The voice recognition unit 13A advances the process to step S108 after step S107 ends.

(Step S108) The translation unit 24 translates the utterance content based on the utterance content input from the speech recognition unit 13A, information indicating the speaker, and information indicating the language of each speaker, and information indicating the translated utterance content Is added to or replaced with the information input from the voice recognition unit 13A and output to the image processing unit 14. The translation part 24 advances a process to step S109 after completion | finish of step S108.
(Steps S109 to S110) Steps S109 to S110 are performed in the same manner as steps S5 to S6 (see FIG. 9).
Above, the process which conversation support apparatus 1A performs is complete | finished.

  In the example illustrated in FIG. 19, the conversation support apparatus 1 </ b> A includes all the functional units illustrated in FIG. 16 and uses all the functional units. However, the present invention is not limited thereto. As illustrated in FIG. 17, the conversation support apparatus 1 </ b> A may select a function unit and a process to be used according to the microphone array.

As described above, the conversation support apparatus 1A according to the present embodiment includes the sound source estimation unit (sound source localization unit 21) that estimates the sound source direction of the user, and the image processing unit 14 is estimated by the sound source estimation unit. The recognition result recognized by the voice recognition unit 13A is displayed in a display area corresponding to each user of the display unit (image display unit 15) at a display angle based on the sound source direction.
In addition, the conversation support apparatus 1A according to the present embodiment includes a sound source separation unit 22 that separates a voice signal input to the voice input unit (sound collection unit 11 and acoustic signal acquisition unit 12) for each user, and performs image processing. The unit 14 displays a recognition result other than the user corresponding to the display region in the display region corresponding to each user of the display unit among the audio signals for each user separated by the sound source separation unit.

  With this configuration, according to the conversation support apparatus 1A of the present embodiment, the sound source localization unit 21 performs the sound source localization process and the sound source separation unit 22 performs the sound source separation process even when the sound source localization or separation situation is bad. Therefore, it is possible to accurately estimate the direction for each speaker and to separate the utterances for each speaker. As a result, according to the conversation support apparatus 1A of the present embodiment, other speakers can visually confirm the other party's utterance on the conversation support apparatus 1A with high accuracy, thereby supporting the hearing of the speaker. be able to.

The conversation support apparatus 1A according to the present embodiment includes a translation unit 24 that translates the recognition result recognized by the voice recognition unit 13A, and the image processing unit 14 displays the translation result translated by the translation unit (display unit). The image is displayed in a display area corresponding to each user of the image display unit 15).
In addition, the conversation support apparatus 1A according to the present embodiment includes a language information detection unit 23 that detects a language spoken by the user, and the translation unit 24 displays the recognition result other than the user corresponding to the display area as the language information. Translate into the language detected by the detector.

  With this configuration, according to the conversation support apparatus 1A of the present embodiment, since the language information detection unit 23 and the translation unit 24 are provided, even if the language used is different for each speaker, as necessary. The utterances of other speakers can be visually displayed on the conversation support apparatus 1A. As a result, according to the conversation support apparatus 1A of the embodiment, other speakers can visually confirm the other party's utterance on the conversation support apparatus 1A, so that the hearing of the speaker can be supported. .

  In the present embodiment, an example in which a plurality of speakers use the conversation support apparatus 1A has been described. However, the present invention is not limited to this. The conversation support device 1A may be used by one speaker. For example, when the speaker registers the language used as Japanese in the initial state and speaks English, the conversation support apparatus 1A translates the English spoken by the speaker into the registered language Japanese. You may make it display on the presentation area | region of the image corresponding to a speaker. Thereby, in the conversation support device 1A of the present embodiment, an effect of performing foreign language learning support is obtained.

Further, in the present embodiment, when one of the speakers leaves, the speaker who leaves may input or select information indicating the departure to the conversation support apparatus 1A when leaving.
For example, when the number of speakers is reduced from four to three, the conversation support apparatus 1A may change the layout shown in FIG. 8 to the layout shown in FIG.
On the other hand, when a speaker participates from the middle, the speaker who participates in the middle may input or select information indicating participation in the conversation support apparatus 1A. For example, when the number of speakers increases from three to four, the conversation support apparatus 1A may change the layout shown in FIG. 7 to the layout shown in FIG.

FIG. 20 is a diagram for explaining processing when the number of speakers according to the present embodiment has changed. In the example illustrated in FIG. 20, three speakers Sp <b> 1 to Sp <b> 3 are using the conversation support device 1 </ b> A. In FIG. 20, the utterance directions of the speakers Sp1 to Sp3 have been estimated.
For example, when the positions where the three speakers Sp1 to Sp3 are hardly changed, the acoustic signal localized by the sound source localization unit 21 includes an acoustic signal having an utterance direction different from that of the speakers Sp1 to Sp3. The support apparatus 1A may determine that a new speaker Sp4 has participated in the conversation. In the example shown in FIG. 20, the speaker Sp4 speaks from the upper right direction toward the page. In this case, the conversation support apparatus 1A estimates the utterance direction of the new speaker Sp4, and switches to the display screen corresponding to the four speakers as shown in FIG. 8 based on the estimated result. May be. In this case, since the position of the speaker Sp4 is between the speakers Sp1 and Sp3, the conversation support apparatus 1A displays the information display area corresponding to the speaker Sp4 as the first presentation image 621 and the third presentation image. Each display area may be re-layed out so as to be inserted between the display areas 641 and 641.

[Third Embodiment]
In the first embodiment and the second embodiment, an example in which the conversation support apparatus 1 or the conversation support apparatus 1A is one has been described. However, in the present embodiment, an example in which the conversation support apparatus 1 or the conversation support apparatus 1A is a plurality is used. explain. A plurality of conversation support apparatuses 1 or conversation support apparatuses 1A may be used for each speaker, for example.

FIG. 21 is a block diagram illustrating the configuration of the conversation support apparatus 1B according to the present embodiment. As shown in FIG. 21, the conversation support apparatus 1B includes a sound collection unit 11, an acoustic signal acquisition unit 12B, a speech recognition unit 13A, an image processing unit 14, an image display unit 15, an input unit 16, a sound source localization unit 21, and a sound source separation. Unit 22, language information detection unit 23, translation unit 24, and communication unit 31. The image processing unit 14 includes an image pattern generation unit 141, a display image generation unit 142, and an image composition unit 143.
In addition, the same code | symbol is used for the function part which has the same function as 1 A of conversation assistance apparatuses demonstrated in FIG. 16, and description is abbreviate | omitted. FIG. 21 illustrates an example of a configuration in which the conversation support apparatus 1B is based on the conversation support apparatus 1A illustrated in FIG. 16, but the conversation support apparatus 1B is based on the conversation support apparatus 1 illustrated in FIG. It may be the configuration. That is, the conversation support apparatus 1B may not include some functional units among the sound source localization unit 21, the sound source separation unit 22, the language information detection unit 23, and the translation unit 24 depending on the application.

  The acoustic signal acquisition unit 12B acquires M acoustic signals recorded by M (M is an integer of 1 or more) microphones 101 of the sound collection unit 11. For example, when M is 2, two acoustic signals recorded by the two microphones 101 are acquired. The acoustic signal acquisition unit 12B outputs the acquired M acoustic signals to the sound source localization unit 21 and the communication unit 31. The acoustic signal acquisition unit 12 </ b> B acquires L (L is an integer of 1 or more) acoustic signals input from the communication unit 31. The acoustic signal acquisition unit 12 </ b> B outputs the acquired L acoustic signals to the sound source localization unit 21. If the acoustic signal acquired from the communication unit 31 includes identification information for identifying the terminal, the acoustic signal acquisition unit 12B may output this identification information to the sound source localization unit 21.

  The communication unit 31 transmits the M acoustic signals input from the acoustic signal acquisition unit 12B to the other conversation support apparatus 1B. Further, the communication unit 31 outputs the L acoustic signals received from the other conversation support apparatus 1B to the acoustic signal acquisition unit 12B. For example, when the communication unit 31 receives two acoustic signals from each of the three conversation support apparatuses 1B, the received six acoustic signals (= 2 × 3 units) are sent to the acoustic signal acquisition unit 12B. Output. Further, the communication unit 31 may include identification information for identifying the terminal in the acoustic signal and output it to the acoustic signal acquisition unit 12B.

FIG. 22 is a diagram illustrating an example of the arrangement of a plurality of conversation support devices 1B according to the present embodiment. In the example shown in FIG. 22, four conversation support apparatuses 1B-1 to 1B-4 are arranged in a lattice pattern. Each of the conversation support devices 1B-1 to 1B-4 includes two microphones 101 among the microphones 101-1 to 101-8. For example, the conversation support apparatus 1B-1 includes a microphone 101-1 and a microphone 101-2.
The configuration of the conversation support devices 1B-1 to 1B-4 is the configuration shown in FIG. Each of the conversation support devices 1B-1 to 1B-4 communicates with each other via a communication unit 31 included in each device.

  Further, as shown in FIG. 22, each image display unit 15 of the conversation support devices 1 </ b> B- 1 to 1 </ b> B- 4 has one display area of information to be provided, and character presentation images 701 to 704 are also one. One by one. In the character presentation image 701 of the conversation support apparatus 1B-1, text in which the utterance content spoken by the speakers of the conversation support apparatuses 1B-2 to 1B-4 is recognized is displayed. In the character presentation image 702 of the conversation support device 1B-2, text in which the utterance content spoken by the speakers of the conversation support devices 1B-1, 1B-3, 1B-4 is recognized is displayed. In the character presentation image 703 of the conversation support device 1B-3, text in which the utterance content spoken by the speakers of the conversation support devices 1B-1, 1B-2, 1B-4 is recognized is displayed. In the character presentation image 704 of the conversation support apparatus 1B-4, text in which the utterance content spoken by the speakers of the conversation support apparatuses 1B-1 to 1B-3 is recognized is displayed.

  That is, as shown in FIG. 22, when using four conversation support apparatuses 1B-1 to 1B-4, the acoustic signal collected by the conversation support apparatus 1B-1 is communicated with the communication unit 31 via wireless communication. It transmits to other conversation assistance apparatuses 1B-2 to 1B-3. On the other hand, the acoustic signals picked up by the other conversation support apparatuses 1B-2 to 1B-3 are transmitted to the conversation support apparatus 1B-1 via wireless communication with the communication unit 31 of each apparatus. As a result, the conversation support apparatus 1B-1 performs voice recognition on each acoustic signal received from the conversation support apparatuses 1B-2 to 1B-3, and displays characters as a result of the voice recognition on the image display unit 15. To do. Each of the conversation support apparatuses 1B-1 to 1B-4 may directly perform voice recognition processing on the acoustic signal received from the other conversation support apparatus 1B.

  In addition, in FIG. 22, although the example which installs four conversation assistance apparatuses 1B-1 to 1B-4 adjacently was demonstrated, it is not restricted to this. For example, each of the conversation support devices 1B-1 to 1B-4 may be arranged in the vicinity of each speaker.

FIG. 23 is a diagram illustrating an example of an image displayed on the image display unit 15 of each conversation support apparatus 1B according to the present embodiment. FIG. 23 is an example of an image displayed on the image display unit 15 of the conversation support device 1B-3 among the four conversation support devices 1B-2 to 1B-4 as shown in FIG. is there.
In FIG. 23, the image of the area indicated by reference numeral 720 is an image corresponding to the speaker. The image of the area indicated by reference numeral 720 corresponds to an image 721 indicating a speaker corresponding to the conversation support apparatus 1B-1, an image 722 indicating a speaker corresponding to the conversation support apparatus 1B-2, and a conversation support apparatus 1B-3. And an image 724 showing a speaker corresponding to the conversation support apparatus 1B-4. The image 721 is, for example, red, the image 722 is, for example, green, the image 723 is, for example, blue, and the image 724 is, for example, yellow. The images 721 to 724 corresponding to the conversation support devices 1B-1 to 1B-4 are not limited to color images. For example, an avatar, an icon, a name, or the like corresponding to each of the conversation support devices 1B-1 to 1B-4 may be used.
The image displayed on the character presentation image 703 includes an image 731 based on the recognition data of the speaker corresponding to the conversation support device 1B-1, and the recognition data of the speaker corresponding to the conversation support device 1B-2. And an image 734 based on the recognition data of the speaker's utterance corresponding to the conversation support device 1B-4. These images 731 to 734 may be displayed in colors corresponding to the images 721 to 724, or may be displayed with an avatar, an icon, a name, or the like added thereto. In the case of an avatar, an icon, or a name, for example, the image may be added to the left of each image 731 to 734 and displayed. These display processes are performed by the image processing unit 14.

  Further, only one conversation support apparatus 1C may include all the functional units. The other three conversation support devices may include the sound collection unit 11, the acoustic signal acquisition unit 12B, the communication unit 31, the image processing unit 14, and the image display unit 15. In this case, the conversation support apparatus 1C having all functions acquires the acoustic signal from the other conversation support apparatus 1C by communication, and performs sound source localization processing, sound source separation processing, voice recognition processing, image generation processing, and the like. May be. Then, the generated image data may be transmitted to each conversation support apparatus 1C.

  FIG. 24 is a block diagram illustrating a configuration of a conversation support apparatus 1C according to the present embodiment. As shown in FIG. 24, the conversation support apparatus 1C includes a sound collection unit 11, an acoustic signal acquisition unit 12C, a speech recognition unit 13A, an image processing unit 14C, an image display unit 15, an input unit 16, a sound source localization unit 21, and a sound source separation. Unit 22, language information detection unit 23, translation unit 24, and communication unit 31C. In addition, the same code | symbol is used for the function part which has the same function as the conversation assistance apparatus 1B demonstrated in FIG. 21, and description is abbreviate | omitted. Note that although FIG. 24 illustrates an example in which the conversation support apparatus 1C has a configuration based on the conversation support apparatus 1B illustrated in FIG. 21, the conversation support apparatus 1C is based on the conversation support apparatus 1 illustrated in FIG. The structure made into may be sufficient. That is, the conversation support apparatus 1 </ b> C may not include some functional units among the sound source localization unit 21, the sound source separation unit 22, the language information detection unit 23, and the translation unit 24 depending on the application.

  The acoustic signal acquisition unit 12C acquires M acoustic signals recorded by M (M is an integer of 1 or more) microphones 101 of the sound collection unit 11. The acoustic signal acquisition unit 12 </ b> C outputs the acquired M acoustic signals to the sound source localization unit 21. The acoustic signal acquisition unit 12 </ b> C acquires L (L is an integer of 1 or more) acoustic signals input from the communication unit 31 </ b> C, and outputs the acquired L acoustic signals to the sound source localization unit 21. Note that the acoustic signal acquisition unit 12C may output the identification information to the sound source localization unit 21 when the acoustic signal acquired from the communication unit 31C includes identification information for identifying the terminal.

  The image processing unit 14C, based on the information indicating the speaker output from the translation unit 24, the information indicating the speaker direction, and the recognition data, character data for each terminal corresponding to the speaker and the image indicating the speaker direction. Is generated. The image processing unit 14C outputs character data for each terminal corresponding to the generated speaker and an image indicating the speaker direction to the communication unit 31C. In addition, the image processing unit 14 </ b> C causes the image display unit 15 to display character data for each terminal corresponding to the speaker corresponding to the own device and an image indicating the direction of the speaker.

  The communication unit 31C outputs the L acoustic signals received from the other conversation support device 1C to the acoustic signal acquisition unit 12C. The communication unit 31C transmits the character data for each terminal corresponding to the speaker and the image indicating the speaker direction input from the image processing unit 14C to the corresponding other conversation support apparatus 1C via wireless communication. .

For example, in FIG. 22, when the conversation support apparatus 1B-1 is a conversation support apparatus 1C-1 including all functional units, each of the conversation support apparatuses 1B-2 to 1B-4 includes conversation support units. Let it be apparatuses 1C-2 to 1C-4. Further, the first speaker Sp1 uses the conversation support device 1C-1, the second speaker Sp2 uses the conversation support device 1C-2, the third speaker Sp3 uses the conversation support device 1C-3, Assume that the fourth speaker Sp4 uses the conversation support apparatus 1C-4.
In this case, the conversation support apparatuses 1C-2 to 1C-3 transmit the collected M acoustic signals to the conversation support apparatus 1C-1 via wireless communication with the communication unit 31C. Then, the conversation support apparatus 1C-1 performs voice recognition on all of the acoustic signals collected by the own apparatus and the received acoustic signals.

Then, the image processing unit 14C displays an image indicating the direction of the first speaker Sp1 and character data that recognizes the utterance contents of the second speaker Sp2 to the fourth speaker Sp4 on the image display unit 15 of the own device. Let
The image processing unit 14C generates an image indicating the direction of the second speaker Sp2, and character data that recognizes the utterance contents of the first speaker Sp1, the third speaker Sp3, and the fourth speaker Sp4. Then, the communication unit 31C wirelessly communicates the generated image indicating the direction of the second speaker Sp2 and character data in which the utterance contents of the first speaker Sp1, the third speaker Sp3, and the fourth speaker Sp4 are recognized. To the conversation support apparatus 1C-2.

Similarly, the image indicating the direction of the third speaker Sp3 generated by the image processing unit 14C and the character data in which the utterance contents of the first speaker Sp1, the second speaker Sp2, and the fourth speaker Sp4 are recognized are communicated. The unit 31C transmits to the conversation support apparatus 1C-3 via wireless communication.
Further, the communication unit 31C performs wireless communication with the image indicating the direction of the fourth speaker Sp4 generated by the image processing unit 14C and the character data in which the utterance contents of the first speaker Sp1 to the third speaker Sp3 are recognized. To the conversation support apparatus 1C-4.

  As described above, the conversation support devices 1B and 1C according to the present embodiment include the communication units 31 and 31C that communicate with other conversation support devices, and include the voice input unit (sound collection unit 11 and acoustic signal acquisition unit 12B). 12C) inputs a voice signal received from another conversation support device received by the communication unit, and the voice recognition unit 13A uses the voice signal input from the voice input unit corresponding to the display area. Recognize audio signals other than the person.

  With this configuration, the conversation support apparatuses 1B and 1C according to the present embodiment can perform voice recognition using a plurality of conversation support apparatuses 1B.

[Fourth Embodiment]
In the first to third embodiments, an example of an image based on recognition data obtained by recognizing the utterance content of another speaker in the character presentation image corresponding to each speaker has been described. However, the present invention is not limited to this. In the present embodiment, an example will be described in which an image based on recognition data in which utterance content is recognized including not only another speaker but also one's utterance is displayed.

  FIG. 25 is a block diagram illustrating a configuration of a conversation support apparatus 1D according to the present embodiment. As shown in FIG. 25, the conversation support apparatus 1D includes a sound collection unit 11, an acoustic signal acquisition unit 12B, a voice recognition unit 13A, an image processing unit 14D, an image display unit 15, an input unit 16, a sound source localization unit 21, and a sound source separation. Unit 22, language information detection unit 23, translation unit 24, and communication unit 31D. In addition, the same code | symbol is used for the function part which has the same function as the conversation assistance apparatus 1B demonstrated in FIG. 21, and description is abbreviate | omitted. In FIG. 25, an example in which the conversation support apparatus 1D has a configuration based on the conversation support apparatus 1B illustrated in FIG. 21 has been described. However, the conversation support apparatus 1D is based on the conversation support apparatus 1 illustrated in FIG. The structure made into may be sufficient. That is, the conversation support apparatus 1D may not include some functional units among the sound source localization unit 21, the sound source separation unit 22, the language information detection unit 23, and the translation unit 24 depending on the application.

  The image processing unit 14D displays on the image display unit 15 character data in which the utterance content of the speaker using the conversation support device 1D is recognized. The image processing unit 14D outputs the character data in which the utterance content of the speaker using the conversation support device 1D is recognized to the communication unit 31D. The image processing unit 14D corrects the character data in which the utterance content of the speaker using the conversation support device 1D is recognized based on the operation of the input unit 16 that is a touch panel provided on the image display unit 15. Then, the corrected character data is output to the communication unit 31D.

  The communication unit 31D transmits to the other conversation support apparatus 1D via wireless communication the character data in which the utterance content of the speaker using the conversation support apparatus 1D input from the image processing unit 14D is recognized. . In addition, the communication unit 31D transmits the corrected character data input from the image processing unit 14D to another conversation support apparatus 1D via wireless communication. The communication unit 31D waits for transmission of the character data for a predetermined time after the character data in which the utterance content of the speaker using the conversation support device 1D is recognized, and the corrected character data. It may be determined whether or not is input. And the communication part 31D may be made to transmit the character data made to wait to other conversation assistance apparatus 1D, when the character data corrected within the predetermined time are not input. On the other hand, when the character data corrected within a predetermined time is input, the communication unit 31D does not transmit the character data that has been waiting, and only the input corrected character data is transmitted to the other conversation support device 1D. You may make it transmit to.

  FIG. 26 is a diagram illustrating an example of an image displayed on the image display unit 15 of the conversation support apparatus 1D according to the present embodiment. In the example shown in FIG. 26, four conversation support apparatuses 1D are used and displayed on the image display unit 15 of the conversation support apparatus 1D arranged at a position corresponding to the conversation support apparatus 1B-3 in FIG. It is an example of an image. In the following example, in FIG. 22, it is assumed that the conversation support apparatus 1B-1 is the conversation support apparatus 1D-1, and the conversation support apparatus 1D-1 is used by the first speaker Sp1. Similarly, it is assumed that the conversation support apparatus 1B-2 is the conversation support apparatus 1D-2, and the conversation support apparatus 1D-2 is used by the first speaker Sp2. It is assumed that the conversation support apparatus 1B-3 is the conversation support apparatus 1D-3, and the conversation support apparatus 1D-3 is used by the third speaker Sp3. It is assumed that the conversation support apparatus 1B-4 is the conversation support apparatus 1D-4, and the conversation support apparatus 1D-4 is used by the fourth speaker Sp4.

In FIG. 26, an image 751 is character data in which the utterance content uttered by the third speaker Sp3 is recognized. The image 751 is a result of voice recognition of an acoustic signal spoken by the third speaker Sp3 “Did you meet Mr. Kawai yesterday?”. However, the third speaker Sp3 intends to speak “Kawai” as “Mr. Kawai”, but the recognition result is displayed as “Mr. Kawai”. Even if this image 751 is not displayed on the image display unit 15 of the conversation support device 1D-3 but is displayed only on each image display unit 15 of the other conversation support devices 1D-1, 1D-2, 1D-4. Since the first speaker Sp1, the second speaker Sp2, and the fourth speaker Sp4 do not know “Mr. Kawai”, the conversation may not be established.

For this reason, in this embodiment, the character data in which the utterance content that the user uttered is recognized is also displayed on the image display unit 15.
As a result, the third speaker Sp3 confirms the image 751, and selects, for example, the image 752 at a location with different recognition by operating the input unit 16 of the touch panel provided in the image display unit 15.
Then, the image processing unit 14D displays an image 753 including other conversions “Mr. Kawai”, “Mr. Kawai”, and the like corresponding to the selected image 752 in the vicinity of the selected image 752, for example, as shown in FIG. To do. Thereby, the third speaker Sp3 selects a desired “Mr. Kawai” from the image 753. The input unit 16 may output information indicating the selected “Mr. Kawai” to the image processing unit 14D. Then, the communication unit 31D of the conversation support apparatus 1D-3 may retransmit the character data corrected by the image processing unit 14D to another conversation support apparatus 1D.

In the example described above, an example in which the present invention is applied to a plurality of conversation support apparatuses 1D as illustrated in FIG. 22 has been described, but the present invention is not limited thereto. As described in the first and second embodiments, the present invention may also be applied to one conversation support device 1 or 1A.
For example, in FIG. 8, the conversation support apparatus 1 or 1A may display character data that recognizes the utterance content of the first speaker Sp1 on the first character presentation image 322A.

As described above, the conversation support apparatus 1D of the present embodiment includes the input unit 16 that selects a part of the image displayed on the display unit (image display unit 15), and the image processing unit 14D is selected by the input unit. If a part of the selected image is a recognition result, another recognition candidate corresponding to the selected recognition is displayed on the display unit, and the recognition result is corrected to the candidate selected by the input unit among the recognition candidates. The recognized result is transmitted to another conversation support apparatus via the communication unit 31D.
With this configuration, the user's utterance content can be correctly presented to other users.

In the first to fourth embodiments, an example in which there are 2 to 4 speakers has been described. However, the number of speakers is not limited to this, and may be 5 or more. In this case, the image pattern generation unit 141 generates a display pattern according to the number of people. Alternatively, as described in the third embodiment, it is possible to cope with five or more people by using a conversation support device for each speaker.
When a plurality of conversation support devices (1, 1A, 1B, 1C, 1D) are used, the images of one conversation support device (1, 1A, 1B, 1C, 1D) are used in the third and fourth embodiments. Although the example which displays for one person on the display part 15 was shown, it is not restricted to this. When a plurality of conversation support devices (1, 1A, 1B, 1C, 1D) are used, display patterns displayed on each conversation support device (1, 1A, 1B, 1C, 1D) are shown in FIGS. As described above, a screen corresponding to a plurality of speakers may be used. For example, according to the third and fourth embodiments, conversation support is provided by displaying the display pattern shown in FIG. 6 on the image display unit 15 of each conversation support device (1, 1A, 1B, 1C, 1D). Two devices (1, 1A, 1B, 1C, 1D) can be used to support four speakers.

  In the first to fourth embodiments, a tablet terminal or the like has been described as an example of the conversation support devices 1, 1A, 1B, and 1D. However, the present invention is not limited to this. For example, the conversation support devices 1, 1 </ b> A, 1 </ b> B, and 1 </ b> D may be applied to a device that includes the image display unit 15 on the table. Alternatively, the conversation support devices 1, 1A, 1B, and 1D may be applied to an electronic blackboard or the like. Further, as described in the third embodiment, when a conversation support device is configured with a plurality of terminals, these terminals do not have to be arranged in the same room, for example. For example, the plurality of conversation support devices 1, 1A, 1B, 1D may be arranged in different rooms. Furthermore, the plurality of conversation support apparatuses 1, 1A, 1B, and 1D may be mounted on, for example, a plurality of vehicles. In addition, when a plurality of terminals are connected via a network, the plurality of conversation support devices 1, 1A, 1B, and 1D may be arranged in different countries or regions, for example. As a result, it is possible to support the hearing of a plurality of speakers located at distant positions.

In the first to fourth embodiments, the conversation support devices 1, 1A, 1B, and 1D correspond to, for example, the latest utterance with an image showing characters recognized by the speaker displayed in the image display area. The image to be displayed may be displayed darkly, and the image corresponding to the past utterance may be displayed lightly. For example, the conversation support devices 1, 1 </ b> A, 1 </ b> B, and 1 </ b> D may display an image corresponding to the latest utterance in bold and display an image corresponding to the past utterance in thin characters. Or you may make the size of the character used for the image corresponding to the past utterance larger than the size of the character used for the image corresponding to the latest utterance.
Further, in the first to fourth embodiments, for example, FIG. 14 illustrates an example in which the position where an image in which a speaker's utterance is recognized is displayed in order from top to bottom is not limited to this. In FIG. 14, for example, the conversation support apparatuses 1, 1 </ b> A, 1 </ b> B, and 1 </ b> D display an image corresponding to the latest utterance at, for example, the approximate center of the first character presentation image 522 of the first presentation image 521. An image corresponding to the utterance may be displayed at the top.

  In the first to fourth embodiments, the conversation support devices 1, 1 </ b> A, 1 </ b> B, and 1 </ b> D display, for example, a display area (for example, the first presentation) of information presented for each speaker according to the utterance amount for each speaker. The brightness of the image 321A) or the character presentation image (for example, the first character presentation image 322A) in the display area may be changed and displayed. The conversation support apparatuses 1, 1A, 1B, and 1D detect, for example, the number of utterances or utterance time for each speaker, and the display area or character presentation image of the speaker with the detected number of utterances or utterance time being smaller than that of other speakers May be controlled to be lower than the luminance in the initial state. Alternatively, the conversation support devices 1, 1A, 1B, and 1D control the speaker display area or the character presentation image so that the detected number of utterances or utterance time is higher than that of other speakers so as to be higher than the initial brightness. Also good. Thus, the user can recognize his / her utterance time or the number of utterances. Alternatively, the moderator can effectively use the display in the progress of the conference or the like by urging the speaker with a small number of utterances or a short utterance time to see the display.

  In the first to fourth embodiments, an example in which the conversation support devices 1, 1A, 1B, and 1D include the voice recognition unit 13 or 13A has been described. However, the voice recognition unit 13 or 13A is provided via a network, for example. You may be made to do.

  Note that a program for realizing the functions of the conversation support apparatuses 1, 1A, 1B, and 1D according to the present invention is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system. Various processes described above may be performed by executing them. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer system” includes a WWW system having a homepage providing environment (or display environment). The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

DESCRIPTION OF SYMBOLS 1, 1A, 1B, 1C, 1D ... Conversation support apparatus, 11 ... Sound collection part, 12, 12B, 12C ... Acoustic signal acquisition part, 13, 13A ... Voice recognition part, 14, 14C, 14D ... Image processing part, 15 ...
Image display unit, 16 ... input unit, 21 ... sound source localization unit, 22 ... sound source separation unit, 23 ... language information detection unit, 24 ... translation unit, 31, 31C, 31D ... communication unit, 101-1 to 101-N, 101-
M ... Microphone, 141 ... Image pattern generation unit, 142 ... Display image generation unit, 143 ... Image composition unit, 321A, 321B, 321C, 521, 621 ... First presentation image, 322A,
322B, 322C, 522, 622 ... first character presentation image, 323A, 333A, 323
B to 324B, 323C to 353C, 523, 533 ... orientation images, 331A, 331B,
331C, 531, 631 ... second presentation image, 332A, 332B, 332C, 532, 6
32 ... 2nd character presentation image, 341B, 341C, 641 ... 3rd presentation image, 342B, 34
2C, 642 ... third character presentation image, 351C ... fourth presentation image, 352C ... fourth character presentation image, 524A-524C, 534A-534D, 634A, 644A, 625B, 645
B ... Character image with utterance recognized

Claims (12)

An audio input unit for inputting audio signals of two or more users;
A voice recognition unit for recognizing a voice signal input to the voice input unit;
A sound source estimation unit that estimates a sound source direction of a sound signal input to the sound input unit;
A display unit for displaying a recognition result recognized by the voice recognition unit;
If the display area corresponding to each user is set as the image display area of the display unit and a voice is detected from a direction different from that of the speaker already localized, a new speaker joins the conference. An image processing unit that determines that the utterance content is displayed between the text display frames of the adjacent already recognized speakers,
A conversation support device. The image processing unit
The display color of the display area corresponding to each user, the pattern, the icon displayed in the display area, and the avatar displayed in the display area are displayed differently for each user. Item 4. The conversation support device according to Item 1. The image processing unit
The conversation support according to claim 1 or 2, wherein an image based on the sound source direction estimated by the sound source estimation unit is displayed in the display area corresponding to each user of the display unit. apparatus. A sound source separation unit for separating the sound signal input to the sound input unit for each user;
The image processing unit
Of the audio signals for each user separated by the sound source separation unit, the recognition result other than the user corresponding to the display region is displayed in a display region corresponding to the user of the display unit. The conversation support apparatus according to any one of claims 1 to 3, wherein A position estimation unit for estimating the position of the user;
The image processing unit
The display area corresponding to each user is set or rearranged in the image display area of the display section at a position corresponding to the position of the user estimated by the position estimation section. The conversation support device according to claim 1. The position estimation unit
The conversation support apparatus according to claim 5, wherein the position of the user is estimated using a voice signal input to the voice input unit. A translation unit that translates the recognition result recognized by the voice recognition unit;
The image processing unit
The conversation support apparatus according to any one of claims 1 to 6, wherein a translation result translated by the translation unit is displayed in the display area corresponding to each user of the display unit. . A language information detection unit for detecting a language spoken by the user;
The translation unit
The conversation support apparatus according to claim 7, wherein the recognition result other than the user corresponding to the display area is translated into a language detected by the language information detection unit. Provided with a communication unit that communicates with other conversation support devices,
The voice input unit
Input the voice signal received from the other conversation support device received by the communication unit,
The voice recognition unit
The conversation according to any one of claims 1 to 7, wherein a speech signal other than the user corresponding to the display area is recognized among the speech signals input from the speech input unit. Support device. An input unit for selecting a part of the image displayed on the display unit;
The image processing unit
When a part of the image selected by the input unit is a recognition result, another recognition candidate corresponding to the selected recognition is displayed on the display unit, and the recognition unit selected by the input unit among the recognition candidates The conversation support apparatus according to claim 9, wherein the recognition result is corrected by a candidate, and the corrected recognition result is transmitted to the other conversation support apparatus via the communication unit. A voice input procedure in which a voice input unit inputs voice signals of two or more users;
A speech recognition unit for recognizing a speech signal input by the speech input procedure;
A sound source estimation unit for estimating a sound source direction of the audio signal input by the audio input procedure;
The image processing unit sets a display region corresponding to each user as an image display region of a display unit on which a recognition result recognized by the voice recognition procedure is displayed, and a speaker that has already been sound source localized When speech is detected from a different direction, it is determined that a new speaker is participating in the conference, and the content of the utterance is displayed between the text display frames of adjacent already recognized speakers. Processing procedure and
A method for controlling a conversation support apparatus, comprising: In the computer of the conversation support device,
A voice input procedure for inputting voice signals of two or more users;
A voice recognition procedure for recognizing a voice signal input by the voice input procedure;
A sound source estimation procedure for estimating a sound source direction of the audio signal input by the audio input procedure;
The display area corresponding to each user is set as the image display area of the display unit on which the recognition result recognized by the voice recognition procedure is displayed, and the voice is heard from a direction different from that of the speaker already localized. If detected, an image processing procedure for determining that a new speaker is participating in the conference and displaying the utterance content between the text display frames of the adjacent already recognized speakers;
A program for a conversation support apparatus, characterized in that

Images