JP7361460B2 - Communication devices, communication programs, and communication methods - Google Patents

Description

The present invention relates to a communication device, a communication program, and a communication method, and is applicable to, for example, a communication device used in a video conference system, a telephone conference system, and the like.

2. Description of the Related Art In recent years, there have been an increasing number of opportunities to conduct calls and communicate with remote locations using communication systems such as video conference systems and telephone conference systems.

In a remote call system, in order to make a call to a remote party, you enter contact information such as the telephone number of the other party using an input device (e.g., mouse, keyboard, remote control, etc.) connected to the remote call system. Select and connect. Furthermore, in recent years, with the spread of mobile terminals (for example, terminals such as smartphones and tablet computers), some remote call systems operate on mobile terminals. In this case, you can connect by entering the contact information using the keyboard displayed on the screen of the mobile device, connect by touching the contact information displayed on the touch panel display of the mobile device, or connect by entering the contact information on the keyboard displayed on the screen of the mobile device. A remote call system often connects to the other party by touching the displayed image of the other party.

Further, Patent Document 1 proposes a communication support robot system that incorporates a remote call system into a robot and supports communication between close relatives and single elderly people.

The communication support robot system described in Patent Document 1 connects to the other party by touching an image of a close relative or elderly person displayed on a touch panel display, and starts a call.

Japanese Patent Application Publication No. 2015-184597

However, in the communication support robot system described in Patent Document 1, the contact information of the other party is input using an input device, or the contact information displayed on the touch panel display is touched, similar to the connection method of a conventional remote call system. Start or end a call by doing something like Connecting to a remote location using conventional connection methods is different from an actual face-to-face conversation, so the sense of realism (for example, the feeling of having a face-to-face conversation) is very low.

In order to solve the above problem, for example, the voice recognition system installed in the communication support robot described in Patent Document 1 is used to identify the name of the other party to connect to and a command to start a conversation (for example, " The user utters a calling voice (hereinafter referred to as a calling voice) such as "Person's name + Hello", "Person's name + Good evening", etc., inputs the words into the voice recognition system, and the communication support robot determines the connection destination from the voice recognition result. It is also possible to make the determination and start the connection.

However, after the calling voice is input to the voice recognition system and the voice recognition result is obtained, the other party is determined from the voice recognition result and the person is connected to the other party, so the calling voice is not transmitted to the other party. For this reason, from the perspective of the other party, the connection is suddenly made, which makes the other party feel uncomfortable and uneasy, and the sense of realism is not improved.

In addition, for example, if there are people talking around you or there is a lot of noise from an air conditioner, etc., the noise in the usage environment is large, and even if the user utters a calling voice, the communication support described in Patent Document 1 Robots are unable to pick up voice calls with good quality, and cannot connect or disconnect from the other party properly.

Furthermore, in the communication support robot system described in Patent Document 1, since the user of the robot is for a single elderly person, the remote call system can only be used by one person, and the remote call system can be used by multiple users. I can't do it.

Therefore, even if the usage environment is noisy, it can collect the calling voice uttered by any one of multiple users, correctly recognize the collected calling voice, and start a connection with the other party using that calling voice. After that, you can start a realistic call by transmitting the calling voice to the other party, and when ending the call, a voice ( There is a need for a communication device, a communication program, and a communication method that can correctly recognize the disconnection voice (hereinafter referred to as disconnection voice) and disconnect the call with the other party after the call ends.

The present invention has been made in consideration of the above points, and uses a microphone array to perform signal processing that emphasizes voices uttered by multiple speakers, and simultaneously stores the processed signals in a buffer. Speech recognition is performed on the processed signal. Then, it is determined whether the voice recognition result is a calling voice, and if it is a calling voice, it connects to the other party, outputs the calling voice held in the buffer, transmits the calling voice to the other party, and completes the call. You can start. Furthermore, when ending a call, signal processing is performed to emphasize the sounds uttered by a plurality of users using a microphone array, and speech recognition is performed on the signal-processed signals. The present invention attempts to provide a calling processing device that can determine whether the voice recognition result is a disconnection voice, and if it is a disconnection voice, disconnect the call with the other party to reproduce a situation that is more similar to a face-to-face conversation. It is.

For example, collecting the voices of multiple speakers in a noisy environment can be solved by signal processing that uses a microphone array to enhance the voices. Reproducing the calling voice of the other party and improving the sense of presence are solved by outputting the calling voice held in the buffer.

A communication device according to a first aspect of the present invention includes (1) a person detection unit that detects one or more people from an input video signal and acquires information regarding the position of each detected person; and (2) person detection. (3) a signal processing unit that forms the directivity of one or more microphones based on information regarding the position of each person in the part and extracts the audio signal of each person; and (3) a connection command that is sent after connecting with the other party. (4) a holding unit that is a buffer for reproducing audio on the other party's side and holds audio signals of each person for a certain period of time including the connection command audio by the signal processing unit; (5) a voice recognition unit that performs voice recognition based on the input voice signal when a connection is detected; a command storage unit that stores commands; (6) a command determination unit that determines whether the voice recognition result by the voice recognition unit includes a connection command or a disconnection command stored in the command storage unit; and (7) command determination. (8) an output switching unit that determines an output audio signal according to the command determination result by the unit; and (8) a connection determination unit that performs connection processing with the other party's connection destination based on the voice recognition result and the command determination result. and when the voice recognition result includes a connection command, the connection determination unit performs connection processing with the other party's connection destination, and the output switching unit is held in the holding unit after connection with the other party's connection destination. The present invention is characterized in that after outputting an audio signal including a connection command , a signal processed by a signal processing section is outputted.

The communication program according to the second aspect of the present invention includes: (1) a person detection unit that detects one or more people from an input video signal and acquires information regarding the position of each detected person; ) A signal processing unit that forms the directivity of one or more microphones based on the information regarding the position of each person in the person detection unit and extracts the audio signal of each person, and (3) transmits after connecting with the other party. (4) a buffer for reproducing the connection command voice on the other party's side, and a holding section that holds each person's voice signal including the connection command voice by the signal processing section for a certain period of time; (5) a voice recognition unit that performs voice recognition based on input voice signals when multiple people are detected; and (5) at least a connection command to start a connection with a connection destination and a disconnection command to disconnect the connection. a command storage unit that stores a plurality of commands including (6) a command determination unit that determines whether the voice recognition result by the voice recognition unit includes a connection command or a disconnection command stored in the command storage unit; ) an output switching unit that determines an output audio signal according to the command determination result by the command determination unit; and (8) a connection determination unit that performs connection processing with the other party's connection destination based on the voice recognition result and the command determination result. When the voice recognition result includes a connection command, the connection determination unit performs connection processing with the other party's connection destination, and the output switching unit sends the message to the holding unit after connection with the other party's connection destination. It is characterized in that after outputting the audio signal including the held connection command , the signal processed by the signal processing section is outputted.

In the communication method according to the third aspect of the present invention, (1) the person detection unit detects one or more people from the input video signal and acquires information regarding the position of each detected person; The processing unit forms the directivity of one or more microphones based on the information regarding the position of each person in the person detection unit, and extracts the audio signal of each person, and (3) the holding unit extracts the audio signal of each person . This is a buffer for reproducing the connection command voice transmitted after connection with the other party, and holds the voice signal of each person including the connection command voice by the signal processing unit for a certain period of time , and (4) the voice recognition unit, When one or more people are detected by the person detection unit, voice recognition is performed based on the input voice signal, and (5) the command storage unit stores at least a connection command and a command to start a connection with the connection destination. A plurality of commands including a disconnection command for disconnecting the connection are stored, and (6) a command determination unit determines whether the voice recognition result by the voice recognition unit includes a connection command or a disconnection command stored in the command storage unit. (7) The output switching unit determines the output audio signal according to the command determination result by the command determination unit, and (8) the connection determination unit determines the output audio signal of the other party based on the voice recognition result and the command determination result. (9) When the voice recognition result includes a connection command, (10) the connection determination section performs connection processing with the other party's connection destination; (11) the output switching section: The device is characterized in that after connection with the other party's connection destination, an audio signal including a connection command held in the holding unit is output, and then a signal processed by the signal processing unit is output.

According to the present invention, even when the noise in the usage environment is large, voices uttered by multiple users are emphasized, it is determined whether the voice is a calling voice, and if it is a calling voice, a connection with the other party is started using the calling voice. Then, by transmitting the calling voice to the other party, it is possible to start a realistic call. Also, when ending a call, the voice uttered by multiple users is emphasized and it is determined whether it is a disconnection voice. However, in the case of disconnection voice, the call with the other party can be disconnected.

Furthermore, according to the present invention, when the user connects to the other party even when the user is away from the microphone, the name, etc. of the other party to be connected and the words used to start the conversation are the same as when talking face-to-face. By starting the connection with and ending the connection with the words that end the conversation, both parties can experience a high sense of realism by recreating the beginning and end of a conversation.

FIG. 1 is a block diagram showing the configuration of a communication device according to a first embodiment. FIG. 2 is an explanatory diagram illustrating an example of the arrangement of equipment related to a communication device installed in a room of one base and the positional relationship with a user in the first embodiment. FIG. 2 is an explanatory diagram illustrating a configuration example of a command list section according to the first embodiment. FIG. 2 is a block diagram showing the configuration of a communication device according to a second embodiment. FIG. 7 is an explanatory diagram (part 1) illustrating an example of the arrangement of devices related to a communication device installed in a room of one base and the positional relationship with a user in the second embodiment; FIG. 7 is an explanatory diagram (Part 2) illustrating an example of the arrangement of devices related to communication devices installed in a room of one base and their positional relationship with users in the second embodiment;

(A) First Embodiment Below, embodiments of a communication device, a communication program, and a communication method of the present invention will be described in detail with reference to the drawings.

The first embodiment exemplifies a case where the above-described communication device, communication program, and communication method of the present invention are applied to a microphone input unit of a video conference system, a telephone conference system, or the like.

(A-1) Configuration of First Embodiment FIG. 1 is a block diagram showing the configuration of a communication device 100 according to the first embodiment.

The communication device 100 of the first embodiment may be constructed as a dedicated board, or may be realized by writing a remote communication program to a DSP (digital signal processor), for example. Although it may be realized by a CPU and software (for example, a remote communication program) executed by the CPU, it can be functionally represented as shown in FIG.

The communication device 100 communicates with the other party by communicating video and audio signals with the communication device installed at the other party's base in a remote location via the network 107. . Here, it is assumed that the communication device 100 illustrated in FIG. 1 is also located at the other party's base.

In FIG. 1, a communication device 100 according to the first embodiment includes a microphone array 101, a microphone amplifier 102, an analog-to-digital (AD) converter 103, a video camera 104, a call processing unit 105, a NW communication unit 106, a digital It includes an analog (DA) converter 108, a speaker amplifier 109, speakers 110a and 110b, and a monitor 111.

The microphone array 101 includes a plurality of microphones (hereinafter also referred to as "microphones") that receive human voices and sounds.

The microphone amplifier 102 amplifies each of the plurality of input signals (analog sound signals) received by each microphone of the microphone array 101 and outputs the amplified signals to the AD converter 103 .

The AD conversion unit 103 converts the plurality of input signals amplified by the microphone amplifier 102 from analog signals to digital signals, and outputs the digital signals to the communication device 100. Hereinafter, the signal converted by the AD converter 103 will also be referred to as a "microphone input signal."

The video camera 104 is a photographing device (imaging device) installed at its own base (the base where the communication device 100 is installed). The video signal captured by the video camera 104 is output to the NW communication unit 106 via the calling processing unit 105, and the video signal is transmitted to the network 107 by the NW communication unit 106.

A video signal from the video camera 104 and a microphone input signal received by the microphone array 101 are input to the calling processing unit 105 . A video signal input from the video camera 104 is output to the NW communication section 106 via the calling processing section 105 and transmitted to the network 107.

The calling processing unit 105 determines whether or not a person is imaged in the video signal input from the video camera 104 . If it is determined that a person is shown in the video signal, the calling processing unit 105 performs signal processing on the plurality of input microphone input signals and outputs the processed signals to the NW communication unit 106. It is saved in the audio buffer section 117.

Further, the calling processing unit 105 performs voice recognition using the processed signal, and determines whether the voice recognition result matches one of the commands in the command list unit 119. If the voice recognition result matches one of the commands, the calling processing unit 105 outputs the connection determination result and the sound signal stored in the audio buffer unit 117 to the NW communication unit 106 for a certain period of time. Then, when the output is completed for a certain period of time, the calling processing section 105 outputs the signal processed again to the NW communication section 106. If the voice recognition result does not match one of the commands, the calling processing unit 105 outputs the connection determination result and the signal processed by the signal to the NW communication unit 106.

On the other hand, if it is determined that no person is shown in the video signal, the calling processing unit 105 stops signal processing. At this time, no audio is output to the NW communication unit 106.

The NW communication unit 106 transmits and receives video signals and audio signals to and from the communication device 100 installed at the other party's base via the network 107. The NW communication unit 106 performs connection processing with the network 107 based on the connection determination result from the calling processing unit 105. That is, upon receiving the instruction to connect to the network 107, the NW communication unit 106 starts connecting to the communication device 100 of the other party to which the instruction was given. After being connected to the network 107, the communication device 100 exchanges audio with the other party's communication device 100 via the NW communication unit 106.

Further, upon receiving an instruction to disconnect from the network 107, the NW communication unit 106 disconnects from the other party's communication device 100.

The DA conversion unit 108 converts the sound signal from the network 107 (the sound signal transmitted via the NW communication unit 106) from a digital signal to an analog signal, and outputs the analog signal to the speaker amplifier 109.

The speaker amplifier 109 amplifies the analog signal converted by the DA converter 108 and outputs it to the speakers 110a and 110b.

The speakers 110a and 110b are speakers that convert electrical signals into air vibrations and output them as sound. In the first embodiment, the speakers 110a and 110b are stereo speakers, but the speakers 110a and 110b are not limited to stereo speakers.

The monitor 111 is a video output device (video output device). The video output by the monitor 111 is, for example, a video shot by a video camera 104 installed at the other party's base, and this video (encoded data) is received by the NW communication unit 106 via the network 107. After being decoded, it is input to the monitor 111.

Next, a detailed configuration of the call processing unit 105 according to the first embodiment will be explained.

The calling processing section 105 includes a sound input terminal 115, a video input terminal 112, a video output terminal 113, a person position detection section 114, a signal processing section 116, an audio buffer section 117, a voice recognition section 118, a command list section 119, and a command determination section. 120, an output switching section 121, a sound output terminal 122, a connection determination section 123, and a connection determination result output terminal 124.

The video input terminal 112 is an interface unit that inputs a video signal from the video camera 104.

The video output terminal 113 is an interface unit that outputs a video signal input from the video camera 104 to the NW communication unit 106.

The person position detection unit 114 determines whether or not a person is shown in the video signal input from the video camera 104, and outputs the determination result to the signal processing unit 116 and the voice recognition unit 118. For example, the person position detection unit 114 determines whether or not a person is shown in the video frame by image processing using the input video signal, and if a person is detected, it indicates that a person has been detected. A determination result (for example, "1", etc.) is output, and in other cases, a determination result (for example, "0", etc.) indicating that no person is detected is output.

Further, when it is determined that a person is shown in the video signal, the person position detection section 114 outputs information on the direction in which the person is present to the signal processing section 116. Furthermore, if multiple people are detected, direction information for each detected person is output.

The sound input terminal 115 is an interface unit that inputs a microphone input signal from the AD conversion unit 103.

The signal processing section 116 performs signal processing on the input microphone input signal, and sends the signal processed signal (hereinafter also referred to as "microphone array processing signal") to the output switching section 121, the audio buffer section 117, and the speech recognition section 118. Output.

The audio buffer section 117 is a buffer that holds the signal processed by the signal processing section 116 for a certain period of time. The audio buffer section 117 outputs the held signal to the output switching section 121 after a certain period of time has elapsed.

The speech recognition section 118 performs speech recognition on the microphone array processed signal processed by the signal processing section 116 and outputs the speech recognition result to the command determination section 120 .

The command list section 119 is a text file that holds a list of commands. Here, the command can include various commands, and in this embodiment, a connection command and a disconnection command are included as examples of commands. A detailed explanation of the connection command and disconnection command, which are examples of commands, will be given later.

The command determination unit 120 determines whether or not the voice recognition result from the voice recognition unit 118 exists in the commands held in the command list unit 119, and transmits the determination result to the output switching unit 121 and the connection determination unit. It is output to section 123. A detailed explanation of the determination method by the command determination unit 120 will be given later.

The output switching unit 121 is connected to the signal processing unit 116 and the audio buffer unit 117, and switches between the output signal from the signal processing unit 116 and the output from the audio buffer unit 117 according to the determination result by the command determination unit 120. The signal is switched and output to the sound output terminal 122.

The sound output terminal 122 is an interface unit that outputs the sound signal switched by the output switching unit 121. The sound signal output from the sound output terminal 122 becomes the sound signal output from the calling processing section 105.

The connection determination unit 123 determines the connection to the network 107 based on the determination result determined by the command determination unit 120.

For example, when the voice recognition result is determined to match the person's name and the connection command, the connection determination unit 123 determines the communication device 100 to which the other party is connected based on the connection command, and A connection determination result including information regarding the connected destination and an instruction to connect to the connected destination is output to the NW communication unit 106. If it is determined that the voice recognition result matches the disconnection command, the connection determination unit 123 outputs the connection determination result including a connection disconnection instruction with the connected communication device 100 to the NW communication unit 106. .

The connection determination result output terminal 124 outputs the connection determination result from the connection determination section 123 to the NW communication section 106.

(A-2) Operation of First Embodiment Next, processing operations in the communication device 100 according to the first embodiment will be described in detail with reference to the drawings.

FIG. 2 is an explanatory diagram illustrating an example of the arrangement of equipment related to a communication device installed in a room at one base and the positional relationship with a user in the first embodiment. Note that it is assumed that the remote communication 100 is also installed at the other base as in FIG. 2.

In FIG. 2, the room 151 is, for example, a conference room, and the height of the room 151 is set to a height where the monitor 111 can be easily installed and has enough room (for example, the height of the monitor 111 + several meters, or 2 meters). The size (area) of the room 151 should be large enough to easily install the monitor 111, the microphone array 101, the speakers 110a and 110b, etc., and have enough room for the users 152a and 110b. 152b should be sufficiently wide (for example, several meters in width and height) for conversation.

First, when the communication device 100 starts operating, the monitor 111 displays the video captured by the video camera 104 of the communication device 100 at the other party's base.

That is, when the communication device 100 starts operating and the video camera 104 at its own site is activated, the video signal captured by the video camera 104 is given to the NW communication unit 106 via the call processing unit 105, and transmits the video signal to the NW communication section 106 of the other party's base via the network 107. As a result, the image of the own base is displayed on the monitor 111 of the other party's base. Similarly, the image of the other party's base is displayed on the monitor 111 of the own base.

At this time, the communication devices 100 at both bases are not transmitting or receiving audio signals, and only the video captured by the video camera 104 of the other base is displayed on the monitor 111 at both bases, so that the situation at each base can be confirmed.

Further, the video signal captured by the video camera 104 is input to the video input terminal 112 of the calling processing section 105, and the video signal is input to the person position detection section 114.

After the communication device 100 starts operating until a person approaches the communication device 100, the communication device 100 at each location does not send or receive audio signals, and both locations monitor only the video captured by the video camera 104 on the other side. 111, allowing you to check the status of each other's bases.

After a while after the communication device 100 starts operating, users 152a and 152b who are trying to talk to a person at the other party's base look for the image of the other party's base and search for or confirm the other party. To do this, approach monitor 111. At this time, as illustrated in FIG. 2, the video camera 104 is installed near the monitor 111 (in the example of FIG. 2, the video camera 104 is installed above the monitor 111). , the users 152a and 152b approaching the monitor 111 are photographed, and a video signal showing the users 152a and 152b is input to the video input terminal 112 of the calling processing section 105.

When a video signal showing the users 152a and 152b begins to be input to the video input terminal 112 of the calling processing unit 105, a video signal from the video camera 104 is input to the person position detection unit 114.

The person position detection unit 114 detects the two users 152a and 152b shown in the video signal, and the person position detection unit 114 detects a determination result (for example, “1”) indicating that a person is shown. The information is output to the signal processing unit 116 and the voice recognition unit 118, and information (for example, direction information) regarding the positions of the users 152a and 152b in the video frame is output to the signal processing unit 116.

Furthermore, either user 152a or 152b who attempts to talk to a person at the other party's base utters a calling voice in order to call out the person with whom the user wishes to talk. Here, the calling voice is a voice that calls out to the person with whom the user wants to talk at the other party's base, and also functions as a voice that starts a call with the other party's base. It is preferable that the greeting voice includes words used when actually having a face-to-face conversation (for example, "person's name + hello", "person's name + good evening", etc.). Thereby, when talking to a person at the other party's base through the communication device 100, it is possible to start the call without feeling uncomfortable.

The calling voice uttered by either the user 152a or 152b is received by each microphone of the microphone array 101. At this time, since the environmental sounds in the room 151 are also received by each microphone, the sound signals received by each microphone are signals in which the environmental sounds are superimposed on the audio signals uttered by the users 152a and 152b.

Analog sound signals input to each microphone of the microphone array 101 are amplified by the microphone amplifier 102, converted from analog signals to digital signals by the AD converter 103, and sent to the sound input terminal 115 of the calling processor 105 as the microphone input signal x. Input as (m, n). Note that in the microphone input signal x(m, n), m is a parameter that identifies each microphone in the microphone array 101, and n is a parameter that indicates the time series of the input signal.

When a signal starts to be input to the sound input terminal 115 of the calling processing section 105, first, a microphone input signal x (m, n) is input to the signal processing section 116.

When the person position detection unit 114 determines that a person is reflected in the video signal of the video camera 104, the signal processing unit 116 uses information regarding the position of each person (for example, direction information) from the person position detection unit 114. Then, microphone array processing is performed on the microphone input signal x(m, n), and directionality processing and sound source separation processing for separating the sound sources are performed.

In this way, when it is determined that a person is reflected in the video signal, the signal processing unit 116 performs signal processing, so that even if there are people other than the user in the room 151, which is the usage environment, The voice of the user in front of the monitor 111 can be correctly captured without the microphone receiving the voice of a person other than the user and erroneously connecting to the other party's base.

Further, when it is determined that a person is shown in the video signal, the signal processing unit 116 treats the sound as the voice of each person shown in the video signal based on the direction information of the person from the person position detection unit 114, The directivity formed in the microphone array 101 and sound source separation processing are performed. The method of detecting a person by the person position detection unit 114 is not particularly limited, and various methods can be widely applied. In order to make the X-Y coordinate system of the video signal (video frame) correspond to the X-Y coordinate system that determines the position of each microphone in the microphone array 101, Information on the direction in which the person is located may be calculated by performing coordinate transformation processing between the origin of the system and the origin of the system.

An example of a directivity processing method is a delay-sum array processing which is conventional microphone array processing, and when the microphone array 101 is a linear microphone array, processing is performed according to the following equation (1).

In the above equation (1), x'_k(n) is the microphone array processed signal, Dm is the amount of delay added to each microphone signal, K is the number forming directivity, M is the number of microphones, and D0 in equation (2) is the fixed delay amount, τ_k in equation (3) is the delay amount between the microphones, d is the microphone interval, θk is the angle forming the directivity (direction information of the person from the person position detection unit 114), and c is the speed of sound .

For example, when one directivity is formed in the front direction of the microphone array 101, K=1 and the angle forming the directivity θ1=0, so from equation (3) above, τ_1=0. Become. For example, when forming two directivity in the 90 degree direction of the microphone array 101, K=2, the angle θ2 forming the directivity=π/2 (π is pi), and the above From equation (3), τ_2=d/c.

Note that various methods can be widely applied to the signal processing calculation means, such as conventional separate microphone array processing other than delay-sum array processing, or calculation of a certain area using two microphone arrays. Microphone array processing that can collect sound may also be used.

Then, when the person position detection unit 114 determines that a person is reflected in the video signal of the video camera 104, the signal processing unit 116 transmits the calculated microphone array processed signal x′_k(n) to the audio buffer unit. 117, the voice recognition unit 118, and the output switching unit 121, and when the person position detection unit 114 determines that no person is reflected in the video signal of the video camera 104, as shown in equation (4), , the wireless signal is output to the audio buffer section 117, the speech recognition section 118, and the output switching section 121.
x'_k(n)=0...(4)

Further, the calling processing unit 105 simultaneously holds the microphone array processed signal x′_k(n) at the write position write_index of the audio buffer buffer_k(n) of the audio buffer unit 117 according to the following equation (5). After holding, the call processing unit 105 increments the value of the write position write_index by "1" and proceeds with the process, as shown in equation (6) below.

BUFFER_SIZE in the above equation (6) is the length of the buffer in the audio buffer unit 117.

Furthermore, the calling processing unit 105 simultaneously performs voice recognition on the microphone array processed signal x'_k(n) using the voice recognition unit 118. Then, the voice recognition result of the microphone array processed signal x'_k(n) is output to the command determination unit 120 for each microphone array processed signal.

The command determination unit 120 compares the voice recognition result with the command list held in the command list unit 119 (for example, the command list in FIG. 3), and selects the “person name” in the command list and the “connection command” in the command list. ” is then judged whether or not the voice has been recognized. For example, if the user utters something like "Hello Mr. XXX" and the voice recognition results show that the "person's name" and "connection command" set in the command list are consecutively recognized, the judgment will be made. If a "disconnect command" to be described later is voice recognized, "2" is output as the determination result; otherwise, "0" is output. The command determination unit 120 then outputs the determination result to the output switching unit 121 and outputs the determination result and the voice recognition result to the connection determination unit 123.

The command list section 119 stores a list of commands in the form of a text file, as shown in FIG. 3, for example. For example, the command list illustrated in FIG. 3 can be roughly divided into "person's name," which indicates at least the name of the person who can be called at the other party's base, and words used when starting a conversation with the person you actually meet. , and a "connection command" that functions as a command to start a connection with the other party's base, a word used when ending a conversation with the other party that you actually meet, and which also serves as a command to start a connection with the other party's base. It has a "disconnect command" that functions as a command to execute termination. Note that the command list in FIG. 3 is an example, and various values (formats) can be applied to the content and format of the data held by the command list section 119.

The connection determination unit 123 performs connection determination based on the voice recognition result by the voice recognition unit 118 and the command determination result based on the command determination unit 120, and outputs the connection determination result to the NW communication unit 106.

For example, if the judgment result of the command judgment unit 120 is “1” and the recognition result of the speech recognition result 118 is “Hello Mr. XXX”, the connection judgment unit 123 When "Mr. ○○" is near where the device 100 is installed, a signal for connecting to the communication device 100 at the other party's base is output to the connection determination result output terminal 124. Determination as to whether "Mr. ○○" will enter the vicinity where the communication device 100 of the base is installed uses information that has been registered in advance of people who are near the terminal.

The NW communication unit 106 performs connection processing with the network 107 based on the connection determination result outputted via the connection determination result output terminal 124.

When the command determination unit 120 recognizes the “person's name” and the “connection command” in succession, the audio buffered sound of the corresponding microphone array processed signal held in the audio buffer unit 117 is output.

In order to output the sound held in the audio buffer section 117, a read position read_index is calculated according to the following equation (7).

LEN in the above equation (7) is the length for reproducing the processed signal held in the audio buffer unit 117. Note that various methods can be widely applied to determine LEN, and for example, there is a method of setting it to a constant such as setting it to the same length as the buffer size of the audio buffer unit 117 (LEN=BUFFER_SIZE). Alternatively, a method may be used in which the microphone input signal held in the audio buffer unit 117 is subjected to voice section processing, the length of the sound held in the buffer is determined, and the length is set as LEN.

Then, the output switching unit 121 outputs the sound signal held in the audio buffer unit 117 as an output signal y(n) to the sound output terminal 122 for a certain period of time (for example, a period of LEN) as shown in equation (8) below. (length)), and the read position read_index is advanced (incremented) as shown in equation (9) below.

The NW communication unit 106 transmits the output signal y(n) outputted from the sound output terminal 122 to the NW communication unit 106 of the communication device 100 on the other side connected via the network 107.

When the output switching unit 121 outputs the sound signal held in the audio buffer unit 117 for a certain period of time, the output switching unit 121 changes the microphone array processed signal x′_k(n) to the output signal y(n ) is output to the sound output terminal 122.
y(n)=x'_k(n)...(10)

On the other hand, if the command determination unit 120 does not recognize the voice recognition result of the voice recognition unit 118 as “person name” and “connection command” consecutively, the output switching unit 121 determines that x' Since _k(n) becomes a silent signal, y(n) also becomes a silent signal as shown in equation (10), and the silent signal continues to be output to the sound output terminal 122.
y(n)=0...(11)

The NW communication unit 106 continues to transmit the output signal y(n) output through the sound output terminal 122 to the NW communication unit 106 of the communication device 100 on the other side connected to the network 107.

On the other hand, the other party's audio signal transmitted from the network 107 is input to the DA conversion unit 108 via the NW communication unit 106. After the digital signal is converted into an analog signal by the DA converter 108, the audio signal is amplified by the speaker amplifier 109, and the audio is output from the speaker 110.

After the calling voice is played back, the communication device 100 at the own location and the communication device 100 at the other party's location are connected, and video camera images and audio are exchanged between the two locations.

If the call is to be ended after a while, either user 152a or 152b utters a disconnection voice to end the conversation.

The sound emitted by either user 152a or 152b is input to each microphone of microphone array 101 with environmental sound superimposed thereon.

The analog sound signal input to the microphone array 101 is amplified by the microphone amplifier 102, converted from the analog signal to a digital signal by the AD converter 103, and the microphone input signal x_k(m , n), and the microphone input signal x_k(m, n) is input to the signal processing unit 116.

The signal processing unit 116 performs microphone array processing on the microphone input signal x_k(m, n) as shown in equations (1), (2), and (3), and performs directional processing and sound source separation to separate sound sources. The separation process is performed, and the calculated microphone array processed signal x'_k(n) is output to the audio buffer section 117, the speech recognition section 118, and the output switching section 121.

The output switching unit 121 outputs the microphone array processed signal x'_k(n) to the sound output terminal 122 as an output signal y(n), as shown in equation (10).

Further, the calling processing unit 105 simultaneously holds the microphone array processed signal x'_k(n) at the write position write_index of the audio buffer buffer_k(n) of the audio buffer unit 117 according to equation (5). After holding, the call processing unit 105 advances the write position write_index (that is, increments the write position) as shown in equation (6).

Furthermore, the calling processing section 105 simultaneously performs speech recognition on the microphone array processed signal x'_k(n) using the speech recognition section 118 and outputs the speech recognition result to the command determination section 120 .

The command determination unit 120 compares the voice recognition result with the command list held in the command list unit 119 (command list in FIG. 3), and determines whether the voice recognition result exists in the list of “disconnection commands”. Make a judgment. Then, when the “disconnection command” in the command list is voice recognized (for example, “goodbye”), the command determination unit 120 outputs the determination result to the output switching unit 121 and the connection determination unit 123. For example, if the user utters something like "Hello Mr. XXX" and the voice recognition results show that the "person's name" and "connection command" set in the command list are consecutively recognized, the judgment will be made. If the "disconnection command" is voice recognized, "2" is output as the determination result; otherwise, "0" is output.

The connection determination unit 123 makes a disconnection determination based on the voice recognition result by the voice recognition unit 118 and the command determination result based on the command determination unit 120, and connects the NW communication unit 106 with a signal to disconnect from the other party's NW communication unit. The determination result is output to the output terminal 124.

The NW communication unit 106 performs a disconnection process from the NW communication unit 106 of the communication device 100 on the other side based on the connection determination result outputted via the connection determination result output terminal 124.

When the command determination unit 120 determines that the voice recognition result of the voice recognition unit 118 does not exist in the disconnection command list of the command list unit 119, the output switching unit 121 outputs the microphone array processed signal to the sound output terminal 122. Continue to do so.

On the other hand, if the command determination unit 120 determines that the voice recognition result of the voice recognition unit 118 is present in the disconnection command list of the command list unit 119, the output switching unit 121 selects a silent mode as shown in equation (11). The signal is outputted to the sound output terminal 122 as an output signal y(n).

(A-3) Effects of First Embodiment As described above, according to the first embodiment, the communication device 100 is able to identify each user based on the audio signal received by the microphone array and the direction information of the person. The processed signal is once held in the audio buffer section, and at the same time, the speech recognition section performs speech recognition on the signal processed by the speech recognition section. Then, it is determined whether the voice recognition result is a calling voice or not, and if it is a calling voice, the calling voice is output by connecting with the communication device of the other party and outputting the calling voice held in the buffer. You can start a conversation after the other person understands what you are saying. In addition, after the conversation with the other party starts, the voice recognition unit performs voice recognition on the processed signal, determines whether the voice recognition result is disconnected voice, and disconnects if it is disconnected voice. do. This makes it possible to reproduce a situation similar to a face-to-face conversation, and allows multiple speakers to start a conversation with a high sense of realism.

Furthermore, even in a noisy usage environment, the communication device 100 of the first embodiment uses a microphone array to collect the calling voice and performs signal processing to enhance the voice. can be recognized correctly, and calls can be made even in noisy environments.

(B) Second Embodiment Next, a second embodiment of the communication device, communication program, and communication method of the present invention will be described in detail with reference to the drawings.

The second embodiment exemplifies a case where the sound output method of the communication device of the present invention is different from the first embodiment.

(B-1) Configuration of Second Embodiment FIG. 4 is a block diagram showing the configuration of a communication device 200 according to the second embodiment.

In FIG. 4, a communication device 200 according to the second embodiment includes a microphone array 101, a microphone amplifier 102, an analog-digital (AD) converter 103, two video cameras 104a and 104b, a calling processor 201, and a NW communication 106, a digital-to-analog (DA) converter 108, a speaker amplifier 109, two speakers 110a and 110b, and a monitor 111.

The calling processing unit 201 also includes a sound input terminal 115, video input terminals 112a and 112b, video output terminals 113a and 113b, a person position detection unit 202, a signal processing unit 116, an audio buffer unit 117, a voice recognition unit 118, and a command list. section 119, command determination section 120, output switching section 203, sound output terminals 122a and 122b, connection determination section 123, and connection determination result output terminal 124.

The communication device 200 according to the second embodiment includes two video cameras 104a and 104b, two speakers 110a and 110b, and further includes video input terminals 112a and 112b of the calling processing section 201, and a video output terminal. 113a and 113b and sound output terminals 122a and 122b are increased to two, the operations of the person position detection section 202 and the output switching section 203 differ from those in the first embodiment.

The other components are the same as or correspond to the components of the communication device 100 of FIG. 1 according to the first embodiment. Note that in FIG. 4, components that are the same as or correspond to the components of the communication device 100 according to the first embodiment are given the same reference numerals. Furthermore, since detailed explanations of components that are the same as or correspond to those in the first embodiment are redundant, they will be omitted here.

The calling processing unit 201 is connected to two video cameras 104a and 104b, and determines whether a person is visible in the input nesting signals from the respective video cameras 104a and 104b. Only when it is determined that a person is reflected in one or both of the video signals, the calling processing unit 201 performs signal processing on the plurality of input microphone input signals, and outputs the processed signal to the sound output terminal. . At the same time, the call processing section 201 stores the processed signal in the audio buffer section 117. Furthermore, the call processing unit 201 performs voice recognition on the signal processed signal, and when the voice recognition result matches one of the commands in the command list unit 119, the call processing unit 201 uses the connection determination result and the sound signal stored in the audio buffer as a constant. The signal is output for a certain period of time, and when the output is completed for a certain period of time, the processed signal is output again.

Next, the detailed configuration of the call processing section 201 will be explained.

The video input terminals 112a and 112b are interface units that input video signals from the video cameras 104a and 104b to the calling processing unit 201.

The video output terminals 113a and 113b are interface units that output video signals from the video cameras 104a and 104b from the calling processing unit 201.

The person position detection unit 202 determines whether or not a person is shown in the video signals of the video cameras 104a and 104b inputted from the video input terminals 112a and 112b.

The output switching unit 203 determines the sound signal to be output based on the command determination result by the command determination unit 120, and outputs the sound signal.

(B-2) Operation of Second Embodiment The basic operation of voice processing in the communication device 200 according to the second embodiment is the same as the voice processing described in the first embodiment.

Below, processing operations in the person position detection section 202 and the output switching section 203, which are different from the first embodiment, will be explained in detail.

Further, in the following, as shown in FIG. 5, it is assumed that one user 152a is communicating with a person at the other party's base, and then a second user 152b joins the communication. and explain. In this case, the description will be made assuming that the user 152a is photographed by the video camera 104a, and the user 152b is photographed by the video camera 104b.

First, when the communication device 200 starts operating, the monitor 111 displays images captured by the video cameras 104a and 104b of the communication device 100 at the other party's base.

The images captured by the video cameras 104a and 104b at the own base are given to the NW communication unit 106 via the calling processing unit 201, and the NW communication unit 106 transmits a video signal to the other party's base through the NW. The video signal is received by the NW communication unit 106 of the other party's base, and the video captured by the video cameras 104a and 104b of the own base is displayed on the monitor 111 of the other party's base.

At this time, the communication devices 200 at both bases are not transmitting or receiving audio signals, and only the images shot by the video cameras 104a and 104b of the other base are displayed on the monitor 111 at both bases, allowing the two bases to check the status of each other's bases. can. Further, audio signals from each base may be transmitted and received from each other, and in that case, each other's images are displayed on the monitor 111, and each other's sounds can be heard.

Further, video signals captured by the video cameras 104a and 104b are input to video input terminals 112a and 112b of the calling processing section 201, and then input to the person position detection section 202.

The person position detection section 202 determines whether or not a person is shown in the video signals captured by the video cameras 104a and 104b, and outputs the determination result to the signal processing section 116 and the voice recognition section 118. For example, when the person position detection unit 202 determines that a person is reflected in the video camera 104a, the determination result is set to "1," and when it determines that a person is reflected in the video camera 104b, the determination result is set to "2." Otherwise, the determination result is output as "0" or the like.

For example, when the user 152a has a call with a person who is displayed in the overall image of the other party's base, in order to view the image of the other party's base displayed on the monitor 111, as shown in FIG. The user 152a approaches the monitor 111 and checks the image of the other party's base displayed on the monitor 111.

At this time, as illustrated in FIG. 5, the video camera 104a installed near the monitor 111 photographs the user 152a, so the person position detection unit 202 that monitors the video signal of the video camera 104a A determination result that a person is shown in the video signal (for example, determination result "1", etc.) is output to the signal processing unit 116 and the voice recognition unit 118. Further, the person position detection unit 202 outputs direction information of the user 152a in the video frame of the video camera 104a to the signal processing unit 116.

The user 152a utters a calling voice in order to call the person with whom the user wants to talk. The sound emitted by the user 152a is superimposed with environmental sounds and input to each microphone of the microphone array 101a.

The analog sound signal input to the microphone array 101a is amplified by the microphone amplifier 102, converted from the analog signal to a digital signal by the AD converter 103, and the microphone input signal x(m , n).

When a signal starts to be input to the sound input terminal 115 of the calling processing section 201, first, a microphone input signal x (m, n) is input to the signal processing section 116.

When the person position detection unit 202 determines that a person is reflected in the video signal of the video camera 104a, the signal processing unit 116 performs microphone array processing on the input signal. At this time, the signal processing unit 116 detects the voice of the user 152a coming from the direction of the user 152a's position based on the direction information of the user 152a in the image of the video camera 104a acquired from the person position detection unit 202. The array 101 performs directional processing for collecting sound and sound source separation processing for extracting the voice of the user 152a.

Then, when the person position detection unit 202 determines that a person is reflected in the video signal of the video camera 104, the signal processing unit 116 transmits the calculated microphone array processed signal x′_1(n) to the audio buffer unit. 117, the voice recognition unit 118, and the output switching unit 203, and when the person position detection unit 202 determines that no person is reflected in the video signal of the video camera 104, as shown in equation (12), , the wireless signal is output to the audio buffer section 117 and the speech recognition section 118, and the wireless signal is output to the output switching section 203.
x'_1(n)=0...(12)

At the same time, the calling processing unit 201 holds the microphone array processed signal x'_1(n) at the write position write_index of the audio buffer buffer_1(n) of the audio buffer unit 117 according to equation (5). After holding, the call processing unit 201 increments and advances the value of the write position write_index, as shown in equation (6).

Furthermore, the calling processing unit 201 simultaneously inputs the microphone array processed signal x'_1(n) to the voice recognition unit 118, and the voice recognition unit 118 performs voice recognition to generate the voice of the microphone array processed signal x'_1(n). The recognition result is output to the command determination unit 120.

The command determination unit 120 compares the voice recognition result of the microphone array processed signal x'_1(n) with the command list held in the command list unit 119 (for example, the command list in FIG. 3), and It is determined whether or not the "person's name" and the "connection command" in the command list have been voice recognized in succession (for example, "Hello Mr. XXX").

Then, the command determination unit 120 outputs the determination result to the output switching unit 203 and outputs the determination result and the voice recognition result to the connection determination unit 123. For example, if the voice recognition result of the microphone array processed signal x'_1(n) is that "person's name" and "connection command" are consecutively recognized, the command determination unit 120 sets the determination result to "1" and the microphone If the speech recognition result of the array processing signal x'_2(n) is a sequence of "person's name" and "connection command", the judgment result is output as "2", otherwise it is output as "0", etc. .

The connection determination unit 123 performs connection determination based on the voice recognition result by the voice recognition unit 118 and the command determination result based on the command determination unit 120, and outputs the connection determination result to the NW communication unit 106. For example, if the determination result is "1" and the command determination unit 120 outputs a voice recognition result of "Hello Mr. XXX," the connection determination unit 123 determines that the communication device 100 of the other party's base is installed. When "Mr. ○○" is nearby, the video camera 104a and the microphone array 101a are connected to the communication device 200 of the other party's base, which connects the signal to be connected to the video camera and microphone array near "Mr. ○○" as a connection determination result. It is output to the output terminal 124. To determine whether or not Mr. ○○ will enter the vicinity where the communication device 200 of the base is installed, information that has been registered in advance about people near the terminal is used.

The NW communication unit 106 performs connection processing with the network 107 based on the connection determination result outputted via the connection determination result output terminal 124.

If the voice recognition result of the voice recognition unit 118 in the command determination unit 120 is that “person name” and “connection command” are not recognized consecutively, the output switching unit 203 outputs a silent signal as shown in equation (13). It is output to the sound output terminal 122a as an output signal y_1(n).
y_1(n)=0...(13)

On the other hand, when the command determination unit 120 successively recognizes the “person name” and the “connection command”, the output switching unit 203 calculates the read position read_index_1 of the audio buffer unit 117 according to equation (14) below. do.

Further, if the "person's name" and "connection command" are not voice recognized consecutively, the output switching unit 203 may output the microphone input signal x(m,n) from the signal processing unit 116.

Then, the output switching unit 203 outputs the sound signal held in the audio buffer unit 117 as an output signal y_1(n) to the sound output terminal 122 for a certain period of time (for example, the time of LEN) as shown in equation (15) below. (length)), and the read position read_index_1 is incremented and advanced as shown in equation (16) below.

The NW communication unit 106 transmits the output signal y_1(n) output from the sound output terminal 122a to the NW communication unit 106 of the communication device 200 on the other side connected via the network 107.

When the output switching unit 203 outputs the sound signal held in the audio buffer unit 117 for a certain period of time, the output switching unit 203 changes the microphone array processed signal x'_1(n) to the output signal y_1( n) is output to the sound output terminal 122a.
y_1(n)=x'_1(n)...(17)

The NW communication unit 106 continues to transmit the output signal y_1(n) output from the sound output terminal 122 to the NW communication unit 106 of the communication device 200 on the other side connected via the network 107.

On the other hand, the audio signal of the other party transmitted from the network 107 is input to the DA converter 108 via the NW communication unit 106, and after being converted from a digital signal to an analog signal by the DA converter 108, the audio signal is sent to the speaker amplifier. 109, and the sound is output by speaker 110a. In other words, the user's 152a's voice is output from the speaker 110a.

After the calling voice is played back, the communication device 200 at the own location and the communication device 200 at the other party's location are connected, and video camera images and audio are exchanged between the two locations.

Next, as shown in FIG. 6, a case where the second user 152b participates in communication and speaks with a person at the other party's base will be described.

In this case as well, in order for the user 152b to view the image of the other party's base displayed on the monitor 111, the user 152b approaches the monitor 111 and views the other party's base displayed on the monitor 111, as shown in FIG. Check the video of the base. Then, the person position detection unit 114 determines that a person is captured on the video camera 104b, and outputs the determination result to that effect to the signal processing unit 116 and the voice recognition unit 118.

When the user 152b sees the person he or she wants to talk to in the video, the user 152b utters a calling voice. The voice emitted by the user 152b is superimposed with environmental sounds and input to each microphone of the microphone array 101b.

The analog sound signal input to the microphone array 101b is amplified by the microphone amplifier 102, the analog signal is converted to a digital signal by the AD converter 103, and the audio signal is input to the sound input terminal 115 of the calling processor 201 by the microphone. It is input as a signal x(m,n).

When a signal starts to be input to the sound input terminal 115 of the calling processing section 201, first, a microphone input signal x (m, n) is input to the signal processing section 116.

When the person position detection unit 202 determines that a person is shown in the image of the video camera 104b, the signal processing unit 116 performs microphone array processing on the input signal, and performs directional processing and sound source separation processing to separate the sound source. do.

Then, the signal processing section 116 outputs the calculated microphone array processed signal x'_2(n) to the audio buffer section 117, the speech recognition section 118, and the output switching section 203.

Then, when the person position detection unit 202 determines that a person is reflected in the video signal of the video camera 104b, the signal processing unit 116 transmits the calculated microphone array processed signal x′_2(n) to the audio buffer unit. 117, the voice recognition unit 118, and the output switching unit 203, and when the person position detection unit 202 determines that no person is reflected in the video signal of the video camera 104, as shown in equation (18), , the wireless signal is output to the audio buffer section 117 and the speech recognition section 118, and the wireless signal is output to the output switching section 203.
x'_2(n)=0...(18)

At the same time, the calling processing unit 201 holds the microphone array processed signal x'_2(n) at the write position write_index of the audio buffer buffer_2(n) of the audio buffer unit 117 according to equation (5). After holding, the call processing unit 201 increments the value of the write position write_index by "1" and proceeds as shown in equation (6).

Furthermore, the calling processing unit 201 simultaneously inputs the microphone array processed signal x'_2(n) to the voice recognition unit 118, and the voice recognition unit 118 performs voice recognition to generate the voice of the microphone array processed signal x'_2(n). The recognition result is output to the command determination unit 120.

The command determination unit 120 compares the voice recognition result of the microphone array processed signal x'_2(n) with the command list held in the command list unit 119 (for example, the command list in FIG. 3), and compares the " A determination is made as to whether or not the words ``person's name'' and ``connection command'' in the command list have been voice-recognized (for example, ``Hello Mr. ___''). Then, the command determination unit 120 outputs the determination result to the output switching unit 121 and outputs the determination result and the voice recognition result to the connection determination unit 123.

The connection determination unit 123 performs connection determination based on the voice recognition result by the voice recognition unit 118 and the command determination result based on the command determination unit 120, and outputs the connection determination result to the NW communication unit 106.

For example, if the voice recognition result by the voice recognition unit 118 is “2” and the command determination unit 120 outputs the voice recognition result “Hello, Mr. XXX”, the connection determination unit 123 If there is "Mr. XX" near where 200 is installed, connect the video camera 104b and microphone array 101b to the unconnected video camera and microphone array of Mr. ○○ of the communication device 100 at the other party's base. A signal is output to the connection determination result output terminal 124. To determine whether "Mr. XX" will be near where the communication device 200 of the base is installed, information that has been registered in advance about people near the terminal is used.

The NW communication unit 106 performs connection processing with the network 107 based on the connection determination result output from the connection determination result output terminal 124.

If the result of voice recognition of the microphone array processed signal x'_2(n) of the voice recognition unit 118 by the command determination unit 120 is that "person's name" and "connection command" are not recognized consecutively, the output switching unit 203 selects ( As shown in equation 19), the silent signal continues to be output as the output signal y_2(n) to the sound output terminal 122b.
y_2(n)=0...(19)

On the other hand, when the command determination unit 120 successively recognizes the “person name” and the “connection command”, the output switching unit 203 calculates the read position read_index_2 of the audio buffer unit 117 according to equation (20) below. do.

Then, the output switching unit 203 outputs the sound signal held in the audio buffer unit 117 as an output signal y_2(n) to the sound output terminal 122b for a certain period of time (for example, LEN) as shown in equation (21) below. time length) and increments the value of the read position read_index_2 by "1" as shown in equation (22) below.

The NW communication unit 106 transmits the output signal y_2(n) output from the sound output terminal 122 to the NW communication unit 106 of the other party connected via the network 107.

When the output switching unit 203 outputs the sound signal held in the audio buffer unit 117 for a certain period of time, the output switching unit 203 changes the microphone array processed signal x′_2(n) to the output signal y_2(n ) is output to the sound output terminal 122b.
y_2(n)=x'_2(n)...(23)

The NW communication unit 106 transmits the output signal y_2(n) output via the sound output terminal 122 to the NW communication unit 106 of the other party connected via the network 107.

On the other hand, the voice of the other party transmitted from the network 107 is input to the DA converter 108 via the NW communication unit 106, and after being converted from a digital signal to an analog signal by the DA converter 108, the voice signal is sent to the speaker amplifier 109. The sound is amplified by the speaker 110b, and the sound is output from the speaker 110b. In other words, the user's 152b's voice is output from the speaker 110b.

After the calling voice is played back, the remote communication device exchanges the video camera image and voice via the NW communication unit 106 after being connected.

(B-3) Effects of the second embodiment As described above, according to the second embodiment, the communication device uses multiple microphone arrays to separately emphasize the voices of multiple speakers. Perform signal processing. Then, the processed signal is once held in the audio buffer section, and at the same time, speech recognition is performed on the signal processed signal, and it is determined for each microphone array signal whether or not the speech recognition result is a calling speech. In the case of a calling voice, by outputting the calling voice held in the audio buffer section after connecting to the other party, it is possible to start a conversation after the calling voice is transmitted to the other party. In addition, when ending a call, voice recognition is performed on the processed signal, and it is determined whether the voice recognition result is a disconnection voice. If it is a disconnection voice, the connection with the other party's base is established. disconnect. This makes it possible to reproduce a situation similar to a face-to-face conversation, and allows multiple speakers to start a conversation with a high sense of realism.

(C) Other Embodiments Although various modified embodiments have been described in each of the above-described embodiments, the present invention can also be applied to the following modified embodiments.

(C-1) The communication device described in each of the above-described embodiments may be installed in a device that starts a call using a voice input command, for example, when starting a call in a conference call.

(C-2) The call processing unit and the NW communication unit in the communication device described in each of the above-described embodiments may be processed by a processing device (for example, a server, etc.) provided on the network.

(C-3) In the communication devices described in each of the above-described embodiments, the microphone array 101 is arranged in front of the monitor 111, as illustrated in FIGS. 2, 5, and 6. However, the arrangement examples of the microphone array 101 are not limited to those shown in FIGS. 2, 5, and 6. For example, the microphone array 101 may be placed on the top or side of the monitor 111. Furthermore, if the communication device includes a projector and a screen, the monitor 111 may be replaced with a screen for forming a projected image from the projector. Various types of screens can be used, such as a normal screen that forms a projected image, or a screen that transmits sound. In the case of a screen that transmits sound, the microphone array 101 may be placed behind the screen.

(C-4) In the communication device described in each of the embodiments described above, one microphone array 101 is provided, but two microphone arrays may be provided. Here, for example, microphone arrays 101a and 101b are used. In that case, for example, the microphone array 101a shall collect the voice of the user 152a, and the microphone array 101b shall collect the voice of the user 152b.

100 and 200... communication device, 101... microphone array, 102... microphone amplifier, 103... AD converter, 104, 104a and 104b... video camera, 105 and 201... call processing unit, 106... NW communication unit, 107... network, 108...DA converter, 109...Speaker amplifier, 110a and 110b...Speaker, 111...Monitor, 112, 112a and 112b...Video input terminal, 113, 113a and 113b...Video output terminal, 114...Person position detection unit, 115... Sound input terminal, 116...Signal processing section, 117...Audio buffer section, 118...Speech recognition section, 119...Command list section, 120...Command judgment section, 121 and 203...Output switching section, 122, 122a and 122b...Sound output Terminal, 123...Connection determination unit, 124...Connection determination result output terminal.

Claims (5)

a person detection unit that detects one or more people from an input video signal and acquires information regarding the position of each detected person;
a signal processing unit that forms the directivity of one or more microphones based on the information regarding the position of each person from the person detection unit and extracts the audio signal of each person;
a holding unit that is a buffer for causing the other party to reproduce a connection command voice transmitted after connection with the other party, and retains the voice signal of each person including the connection command voice produced by the signal processing unit for a certain period of time ;
a voice recognition unit that performs voice recognition based on an input voice signal when the one or more people are detected by the person detection unit;
a command storage unit that stores a plurality of commands including at least a connection command for starting a connection with a connection destination and a disconnection command for disconnecting the connection;
a command determination unit that determines whether the voice recognition result by the voice recognition unit includes the connection command or the disconnection command stored in the command storage unit;
an output switching unit that determines an output audio signal according to a command determination result by the command determination unit;
a connection determination unit that performs connection processing with a connection destination on the other side based on the voice recognition result and the command determination result;
When the above voice recognition result includes the above connection command,
The connection determination unit performs connection processing with the connection destination of the other party,
The output switching unit outputs the audio signal containing the connection command held in the holding unit after connecting with the connection destination of the other party, and then outputs the signal processed by the signal processing unit. A communication device featuring: When the above voice recognition result includes the above disconnection command,
2. The connection determining section disconnects the connection to the connection destination after the output switching section outputs the audio signal including the disconnection command held in the holding section. communication device. Video signals are input from each of multiple video cameras,
The person detection unit detects a person from video signals from each of the plurality of video cameras,
The signal processing section forms the directivity of the audio signal picked up by the one or more microphones based on information regarding the position of each person for each of the video signals detected by the person detection section. , extract the audio signals of each person above,
The communication device according to claim 1 or 2, wherein the output switching unit separately outputs the audio signals of each person extracted by the signal processing unit. computer,
a person detection unit that detects one or more people from an input video signal and acquires information regarding the position of each detected person;
a signal processing unit that forms the directivity of one or more microphones based on the information regarding the position of each person from the person detection unit and extracts the audio signal of each person;
a holding unit that is a buffer for causing the other party to reproduce a connection command voice transmitted after connection with the other party, and retains the voice signal of each person including the connection command voice produced by the signal processing unit for a certain period of time ;
a voice recognition unit that performs voice recognition based on an input voice signal when the one or more people are detected by the person detection unit;
a command storage unit that stores a plurality of commands including at least a connection command for starting a connection with a connection destination and a disconnection command for disconnecting the connection;
a command determination unit that determines whether the voice recognition result by the voice recognition unit includes the connection command or the disconnection command stored in the command storage unit;
an output switching unit that determines an output audio signal according to a command determination result by the command determination unit;
Function as a connection determination unit that performs connection processing with the other party's connection destination based on the voice recognition result and the command determination result,
When the above voice recognition result includes the above connection command,
The connection determination unit performs connection processing with the connection destination of the other party,
The output switching unit outputs the audio signal containing the connection command held in the holding unit after connecting with the connection destination of the other party, and then outputs the signal processed by the signal processing unit. A communication program featuring The person detection unit detects one or more people from the input video signal and acquires information regarding the position of each detected person,
a signal processing unit forms directivity of one or more microphones based on information regarding the position of each person from the person detection unit, and extracts an audio signal of each person;
The holding unit is a buffer for causing the other party to reproduce the connection command voice transmitted after connection with the other party, and retains the voice signal of each person including the connection command voice by the signal processing unit for a certain period of time ;
a voice recognition unit performs voice recognition based on the input voice signal when the one or more people are detected by the person detection unit;
a command storage unit stores a plurality of commands including at least a connection command for starting a connection with a connection destination and a disconnection command for disconnecting the connection;
a command determination unit determines whether the voice recognition result by the voice recognition unit includes the connection command or the disconnection command stored in the command storage unit;
an output switching unit determines an output audio signal according to a command determination result by the command determination unit;
The connection determination unit performs connection processing with the other party's connection destination based on the voice recognition result and the command determination result,
When the above voice recognition result includes the above connection command,
The connection determination unit performs connection processing with the connection destination of the other party,
The output switching unit outputs the audio signal containing the connection command held in the holding unit after connecting with the connection destination of the other party, and then outputs the signal processed by the signal processing unit. A communication method characterized by

Images