JP2020088618A - Video conference system, communication terminal, and method for controlling microphone of communication terminal - Google Patents

Abstract

To appropriately determine the directivity for collecting sound of a microphone.SOLUTION: A video conference system according to one aspect of a disclosed technique includes a plurality of communication terminals, a camera for outputting an image to the communication terminals, and a microphone for outputting voice to the communication terminals, and performs a video conference. The communication terminal includes: a line-of-sight information storage unit that stores light-of-sight information indicating lines of sight of participants in the video conference detected from the image; and a directivity determination unit that determines the directivity for collecting sound of the microphone based on the stored line-of-sight information.SELECTED DRAWING: Figure 3

Description

The present application relates to a video conference system, a communication terminal, and a method of controlling a microphone of the communication terminal.

2. Description of the Related Art Video conferencing systems that connect two or more remote locations to hold a two-way video and audio conference have become popular. In addition, by processing the output voices of multiple microphones to increase the sensitivity of sound collection in a predetermined direction (controlling the directivity of the sound collection), the speaker's voice is collected with high sensitivity, and There is known a beamforming technique that suppresses the collection of unnecessary sound.

On the other hand, in the video conference system, the directivity of the sound collection of the microphone is determined based on the area (direction) in which the sound is generated in any one of the communication terminals arranged in each base for executing the video conference. A technique for determining is disclosed (for example, see Patent Document 1).

However, in the technique of Patent Document 1, when a loud sound is generated around the participants of the video conference, the directivity of the microphone sound collection may be erroneously determined in the direction of the sound source.

The present invention has been made in view of the above points, and an object of the present invention is to appropriately determine the directivity of sound collection of a microphone.

A video conference system according to an aspect of the disclosed technology includes a plurality of communication terminals, a camera that outputs an image to the communication terminals, and a microphone that outputs audio to the communication terminals, and a video that performs a video conference. In the conference system, the communication terminal stores a line-of-sight information storage unit that stores line-of-sight information indicating a line-of-sight of a participant of the video conference detected from the image, and a collection of the microphones based on the stored line-of-sight information. And a directivity determining unit that determines the directivity of the sound.

According to the embodiment of the present invention, it is possible to appropriately determine the directivity of the sound collection of the microphone.

It is a figure explaining an example of composition of a video conference system concerning an embodiment. It is a block diagram which shows an example of the hardware constitutions of the communication terminal which concerns on embodiment. It is a block diagram which shows an example of a functional structure of the communication terminal which concerns on 1st Embodiment. It is a figure explaining an example of a participant's line of sight. It is a figure explaining an example of the line-of-sight information which concerns on 1st Embodiment. It is a figure explaining an example of a gaze change according to time concerning a 1st embodiment. It is a figure explaining an example of the attention area|region which concerns on 1st Embodiment. It is a figure explaining an example of the information contained in the subpacket which concerns on 1st Embodiment. It is a figure explaining the directivity of the sound collection of a microphone, (a) is a figure explaining the sound collection direction of beamforming, (b) is a figure explaining the correspondence of attention area information and a sound collection direction. Is. 6 is a flowchart illustrating an example of a region-of-interest detection process according to the first embodiment. It is a flow chart which shows an example of directivity control processing concerning a 1st embodiment. It is a sequence diagram which shows an example of operation|movement of the video conference system which concerns on 1st Embodiment. It is a figure explaining an example of an effect of a video conference system concerning a 1st embodiment, (a) is a figure explaining a video conference using a communication terminal concerning a comparative example, and (b) is a 1st implementation. It is a figure explaining the video conference using the communication terminal which concerns on a form. It is a block diagram which shows an example of a functional structure of the communication terminal which concerns on 2nd Embodiment. It is a sequence diagram which shows an example of operation|movement of the video conference system which concerns on 2nd Embodiment. It is a block diagram showing an example of functional composition of a communication terminal concerning a 3rd embodiment. It is a figure explaining an example of the input screen which concerns on 3rd Embodiment.

Hereinafter, embodiments for carrying out the invention will be described with reference to the drawings. In each drawing, the same reference numerals are given to the same components, and duplicate description may be omitted.

In the embodiment, the communication terminal 2A or 2B installed in the base A may be described as an example, but the communication terminal 2 installed in any base is the same as the communication terminal 2A or 2B according to the description. It has a function and can execute the same operation.

<Structure of Video Conferencing System According to Embodiment>
FIG. 1 is a diagram illustrating an example of a configuration of a video conference system according to an embodiment. As shown in FIG. 1, the communication terminal 2A is installed at the site A and the communication terminal 2B is installed at the site B. Communication terminals 2A and 2B (hereinafter, referred to as communication terminal 2 if not distinguished) are connected to a server 4 via a network 3 such as the Internet or a LAN (Local Area Network). However, the number is not limited to this, and the number of communication terminals 2 (the number of bases) included in the video conference system 1 can be arbitrarily changed.

The server 4 monitors whether or not each communication terminal 2 is connected to the server 4, and performs control necessary for a video conference such as call control of the communication terminals 2A and 2B at the start of the video conference.

The communication terminals 2A and 2B at the time of the video conference transmit at least one of image data and audio data (hereinafter referred to as image/audio data) to the server 4 at the time of data transmission of the own terminal, and the server 4 The image/voice data is transmitted to the other communication terminal 2 on the partner side.

On the other hand, when receiving data, the image/voice data of the other communication terminal 2 on the partner side is received via the server 4. For example, when a video conference is held between the base A and the base B, the data transmitted by the communication terminal 2A of the base A is transmitted to the communication terminal 2B of the base B via the server 4, and is transmitted to the other communication terminal 2 (for the video conference). It is not transmitted to the communication terminal 2) which has not participated.

Similarly, the image/audio data transmitted by the communication terminal 2B of the base B is transmitted to the communication terminal 2A of the base A participating in the video conference via the server 4 and other communication not participating in the conference. It is not transmitted to the terminal 2. By performing such control, it is possible to hold a video conference with a plurality of communication terminals 2 (between a plurality of bases).

Note that the configuration of the video conference system 1 shown in FIG. 1 is an example and may have another configuration.

The communication terminal 2 is a device having a communication function, such as a PJ (Projector: projector), an image forming device, an IWB (Interactive White Board: a white board having an electronic blackboard function capable of mutual communication), a digital device. Output device such as signage, HUD (Head Up Display) device, industrial machine, imaging device, sound collecting device, medical device, network home appliance, notebook PC (Personal Computer), mobile phone, smartphone, tablet terminal, game machine, PDA ( It may be a Personal Digital Assistant), a digital camera, a wearable PC, a desktop PC, or the like.

<Hardware Configuration of Communication Terminal According to Embodiment>
Next, the hardware configuration of the communication terminal according to the embodiment will be described. FIG. 2 is a diagram illustrating an example of the hardware configuration of the communication terminal 2. The communication terminal 2 is an IWB as an example.

The communication terminal 2 includes a CPU (Central Proccesing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an SSD (Solid State Drive) 204, a network controller 205, and a sensor controller 206. , And a capture device 207. The communication terminal 2 also includes an electronic pen controller 208, an external storage controller 209, a GPU 210, a display controller 211, a camera controller 212, a microphone controller 213, and a speaker controller 214. These are connected to each other via a bus B.

The CPU 201 is an arithmetic device that realizes control and functions of the entire communication terminal 2 by reading programs and data from a storage device such as the ROM 202 and SSD 204 onto the RAM 203 and executing processing.

The ROM 202 is a non-volatile semiconductor memory (storage device) that can retain programs and data even when the power is turned off. The ROM 202 stores programs and data such as a BIOS (Basic Input/Output System) executed when the communication terminal 2 is activated, OS settings, and network settings. The RAM 203 is a volatile semiconductor memory (storage device) that temporarily holds programs and data.

The SSD 204 is a non-volatile memory that stores an application for the IWB coordinate detection system and various data. The application for the coordinate detection system may be acquired while being stored in the external memory 216, or may be downloaded from a server or the like via the network controller 205. The network controller 205 can execute processing based on a communication protocol when communicating with a server or the like via the network 3 (see FIG. 1). The SSD 204 may be an HDD (Hard Disk Drive).

The sensor controller 206 can execute coordinate detection processing of a contact position when the electronic pen 215 and a finger touch the display 217 included in the IWB. The capture device 207 is electrically connected to the PC 221 and can capture an image or video displayed on the display device included in the PC 221. The electronic pen controller 208 is electrically connected to the electronic pen 215, which is an input device, and can control the generation of ultrasonic waves, infrared rays, and the like by the electronic pen 215.

The external storage controller 209 can write to the removable external memory 216 or read from the external memo 216. The external memory 216 is a USB (Universal Serial Bus) memory, an SD card, or the like.

A GPU (Graphics Processing Unit) 210 is a drawing-only processor that calculates the pixel value of each pixel of the display 217. The display controller 211 is electrically connected to the display 217 and can output the image generated by the GPU 210 to the display 217. The display 217 can display the image of the video conference participant transmitted via the network 3 from the other communication terminal 2 that executes the video conference.

The camera controller 212 is electrically connected to the camera 218 and can control imaging by the camera 218. As will be described later, the camera 218 is used to capture an image of the participant during the video conference, and the line of sight of the participant is detected from the image captured by the camera 218. The camera controller 212 can output the image data captured by the camera 218 to the CPU 201. An independent line-of-sight sensor module may be used instead of the camera for detecting the line-of-sight of the participant.

The microphone controller 213 is electrically connected to the microphone 219 and can control the high sensitivity and directivity of the sound collection of the microphone 219. The directivity of the microphone 219 will be described in detail later.

The speaker controller 214 is electrically connected to the speaker 220 and can control the volume of sound generated by the speaker 220 and the like.

<Functional configuration of communication terminal according to first embodiment>
Next, the video conference system according to the first embodiment will be described. First, the functional configuration of the communication terminal according to the first embodiment will be described. FIG. 3 is a block diagram illustrating an example of the functional configuration of the communication terminal according to the present embodiment.

The communication terminal 2A includes an imaging unit 21, a line-of-sight detection unit 22, a line-of-sight information storage unit 23, an attention area detection unit 24, a subpacket generation unit 25, a transmission unit 26, a reception unit 27, and a directivity determination. It has a unit 28 and a directivity control unit 29.

The imaging unit 21 has a function of capturing an image of a participant at the site A participating in the video conference and outputting the captured image information to the line-of-sight detection unit 22. The image of the participant includes at least the eyes of the participant in order to detect the line of sight of the participant.

The line-of-sight detection unit 22 has a function of detecting the line of sight of the participant from the image information input from the image pickup unit 21 and outputting the line of sight to the line-of-sight information storage unit 23. Further, the line-of-sight information storage unit 23 has a function of storing the line-of-sight information input from the line-of-sight detection unit 22.

Here, FIG. 4 is a diagram illustrating an example of the line of sight of the participant. FIG. 4 shows that the participant 101 at the site A is observing the display 217 on which the images of the participants 102a to 102d at the site B are displayed.

For example, when the participant 102a speaks at the site B, the participant 101 looks at the area where the participant 102a is displayed on the display 217, and when the participant 102d speaks, the participant 101 displays on the display. At 217, the line of sight is turned to the area where the participant 102d is displayed. The line-of-sight information of the participant is information indicating the region where the line-of-sight of the participant 101 is directed on the display 217 as described above.

The line-of-sight detection unit 22 can detect the line-of-sight information by performing image processing on the image of the participant captured by the image capturing unit 21 at the site A. The line-of-sight detection unit 22 can execute, for example, a process of extracting an image region corresponding to a black eye of a person's eye from an image and calculating barycentric position coordinates of the black eye as line-of-sight information as image processing. Since a known technique can be applied to the image processing method for detecting the line of sight, detailed description thereof will be omitted here.

FIG. 5 is a diagram illustrating an example of the line-of-sight information detected by the line-of-sight detection unit 22 and stored by the line-of-sight information storage unit 23. The line-of-sight detection unit 22 can detect the line-of-sight of the participant, which changes with time, and sequentially output the lines of sight to the line-of-sight information storage unit 23. Further, as shown in FIG. 5, the line-of-sight information storage unit 23 can store the barycentric position coordinates indicating the line-of-sight information at each time.

Here, FIG. 6 is a diagram illustrating an example of a change in the line of sight according to time. FIG. 6 shows the line-of-sight information distribution 61 that represents the locus of the line of sight on the display 217.

When there are a plurality of participants in the video conference at the site A, the line-of-sight detection unit 22 detects the line-of-sight information of each person, and the line-of-sight information storage unit 23 can store the line-of-sight information for the number of people.

Returning to FIG. 3, the description of each functional unit will be continued.

The attention area detection unit 24 detects the attention area in which the participant is paying attention at the site A based on the line-of-sight information accumulated by the line-of-sight information accumulation unit 23, and outputs the detected attention region information to the subpacket generation unit 25. Have a function.

For example, the attention area detection unit 24 can detect, as a attention area, an area in which the eye gaze is concentrated for a predetermined time threshold or more with a time when the eye gaze is concentrated as a threshold. Specifically, when the time threshold is set to 2 seconds and the detection time interval of the line-of-sight detection is set to 10 milliseconds, when the line of sight is directed to the same region in 20 or more detections, the attention area detection unit 24 A region can be detected as a region of interest.

The "area" is predetermined as a unit area for detecting the attention area. As the unit area, a 1×1 pixel area, a 10×10 pixel area, or the like can be arbitrarily set. By setting the time as the threshold value, the region gazed by the participant can be detected as the attention region.

Further, the attention area detection unit 24 can detect the area where the gaze is concentrated more than a predetermined number of times threshold as the attention area, with the number of times the eyes are concentrated as a threshold. The difference from the case where the time is set as the threshold value is that when the same region is continuously acquired by a plurality of detections, it is counted once regardless of the concentrated time.

Specifically, when the threshold value of the number of times is 10 and the detection time interval of the line-of-sight detection is 10 milliseconds, when the same area is detected 10 times discontinuously, the attention area detection unit 24 detects this area. It can be detected as a region of interest. By setting the number of times to the threshold value, it becomes possible to appropriately detect the attention area even if the line of sight is occasionally missed when attention is paid.

Further, the attention area detection unit 24 can also detect, as a attention area, an area in which the number of participants whose eye gaze is concentrated is a threshold value and the number of participants whose eye gaze is equal to or more than a predetermined number of people eyes is concentrated. When there are a plurality of participants in the video conference at the location A, the line-of-sight information distribution 61 (FIG. 6) is obtained for the number of people, but the attention area detection unit 24 is an area in which the eyes of a number of participants equal to or greater than the threshold are concentrated. Can be detected as a region of interest.

By setting the number of people as the threshold value, it is possible to appropriately detect the attention area from the line-of-sight information of other participants even if there are participants who are looking away.

The threshold value for detecting the attention area may be used in combination with the time, the number of times, and the number of people. By combining them, the detection accuracy of the attention area can be further improved.

FIG. 7 is a diagram illustrating an example of the attention area. In the line-of-sight information distribution 61, areas 62a to 62d surrounded by broken lines are detected as attention areas.

Returning to FIG. 3, the description of each functional unit will be continued.

The subpacket generation unit 25 has a function of receiving the attention area information detected by the attention area detection unit 24, generating a subpacket for transmission to another base such as the base B, and outputting the subpacket to the transmission unit 26. Further, the transmission unit 26 has a function of including the input subpacket in the conference data such as the video and audio of the participant and the synchronization data, and transmitting the subpacket to another base. Note that the communication terminal 2B installed at another site to which the transmission unit 26 transmits the subpacket is an example of “another first communication terminal”. The attention area information detected by the attention area detection unit 24 is an example of “predetermined coordinate information”.

Here, FIG. 8 is a diagram illustrating an example of information included in a subpacket. In FIG. 8, the X and Y coordinates of the attention area are shown in a list. When the unit area is composed of a plurality of pixels (10×10 pixels or the like), the center coordinates of the unit area or the like becomes the coordinate information of the attention area.

Here, the subpacket generation unit 25 may generate the subpacket from some coordinate information instead of all coordinate information of the attention area. As some of the coordinates, at least one of the maximum value of the X coordinate, the minimum value of the X coordinate, the maximum value of the Y coordinate, the minimum value of the Y coordinate, and the like can be given.

The determination of the directivity of the microphone 219 (details will be described later) has a large effect on the end region of the region of interest, so even if a subpacket is generated from the end data of the maximum value and the minimum value. Good. Further, since the X coordinate has a greater influence on the determination of the directivity than the Y coordinate, the subpacket may be generated from the X coordinate. By thus generating the subpacket from the coordinate information of a part of the attention area, it is possible to reduce the amount of data to be transmitted, reduce the communication load, and increase the communication speed.

On the other hand, the display having the same number of pixels and/or the same screen size is not always used at another site where the video conference is held. Therefore, due to this difference, it may not be possible to appropriately determine the directivity of the microphone 219 based on the attention area information.

Therefore, the subpacket generation unit 25 may include the model identification number of the product in the subpacket together with the attention area information. The directivity of the microphone 219 can be appropriately determined by correcting the influence of the difference in the number of display pixels and/or the screen size on the receiving side.

Here, the directivity of the sound collection of the microphone 219 will be described. In the present embodiment, the beamforming technique can be used to control the directivity of the sound collection of the microphone 219. The beam forming of a microphone (hereinafter, simply referred to as beam forming) is a technique of using a plurality of microphones and increasing the directivity of sound waves in a predetermined direction.

Specifically, the sound output from a plurality of microphones is subjected to signal processing so that the sensitivity of sound collection is high in a predetermined direction and low in other directions. Thus, in the video conference, the sound of the speaker can be made easy to hear by increasing the sound collection sensitivity, and unnecessary sounds around can be made hard to hear by lowering the sound collection sensitivity.

FIG. 9 is a diagram illustrating the directivity of the sound collection of the microphone. (A) is a figure explaining the sound collection direction of beamforming, (b) is a figure explaining the correspondence of attention area information and a sound collection direction.

In the present embodiment, as shown in FIG. 9A, the direction perpendicular to the display surface of the display 217 is 0 degrees of the sound collecting direction, the direction rotating to the left side in the drawing is the positive sound collecting direction, and the right side is the right direction. The direction of rotation is the negative sound collection direction. Further, when the direction corresponding to the ±90 degree direction (horizontal direction) in FIG. 9A is the X direction of the image by the image pickup unit 21, the attention area information and the sound collection direction are shown in FIG. 6B. Correspondence.

With reference to the correspondence relationship in FIG. 6B, the angle indicating the maximum value and the minimum value in the sound collection direction of the microphone, that is, the directivity can be set based on the X coordinate of the attention area information. For example, when the maximum value of the X coordinate of the attention area information is 280 pixels and the minimum value is 240 pixels, the directivity is set to the minimum value 0 degree to the maximum value 25 degrees, and the participants 101a and 101b located in this direction have directivity. The sound collection sensitivity for sounds emitted increases, and conversely, the sound collection sensitivity for sounds emitted by the participants 101c and 101d decreases. In this way, the directivity is determined according to the attention area. The data indicating the correspondence relationship in FIG. 9B is determined in advance and stored in the memory such as the SSD 204.

Since the Y coordinate of the attention area information corresponds to the distance from the display to the participant, when the distance to the participant is long, the sound collection sensitivity of the microphone is increased, and the Y coordinate of the microphone is changed according to the value of the Y coordinate. The sound collection sensitivity may be determined.

Returning to FIG. 3, continuing description of each functional unit, the receiving unit 27 has a function of receiving attention area information from another base and outputting it to the directivity determining unit 28. In addition, the communication terminal 2B installed in another site, which transmits the attention area information received by the receiving unit 27, is an example of “another second communication terminal”. The above-mentioned “other first communication terminal” and “other second communication terminal” may be the same communication terminal or different communication terminals.

The directivity determination unit 28 determines the directivity of the sound collection of the microphone 219 based on the attention area information received by the reception unit 27 with reference to the data indicating the correspondence relationship of FIG. 9B stored in the SSD 204 or the like. However, the directivity control unit 29 has a function of outputting the determined directivity information.

The directivity control unit 29 can control the directivity of the sound collection of the microphone 219 based on the input directivity information.

In this embodiment, since a known beamforming technique can be applied, detailed description of the beamforming control technique and the like will be omitted.

<Operation of the video conference system according to the first embodiment>
Next, FIG. 10 is a flowchart showing an example of attention area detection processing by the communication terminal 2A according to the present embodiment.

First, in step S101, the image capturing unit 21 captures an image of a participant in a video conference at the site A, and outputs the captured image information to the line-of-sight detecting unit 22.

Subsequently, in step S102, the line-of-sight detection unit 22 detects the line of sight of the participant from the input image information, and outputs the detected line-of-sight information to the line-of-sight information storage unit 23. When there are a plurality of participants, the line of sight of each participant is detected and the line-of-sight information of each participant is output to the line-of-sight information storage unit 23.

Subsequently, in step S103, the line-of-sight information storage unit 23 stores the input line-of-sight information. If there are multiple participants, the line-of-sight information of each participant is stored.

Subsequently, in step S104, the line-of-sight detection unit 22 determines whether a predetermined time has passed. This "predetermined time" is a predetermined time for accumulating the line-of-sight information.

When it is determined that the predetermined time has not elapsed (No in step S104), the process returns to step S101. On the other hand, when it is determined that the predetermined time has passed (Yes at Step S104), the attention area detection unit 24 at Step S105, based on the line-of-sight information accumulated by the line-of-sight information accumulation unit 23, the base A Detects the attention area in which the participant is paying attention. Then, the detected attention area information is output to the subpacket generation unit 25.

Subsequently, in step S106, the subpacket generation unit 25 generates a subpacket to be transmitted to another base such as the base B from the input attention area information, and outputs the subpacket to the transmission unit 26.

Subsequently, in step S107, the transmission unit 26 includes the input subpacket in the conference data such as the video and audio of the participant and the synchronization data, and transmits the subpacket to another base such as the base B.

In this way, the communication terminal 2A can transmit the attention area information of the participants in the video conference at the base A to another base such as the base B.

Next, FIG. 11 is a flowchart showing an example of directivity control processing by the communication terminal 2A according to the present embodiment.

First, in step S111, the receiving unit 27 determines whether or not a subpacket including the attention area information is received. If the subpacket including the attention area information is not received (step S111, No), the process of step S111 is executed again. On the other hand, when the attention area information is received (Yes in step S111), the receiving unit 27 outputs the received attention area information to the directivity determining unit 28 in step S112. Based on the input attention area information, the directivity determination unit 28 refers to the data indicating the correspondence between the attention area information and the sound collection direction stored in the SSD 204 or the like, and determines the sound collection directivity of the microphone 219. .. Then, the determined directivity information is output to the directivity control unit 29.

Subsequently, in step S113, the directivity control unit 29 controls the directivity of the sound collection of the microphone 219 based on the input directivity information.

In this way, the communication terminal 2A can determine and control the directivity of the microphone 219 included in the communication terminal 2A based on the attention area information received from another site such as the site B.

Next, FIG. 12 is a sequence diagram showing an example of the operation of the video conference system according to the present embodiment.

First, in step S121, the communication terminal 2A transmits a signal requesting the start of a video conference to the communication terminal 2B.

Subsequently, in step S122, when the video conference can be started, the communication terminal 2B transmits a request response signal indicating that to the communication terminal 2A.

Then, in step S123, the communication terminal 2A executes the attention area detection process described in FIG.

Subsequently, in step S124, the communication terminal 2A includes the subpacket including the attention area information in the conference data and transmits the conference data to the communication terminal B.

Subsequently, in step S125, the communication terminal 2B executes the directivity control process described in FIG. 11 based on the attention area information included in the received subpacket.

Subsequently, in step S126, when ending the video conference, the communication terminal 2A transmits a signal requesting the end of the video conference to the communication terminal 2B.

Then, in step S127, when the video conference can be ended, the communication terminal 2B transmits a request response signal indicating that to the communication terminal 2A.

In this way, the video conference system 1 can conduct a video conference.

<Effects of the video conference system according to the first embodiment>
Next, the effect of the video conference system according to the present embodiment will be described.

FIG. 13 is a diagram illustrating an example of effects of the video conference system according to the present embodiment. (A) is a figure explaining the video conference using the communication terminal which concerns on a comparative example, (b) is a figure explaining a video conference using the communication terminal which concerns on this embodiment.

In FIG. 13A, the communication terminal 5A according to the comparative example is installed adjacent to the communication terminal 5Aoth. The video conference 10 in which the communication terminal 5A is used and the video conference 10oth in which the communication terminal 5Aoth is used are different video conferences. Further, the communication terminal 5A has a function of controlling the directivity of the microphone so as to collect the voice in the direction (area) in which the sound is generated.

In the case of FIG. 13A, when the distance between the communication terminal 5A and the communication terminal 5Aoth is short, the communication terminal 5A reacts to the speech of the participant 501a of the video conference 10oth and moves in the direction of the participant 501a (arrow 131). The directivity of the microphone may be erroneously determined so as to collect voice.

On the other hand, in FIG. 13B, the communication terminal 2A according to the present embodiment is installed adjacent to the communication terminal 5Aoth, as in the case of FIG. 13A. Further, the video conference 10 in which the communication terminal 2A is used and the video conference 10oth in which the communication terminal 5Aoth is used are different video conferences.

In the present embodiment, as described above, the directivity of the microphone 219 is controlled based on the attention area information transmitted from the communication terminal 2B of another base such as the base B. Since the participant 501a participating in another video conference is not included in the attention area of the participant at the base B, even if the participant 501a speaks, the communication terminal 2A participates in the directivity of the microphone 219. It does not make an erroneous decision in the direction of the person 501a.

In this way, in the present embodiment, the sound collection directivity of the microphone 219 can be appropriately determined in the direction of the participant 101a of the video conference 10.

The imaging unit 21 included in the communication terminal 2A may have an imaging field of view set in advance so as not to be included in an image captured by a participant participating in another video conference. This ensures that the participant who is participating in another video conference is not included in the attention area of the participant at the base B, so that the directivity of the microphone is directed toward the participant 501a. It is possible to reliably prevent erroneous decisions.

Further, in the present embodiment, an example in which the communication terminal 2A transmits the detected attention area information to another first communication terminal and determines the directivity based on the attention area information received from the other second communication terminal. Although described, the present invention is not limited to this.

For example, the communication terminal 2A transmits the detected line-of-sight information to the other first communication terminal, accumulates the line-of-sight information received from the other second communication terminal, detects the attention area based on the accumulated line-of-sight information, The directivity may be determined based on the detected attention area information. The line-of-sight information transmitted in this case is an example of “predetermined coordinate information”.

Alternatively, the communication terminal 2A transmits the directivity information determined based on the detected attention area to the other first communication terminal, and controls the directivity based on the directivity information received from the other second communication terminal. Good.

Here, when the communication terminal 2A transmits the attention area information or the directivity information to another first communication terminal, the amount of data to be transmitted as compared with the case where the line-of-sight information is transmitted to the other first communication terminal. It is possible to obtain the effect of reducing

[Second Embodiment]
Next, a video conference system according to the second embodiment will be described. Note that description of the same components as those of the above-described embodiment will be omitted.

Here, when the communication terminal 2 is fixedly installed in a predetermined place such as a conference room, the place where the participant is seated is almost the same for each video conference, and thus the place where the participant is seated is determined in the previous video conference. In some cases, the directional information can be applied as it is in the video conference held this time.

Therefore, in the present embodiment, the directivity information in the previous video conference is stored in the directivity information storage unit 30, and the directivity determination unit 28a refers to the directivity information storage unit 30 in the current video conference. Then, the directivity is determined based on the acquired directivity information.

FIG. 14 is a block diagram illustrating an example of a functional configuration of the communication terminal 2Aa included in the video conference system 1a according to this embodiment.

The communication terminal 2Aa includes a directivity determining unit 28a, a directivity information storage unit 30, and a directivity information updating unit 31.

The directivity information storage unit 30 has a function of inputting the directivity information determined by the directivity determining unit 28a via the directivity information updating unit 31 and storing the input directivity information.

In addition, the directivity determination unit 28a has a function of determining the directivity of the sound collection of the microphone 219 based on the attention area information, and the directivity of the sound collection of the microphone 219 based on the directivity information acquired from the directivity information storage unit 30. Has the function of determining sex.

The directional information updating unit 31 has a function of updating the directional information stored in the directional information storage unit 30.

FIG. 15 is a sequence diagram showing an example of the operation of the video conference system according to this embodiment.

First, in step S151, the communication terminal 2Aa transmits a signal requesting the start of a video conference to the communication terminal 2Ba.

Subsequently, in step S152, if the video conference can be started, the communication terminal 2Ba transmits a request response signal indicating that to the communication terminal 2Aa.

Subsequently, in step S153, the communication terminal 2Ba determines the sound collection directivity of the microphone 219 based on the directivity information acquired from the directivity information storage unit 30 of the communication terminal 2Ba.

Subsequently, in step S154, the communication terminal 2Ba controls the directivity of the sound collection of the microphone 219 based on the directivity information.

Then, in step S155, the communication terminal 2Aa executes the attention area detection process described in FIG.

Subsequently, in step S156, the communication terminal 2Aa transmits the subpacket including the attention area information to the communication terminal Ba by including it in the conference data.

Subsequently, in step S157, the communication terminal 2Ba executes the directivity control process described in FIG. 11 based on the attention area information included in the received subpacket.

Subsequently, in step S158, the communication terminal 2Ba updates the directivity information stored in the directivity information storage unit 30 of the communication terminal 2Ba with the directivity information determined this time.

Then, in step S159, when ending the video conference, the communication terminal 2Aa transmits a signal requesting the end of the video conference to the communication terminal 2Ba.

Subsequently, in step S160, if the video conference can be ended, the communication terminal 2Ba transmits a request response signal indicating that to the communication terminal 2Aa.

In this way, the video conference system 1 can conduct a video conference.

As described above, in the present embodiment, the directivity information of the previous video conference is stored in the directivity information storage unit 30, and the directivity determination unit 28a stores the directivity information storage in the current video conference. The directivity is determined based on the directivity information acquired by referring to the unit 30.

Immediately after the start of the video conference, it takes a certain amount of time to determine the directivity due to line-of-sight detection, accumulation of line-of-sight information, etc. According to the present embodiment, immediately after the start of the video conference, the previous video Since the directivity is determined based on the directivity information in the conference, the directivity can be set appropriately immediately after the start of the video conference, and the video conference can be smoothly executed immediately after the start.

The effects other than those described above are the same as those described in the first embodiment.

[Third Embodiment]
Next, a video conference system according to the third embodiment will be described.

FIG. 16 is a block diagram illustrating an example of the functional configuration of the communication terminal 2Ab included in the video conference system 1b according to this embodiment.

The communication terminal 2Ab has an input screen display unit 32 and a setting input unit 33.

The input screen display unit 32 has at least one of a time threshold value, a frequency threshold value, and a person number threshold value for the user of the video conference system 1b to detect the attention area, and the directivity determination unit 28 determines the directivity. It has a function of displaying an input screen for inputting the conditions and.

Here, FIG. 17 is a diagram illustrating an example of such an input screen. As shown in FIG. 17, the input screen 321 has, as input items, “microphone directivity control” for inputting whether or not to execute directivity control processing, and “predetermined time” for obtaining line-of-sight information. The “gaze information acquisition time” for inputting (see FIG. 10) and the “type of attention detection condition” for inputting the type of threshold used in detection of the attention area and the threshold are included.

Such an input screen 321 is displayed on the display 217, and the user of the video conference system 1b can input a threshold value for detecting the attention area and a condition for determining the directivity through the input screen 321. The administrator of the video conference system 1b may make such an input. The input items shown in FIG. 17 are merely examples, and other input items may be added or replaced with the input items shown in FIG.

Returning to FIG. 16, the description will be continued. The setting input unit 33 inputs the setting information input by the user through the input screen 321 displayed on the display 217, and outputs the setting information to the line-of-sight detection unit 22 and the attention area detection unit 24. The output destination can be determined according to the input item.

As described above, in the present embodiment, the user of the video conference system 1b displays the input screen for inputting the setting information such as various thresholds and the conditions for determining directivity, and the communication terminal of the communication terminal is displayed according to the input setting information. Make settings.

Although the appropriate conditions for determining the directivity may differ depending on the environment in which the video conference is performed, such as the volume of unnecessary sound in the surroundings, according to the present embodiment, the directivity is appropriately set according to the conference environment. Can be determined and controlled.

The present invention is not limited to the above specifically disclosed embodiments, and various modifications and changes can be made without departing from the scope of the claims.

The embodiment also includes a method of controlling the microphone of the communication terminal. For example, a method of controlling a microphone of a communication terminal is a method of controlling the microphone of a communication terminal used in a video conference by receiving an image input from a camera and an audio input from the microphone, wherein A step of capturing an image of a participant of the video conference, a step of accumulating line-of-sight information indicating the line-of-sight of the participant detected from the image, and a directivity of sound collection by the microphone based on the accumulated line-of-sight information. And a step of determining. With such a method of controlling the microphone of the communication terminal, it is possible to obtain the same effect as that of the above video conference system.

Further, each function of the embodiments described above can be realized by one or a plurality of processing circuits. Here, the “processing circuit” in the present specification is a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, or designed to execute each function described above. Devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules are included.

1, 1a, 1b Video conference system 2, 2A Communication terminal 2B Communication terminal (an example of another first communication terminal, an example of another second communication terminal)
3 network 4 server 21 imaging unit 22 line-of-sight detection unit 23 line-of-sight information storage unit 24 attention area detection unit 25 subpacket generation unit 26 transmission unit 27 reception unit 28 directivity determination unit 29 directivity control unit 30 directivity information storage unit 31 directivity Sex information updating unit 32 input screen display unit 33 setting input unit 201 CPU
202 RON
203 RAM
204 SSD
205 Network Controller 206 Sensor Controller 207 Capture Device 208 Electronic Pen Controller 209 External Storage Controller 210 GPU
211 Display Controller 212 Camera Controller 213 Microphone Controller 214 Speaker Controller 215 Electronic Pen 216 External Memory 217 Display 218 Camera 219 Microphone 220 Speaker

JP, 2017-034502, A

Claims (8)

A video conferencing system that includes a plurality of communication terminals, a camera that outputs images to the communication terminals, and a microphone that outputs audio to the communication terminals, and that performs a video conference.
The communication terminal,
A line-of-sight information storage unit that stores line-of-sight information indicating the line-of-sight of the participants of the video conference detected from the image,
A video conferencing system comprising: a directivity determining unit that determines a directivity of sound collection of the microphone based on the accumulated line-of-sight information. Based on the accumulated line-of-sight information, an attention area detection unit that detects an attention area in which the participant pays attention,
A transmitter that transmits the detected attention area to another first communication terminal,
The video conferencing system according to claim 1, wherein the directivity determination unit determines the directivity based on the attention area received from another second communication terminal. A sub-packet generation unit that generates a sub-packet including predetermined coordinate information extracted from the line-of-sight information,
The video conference system according to claim 2, wherein the transmitting unit transmits the subpacket. The directivity determination unit is a region where the line of sight of the participant is directed at a time equal to or greater than a predetermined time threshold, a region where the line of sight of the participant is directed at a number of times equal to or greater than a predetermined number of times threshold, and 4. The video conferencing system according to claim 1, wherein the directivity is determined based on at least one of the areas in which the line of sight of the participants is directed by a predetermined number of people or more. A directional information storage unit that stores the determined directional information,
A directional information updating unit for updating the stored directional information,
The video conferencing system according to claim 1, wherein the directivity determining unit determines the directivity based on the directivity information acquired by referring to the directivity information storage unit. A user of the video conference system displays an input screen for inputting at least one of the time threshold value, the number of times threshold value, and the number of people threshold value, and a condition for the directivity determining unit to determine the directivity. The video conference system according to claim 4, further comprising an input screen display unit. A communication terminal used in a video conference, which receives an image input from a camera and an audio input from a microphone,
A line-of-sight information storage unit that stores line-of-sight information indicating the line-of-sight of the participants of the video conference detected from the image,
And a directivity determining unit that determines the directivity of sound collection of the microphone based on the accumulated line-of-sight information. A method of controlling the microphone of a communication terminal used in a video conference, the method including receiving an image input from a camera and receiving an audio input from a microphone,
Capturing an image of the participants in the video conference with the camera;
Accumulating line-of-sight information indicating the line-of-sight of the participant detected from the image,
A method of controlling a microphone of a communication terminal, comprising: determining the directivity of sound collection by the microphone based on the accumulated line-of-sight information.

Images