JP2022165144A - Support device and program - Google Patents

Abstract

To provide a support device capable of appropriately controlling a mute function of a microphone.SOLUTION: A voice output section 15 outputs voice from a speaker 6 on the basis of voice data received by a communication section 13 from an external device. A detection section 16 detects that a person on an image is uttering on the basis of the image photographed by a camera 3. A voice silencing section 17 performs silencing not to output the voice to be input to a microphone 4 from the external device unless the detection section 16 detects utterance. The communication section 13 transmits the data of the voice that has been input to the microphone 4 to the external device when the detection section 16 detects utterance.SELECTED DRAWING: Figure 2

Description

TECHNICAL FIELD The present disclosure relates to devices and programs for supporting meetings.

US Pat. No. 6,200,000 describes a system for supporting meetings. In the system described in Patent Literature 1, multiple terminals are connected via a network. With this system, participants can participate in conferences from multiple remote sites.

JP 2019-61594 A

Participants in such meetings unmute their microphones when they speak. However, there is a problem that the mute function may be forgotten when speaking, and the operation is troublesome.

The present disclosure has been made to solve the problems described above. An object of the present disclosure is to provide a support device capable of appropriately controlling a microphone mute function. Another object of the present disclosure is to provide a program for appropriately controlling a microphone mute function.

The support device according to the present disclosure includes communication means for receiving audio data from an external device, audio output means for outputting audio from a speaker based on the audio data received by the communication means from the external device, and Based on the image, detection means for detecting that the person in the image is speaking, and if the detection means does not detect the speech, the sound input to the microphone is not output from the external device. and muffling means for muffling. The communication means transmits data of the voice input to the microphone to the external device if the detection means detects the utterance.

A program according to the present disclosure includes a first communication process for receiving audio data from an external device, an audio output process for outputting audio from a speaker based on the audio data received from the external device in the first communication process, and a camera. Based on the captured image, detection processing detects that the person in the image is speaking, and if the detection processing does not detect speech, the sound input to the microphone is output from the external device. and a second communication process for transmitting voice data input to the microphone to an external device if speech is detected by the detection process. .

Advantageous Effects of Invention According to the present disclosure, a microphone mute function can be appropriately controlled in a device for supporting a conference.

1 is a diagram showing an example of a system using a support device according to Embodiment 1; FIG. It is a figure which shows the example of a support apparatus. 4 is a flowchart showing an operation example of a control device; 8 is a flowchart showing another operation example of the control device; FIG. 10 is a diagram showing an example of minutes created by an editorial department being displayed on the display; It is a figure which shows the example of the hardware resources of a control apparatus. FIG. 10 is a diagram showing another example of hardware resources of a control device;

A detailed description is given below with reference to the drawings. Duplicate descriptions are appropriately simplified or omitted. In each figure, the same reference numerals denote the same or corresponding parts.

Embodiment 1.
FIG. 1 is a diagram showing an example of a system using a support device 1 according to Embodiment 1. As shown in FIG. In the system shown in FIG. 1, a plurality of support devices 1 are connected via a network 2. FIG. A plurality of support devices 1 may exist in the same building. Each of the plurality of support devices 1 may exist at a remote location.

A user of this system uses the support device 1 to hold a so-called Web conference. The support device 1 is, for example, a personal computer. The support device 1 may be a smart phone. The support device 1 may be a tablet terminal.

As an example, network 2 is an IP network. An IP network is a communication network using IP (Internet Protocol) as a communication protocol. Network 2 may be a closed network or an open network.

FIG. 2 is a diagram showing an example of the support device 1. As shown in FIG. The support device 1 includes a camera 3 , a microphone 4 , a display 5 , a speaker 6 and a control device 7 . The control device 7 includes a storage unit 10 , an image processing unit 11 , an audio processing unit 12 , a communication unit 13 , an image output unit 14 , an audio output unit 15 , a detection unit 16 and a muffling unit 17 .

Camera 3 takes an image. For example, when the user sits in front of the support device 1, the camera 3 takes an image of the user. Data of images captured by the camera 3 are input to the control device 7 . The image processing unit 11 processes image data captured by the camera 3 . In the following, data representing an image is also simply referred to as image data.

Voice data input to the microphone 4 is input to the control device 7 . The audio processing unit 12 processes audio data input to the microphone 4 . As an example, the audio processing unit 12 converts analog data from the microphone 4 into digital data. In the following, data representing audio is also simply referred to as audio data.

The communication unit 13 communicates with external devices. In the example shown in this embodiment, the external device is another support device 1 connected via network 2 . The communication unit 13 transmits image data captured by the camera 3 and audio data input to the microphone 4 to an external device. Specifically, the communication unit 13 transmits image data from the image processing unit 11 and audio data from the audio processing unit 12 to the external device.

Since similar processing is performed in the external device, the communication unit 13 receives image data and audio data from the external device. The image output unit 14 displays an image on the display 5 based on the image data received by the communication unit 13 from the external device. The audio output unit 15 outputs audio from the speaker 6 based on the audio data received by the communication unit 13 from the external device. With such basic operations, the user can hold a web conference with a remote person using the support device 1 .

FIG. 3 is a flow chart showing an operation example of the control device 7. As shown in FIG. Characteristic functions of the support device 1 will be described in detail below with reference to FIG. 3 as well.

When the conference starts, image data taken by the camera 3 is input to the control device 7 (S101). Based on the image captured by the camera 3, the detection unit 16 detects that the person in the image is speaking. As an example, the detection unit 16 identifies a person's face from the image data processed by the image processing unit 11 . The detection unit 16 further identifies the person's mouth from the identified face. The detection unit 16 calculates an index representing the movement of the specified mouth. When the calculated index exceeds the threshold, the detection unit 16 detects that the person in the image is speaking.

As another example, the detection unit 16 may further identify the line of sight of the person from the identified face. The detection unit 16 may detect that the person in the image is speaking based on the calculated index and line of sight.

In the control device 7, it is determined whether or not the detection unit 16 has detected that the person in the image is speaking (S102). If the detection unit 16 has detected the speech, it is determined as Yes in S102.

The muffling section 17 controls the mute function of the microphone 4 . That is, the mute function of the microphone 4 is automatically switched on (enabled) and off (disabled) by the muffling section 17 . If it is determined as Yes in S102, the muffling section 17 turns off the mute function of the microphone 4 (S103). Therefore, when it is determined as Yes in S102, the communication unit 13 transmits data of the voice input to the microphone 4 to the external device. Other users participating in the conference can hear the voice input to the microphone 4 .

On the other hand, if the detection unit 16 has not detected an utterance, a determination of No is made in S102. If it is determined No in S102, the muffling unit 17 turns on the mute function of the microphone 4 (S104). That is, when the determination in S102 is No, the muffling unit 17 muffles the sound input to the microphone 4 so that the sound is not output from the external device. Any method may be used for the muffling. If it is determined as No in S102, the voice data is not transmitted from the communication section 13 to the external device.

As described above, the support device 1 detects that the person in the image is speaking based on the data of the image captured by the camera 3 . Then, if the detection unit 16 detects the speech, the mute function of the microphone 4 is turned off. If the detection unit 16 does not detect speech, the mute function of the microphone 4 is turned on. Therefore, the mute function of the microphone 4 can be appropriately controlled during the conference.

FIG. 4 is a flow chart showing another operation example of the control device 7 . In the example shown in FIG. 4 , the control device 7 further includes a determination section 18 . Also, a first detection criterion and a second detection criterion are set in advance. The first detection criteria and the second detection criteria are criteria for the detection unit 16 to detect speech. The second detection criterion is a criterion different from the first detection criterion.

The process shown in S201 of FIG. 4 is the same as the process shown in S101 of FIG. When the conference starts, image data captured by the camera 3 is input to the control device 7 .

Based on the image captured by the camera 3, the determination unit 18 determines whether or not the person in the image is wearing a mask (S202). As an example, the determination unit 18 identifies human eyes from image data processed by the image processing unit 11 . If the determining unit 18 can determine that a certain area under the identified eye is covered with a cloth-like object in the image captured by the camera 3, it can be determined that the person in the image is wearing a mask. Determine (Yes in S202). If the determination unit 18 can identify the person's mouth under the identified eyes in the image captured by the camera 3, the determination unit 18 determines that the person in the image is not wearing a mask (No in S202). ). The method of determination by the determination unit 18 is not limited to this example.

If determined as Yes in S202, the detection unit 16 selects the first detection criterion as a criterion for detecting speech (S203). When determined as No in S202, the detection unit 16 selects the second detection criterion as a criterion for detecting speech (S204).

The process shown in S205 is the same as the process shown in S102 of FIG. That is, in the control device 7, it is determined whether or not the detection unit 16 has detected that the person in the image is speaking. In addition, in S205, the detection unit 16 detects an utterance based on the first detection criterion if it is determined as Yes in S202. The detection unit 16 detects an utterance based on the second detection criterion if determined as No in S202. As an example, when the determination in S202 is Yes, the detection unit 16 may calculate the index representing the movement based on a wider range of data than when the determination is No.

The processes shown in S206 and S207 are the same as the processes shown in S103 and S104 of FIG. That is, when it is determined as Yes in S205, the muffling unit 17 turns off the mute function of the microphone 4 (S206). If it is determined No in S205, the muffling unit 17 turns on the mute function of the microphone 4 (S207).

In the example shown in FIG. 4, the mute function of the microphone 4 can be appropriately controlled even when the user wears a mask during the season when infectious diseases are prevalent or when hay fever occurs.

As another example, the control device 7 may further include a storage section 19 . In the example shown in this embodiment, the detection unit 16 detects, based on an image captured by the camera 3, that a person in the image is speaking. Therefore, if the processing capability of the control device 7 is low, the timing at which the mute function of the microphone 4 is turned off is slightly delayed from the timing at which the speech is started.

The storage unit 19 stores voice data input to the microphone 4 in the storage unit 10 . Note that this audio data is saved even while the mute function of the microphone 4 is on. Then, when the detection unit 16 detects the speech, the communication unit 13 transmits the voice data stored in the storage unit 19 from a predetermined time before the detection unit 16 detects the speech to the external device. The certain period of time is set in advance. For example, the certain period of time is 0.5 seconds. As a result, it is possible to transmit the voice data from the beginning of the speech to the external device. The communication unit 13 controls the transmission of the voice data so that the data of the voice input to the microphone 4 can be transmitted to the external device as it is after a certain period of time has elapsed since the detection unit 16 detected the utterance. good.

As another example, the control device 7 may further include an editing section 20 . The editing unit 20 edits the audio data and image data stored in the storage unit 10 by the storage unit 19 during the meeting, and creates minutes. The minutes created by the editing unit 20 are output from the display 5 and the speaker 6 according to the user's operation.

In this example, the storage unit 19 preferably stores the result detected by the detection unit 16 in the storage unit 10 in association with the voice data input to the microphone 4 . Furthermore, the storage unit 19 may store the result detected by the detection unit 16 in the storage unit 10 in association with the data of the image captured by the camera 3 . The results detected by the detection unit 16 are, for example, first information indicating "speech" and second information indicating "no speech". Furthermore, the communication unit 13 may transmit the audio data and image data associated with the results detected by the detection unit 16 to the external device.

Similar processing is performed in the external device. Therefore, the communication unit 13 receives audio data and image data linked with the first information or the second information from the external device. The audio data and image data received by the communication unit 13 from the external device are stored in the storage unit 10 by the storage unit 19 while being associated with the first information or the second information.

For example, when a meeting ends, the editing section 20 creates minutes of the meeting. At this time, the editing unit 20 may create the minutes using only the audio data and image data associated with the first information.

FIG. 5 is a diagram showing an example in which the minutes created by the editing unit 20 are displayed on the display 5. As shown in FIG. In the example shown in FIG. 5, the display 5 displays an image display area 5a, a seek bar 5b, and a playlist 5c.

Each content included in the play list 5c indicates the audio data and image data stored in the storage unit 10 during the meeting that are associated with the first information. The contents are arranged in chronological order in the play list 5c. In the example shown in FIG. 5, each content is labeled with the names A to F of the person speaking and the speaking time. By selecting one of the contents included in the reproduction list 5c, the sound of the content is output from the speaker 6, and the image of the content is displayed in the image display area 5a. Note that the minutes created by the editing unit 20 are not limited to the example shown in FIG. For example, the editing unit 20 may create minutes of only audio data.

FIG. 6 is a diagram showing an example of hardware resources of the control device 7. As shown in FIG. The control device 7 includes a processing circuit 30 including a processor 31 and a memory 32 as hardware resources. A plurality of processors 31 may be included in the processing circuitry 30 . A plurality of memories 32 may be included in the processing circuitry 30 .

In the present embodiment, each part indicated by reference numerals 10 to 20 indicates the function of the control device 7. FIG. The functions of the units indicated by reference numerals 11 to 20 can be implemented by software written as a program, firmware, or a combination of software and firmware. The program is stored in memory 32 . The control device 7 implements the functions of the units indicated by reference numerals 11 to 20 by executing the programs stored in the memory 32 by the processor 31 (computer). The functions of the storage unit 10 are implemented by the memory 32 . A semiconductor memory or the like can be used as the memory 32 .

FIG. 7 is a diagram showing another example of hardware resources of the control device 7. As shown in FIG. In the example shown in FIG. 7 , the control device 7 comprises processing circuitry 30 including a processor 31 , memory 32 and dedicated hardware 33 . FIG. 7 shows an example in which a part of the functions of the control device 7 are implemented by dedicated hardware 33 . All the functions of the control device 7 may be realized by dedicated hardware 33 . Dedicated hardware 33 can be a single circuit, multiple circuits, programmed processors, parallel programmed processors, ASICs, FPGAs, or combinations thereof.

1 support device 2 network 3 camera 4 microphone 5 display 6 speaker 7 control device 10 storage unit 11 image processing unit 12 audio processing unit 13 communication unit 14 image output unit 15 audio output unit 16 detection unit 17 muffling unit 18 determination unit 19 storage unit 20 editor 30 processing circuit 31 processor 32 memory 33 dedicated hardware

Claims (6)

a communication means for receiving audio data from an external device;
audio output means for outputting audio from a speaker based on the audio data received by the communication means from the external device;
detection means for detecting, based on an image captured by a camera, that a person in the image is speaking;
mute means for muting the sound input to the microphone so that it is not output from the external device if the detection means does not detect an utterance;
with
The communication means is a support device that transmits data of voice input to a microphone to the external device if the detection means detects an utterance. Based on the image taken by the camera, further comprising determination means for determining whether or not the person in the image is wearing a mask,
The detection means is
When the determining means determines that the mask is being worn, detecting the utterance based on a specific first detection criterion,
2. The assisting device according to claim 1, wherein when said determining means determines that said speech is not masked, said speech is detected based on a specific second detection criterion different from said first detection criterion. Further comprising storage means for storing data of the voice input to the microphone,
1 or claim 1, wherein, when said detection means detects an utterance, said communication means transmits to said external device the voice data stored by said storage means from a predetermined time before said detection means detects an utterance. 3. The support device according to 2. a storage means for storing the result detected by the detection means in association with data of the voice input to the microphone;
Editing means for editing the audio data saved by the saving means to create minutes;
further comprising
3. The support device according to claim 1, wherein said communication means transmits audio data associated with a result detected by said detection means to said external device. a first communication process for receiving audio data from an external device;
an audio output process for outputting audio from a speaker based on the audio data received from the external device in the first communication process;
a detection process for detecting, based on an image captured by a camera, that a person in the image is speaking;
mute processing for muting the sound input to the microphone so that it is not output from the external device if no speech is detected in the detection processing;
a second communication process for transmitting data of voice input to a microphone to the external device if an utterance is detected in the detection process;
A program that causes a computer to run Based on the image captured by the camera, causing the computer to further execute determination processing for determining whether or not the person in the image is wearing a mask,
In the detection process,
When it is determined in the determination process that the mask is being worn, the speech is detected based on a specific first detection criterion,
6. The program according to claim 5, wherein if the determination processing determines that the mask is not applied, the program detects speech based on a specific second detection criterion different from the first detection criterion.

Images