JP2023088360A - Video call device, video call method, and control program of video call device - Google Patents

Abstract

To provide a video call device capable of preventing or reducing voice not intended to transmit to a call destination from being transmitted to the call destination and being heard.SOLUTION: A camera 1 captures a speaker making a video call. A microphone 6 picks up voice uttered by the speaker. A communication unit 5 transmits the captured image of the speaker captured by the camera 1 and the voice of the speaker picked up by the microphone 6 to a network 20. An image analysis unit 3 detects a line of sight of the speaker on the basis of the captured image of the speaker captured by the camera, and detects whether the line of sight is off the camera 1. A voice control unit 4 controls the communication unit 5 so as to cut off the transmission of the voice or reduce volume of the voice, when at least the image analysis unit 3 detects that the line of sight of the speaker is off the camera 1.SELECTED DRAWING: Figure 1

Description

The present invention relates to a video call device, a video call method, and a video call device control program.

2. Description of the Related Art As described in Patent Literature 1, a video call system (teleconference system) that bi-directionally communicates images and voices via a network has become widespread.

JP 2012-100185 A

Suppose a participant on a first side is calling a participant on a second side using a video calling system. At this time, the murmurs that the speaker on the first side intended to convey only to the participants sitting on the first side were picked up by the microphone and transmitted to the participants on the second side. I can put it away. It is not preferable that such voices such as murmurs that are not intended to be transmitted to the called party are transmitted to the called party.

The present invention provides a video call device, a video call method, and a control program for the video call device that can prevent or reduce the transmission of voices not intended for transmission to the callee and being heard by the callee. intended to provide

The present invention comprises a camera for photographing a speaker making a video call, a microphone for picking up the voice uttered by the speaker, an image of the speaker photographed by the camera, and the speaker picked up by the microphone. and an image analysis unit that detects the line of sight of the speaker based on the captured image of the speaker captured by the camera and detects whether the line of sight is off the camera. and at least when the image analysis unit detects that the line of sight of the speaker is out of the camera, the communication unit is controlled to cut off the transmission of the voice or reduce the volume of the voice. and a video call device.

The present invention comprises a camera that captures a video call speaker, a microphone that picks up the voice uttered by the speaker, an image analysis unit that analyzes the captured image of the speaker captured by the camera, and the microphone. a voice analysis unit that analyzes the collected voice of the speaker; a communication unit that transmits to a network the captured image of the speaker captured by the camera and the voice of the speaker captured by the microphone; a recording unit that records the captured image and the sound picked up by the microphone; and the analysis result of the reproduced photographed image analyzed by the image analysis unit and the analysis result of the reproduced sound analyzed by the sound analysis unit. The learning result, the analysis result of the image captured by the new video call analyzed by the image analysis unit, and the sound of the new video call analyzed by the sound analysis unit are learned at the time of the new video call. Based on the analysis result, a learning unit that extracts a section in which a voice that should not be sent to the network is emitted by the communication unit, and a voice that should not be sent to the network extracted by the learning unit is emitted. and a voice control unit for controlling the communication unit so as to cut off transmission of voice in a section where the voice is present or to reduce the volume of the voice.

According to the present invention, a camera photographs a speaker who makes a video call, a microphone picks up the voice uttered by the speaker, and an image analysis unit detects the speaker based on the photographed image of the speaker photographed by the camera. and detects whether or not the line of sight of the speaker is off the camera, and if the image analysis unit does not detect that the line of sight of the speaker is off the camera, the communication unit A captured image of the speaker captured by a camera and the speaker's voice collected by the microphone are transmitted to a network, and at least the image analysis unit is in a state where the speaker's line of sight is off the camera. A video call method is provided in which, when detecting that, the transmission of the voice by the communication unit is cut off or the volume of the voice is lowered.

According to the present invention, a step of detecting, in a computer, a line of sight of a speaker taking a video call based on a photographed image of the speaker, and detecting whether or not the line of sight has moved away from the camera. Then, if it is not detected that the line of sight of the speaker is out of the camera, the communication unit outputs the image captured by the camera of the speaker and the voice of the speaker picked up by the microphone. to the network, and at least if it is detected that the line of sight of the speaker is out of the camera, the transmission of the voice by the communication unit is cut off or the volume of the voice is reduced. and providing a control program for a video call device that causes the steps to be performed.

According to the video calling device, the video calling method, and the control program for the video calling device of the present invention, it is possible to prevent or reduce the transmission of voices not intended to be transmitted to the called party and being heard by the called party. can be done.

1 is a block diagram showing a video calling device according to a first embodiment; FIG. 4 is a flow chart showing the operation of the video call device of the first embodiment and the video call method of the first embodiment; FIG. 11 is a block diagram showing a video calling device of a second embodiment; FIG. 8 is a flow chart showing the operation of the video call device of the second embodiment and the video call method of the second embodiment; FIG. 11 is a block diagram showing a video calling device of a third embodiment; FIG.

A video call device, a video call method, and a control program for the video call device of each embodiment will be described below with reference to the accompanying drawings.

<First embodiment>
In FIG. 1, the video call device 101, the video call server 30, and the video call device 40 of the first embodiment, which are connected to each other via the network 20, constitute a video call system. The video calling device 101 transmits image data and voice data to the video calling device 40 and receives image data and voice data from the video calling device 40 via the network 20 and the video calling server 30 . FIG. 1 shows a state in which image data and audio data are transmitted from the video calling device 101 to the video calling device 40. As shown in FIG.

A video call device 101 includes a camera 1 , temporary storage memory 2 , image analysis section 3 , audio control section 4 , communication section 5 , microphone 6 , and temporary storage memory 7 . A camera 1 photographs a speaker who makes a video call, who is a user of the video call device 101 . A plurality of people including the speaker may be present at the place where the video communication device 101 is arranged. A temporary storage memory 2 temporarily stores captured image data output from the camera 1 . The image analysis unit 3 analyzes the captured image as described later. The voice control unit 4 may block transmission of voice data to the network 20 by the communication unit 5 . The audio control unit 4 may lower the volume of the audio data to be transmitted instead of interrupting the transmission of the audio data.

A microphone 6 picks up the voice uttered by the speaker. Temporary storage memory 7 temporarily stores voice data output from microphone 6 . The communication unit 5 transmits the captured image data and the audio data to the video call server 30 via the network 20 if the audio control unit 4 does not control the communication unit 5 to block the transmission of the audio data. Typically, network 20 is the Internet, and communication unit 5 transmits captured image data and audio data according to Internet protocol. The video call device 40 receives shadow image data and audio data from the video call server 30 via the network 20 .

In the video call system configured as described above, the video call device 101 is arranged on the first side and the video call device 40 is arranged on the second side. Suppose a speaker, one of the participants on the first side, is talking to a participant on the second side. A speaker on the first side utters a whisper-like sound intended only for the participants sitting on the first side rather than the sound intended to be transmitted to the participants on the second side Occasionally, the speaker will face the attendant and utter the sound in a low voice.

In the first embodiment, it is determined whether or not the voice is intended to be transmitted to the participants on the second side depending on whether or not the line of sight of the speaker is directed toward the camera 1. . Specifically, the image analysis unit 3 detects the line of sight of the speaker based on the photographed image indicated by the inputted photographed image data. The image analysis unit 3 detects whether or not the line of sight has left the camera 1 . When the image analysis unit 3 detects that the line of sight has left the camera 1, the image analysis unit 3 notifies the sound control unit 4 of the line of sight separation.

When the image analysis unit 3 notifies that the line of sight is off, the audio control unit 4 controls the communication unit 5 to block transmission of the audio data to the network 20 . In this case, the voice control unit 4 functions as a cutoff control unit that controls to cut off transmission of voice data. As a result, even if the speaker turns his/her face to the other participant and utters a voice such as muttering that is not intended to be transmitted to the other party, the voice will not be transmitted to the other party. It is possible to prevent the participants on the second side from being heard carelessly.

The voice control unit 4 may control the communication unit 5 so as to reduce the volume of the voice data when the image analysis unit 3 notifies that the line of sight is off. In this case, the audio control unit 4 functions as a volume control unit that controls to lower the volume. Usually, sounds such as murmurs that are not intended to be transmitted to the called party are emitted at a low volume. Therefore, even if the communication unit 5 transmits a voice such as murmuring, the volume is such that the participants on the second side cannot hear it. As a result, even if the speaker turns his face to the participant who is present and utters a voice such as muttering that is not intended to be transmitted to the other party, the voice data is transmitted at an extremely low volume. Therefore, it is possible to reduce the chances of the participants on the second side being unintentionally asked.

The voice control unit 4 does not control the communication unit 5 to block transmission of the voice data to the network 20 unless the image analysis unit 3 notifies that the line of sight is off, and also reduces the volume of the voice data. The communication unit 5 is not controlled to allow Therefore, the communication unit 5 sends the captured image data and the audio data to the network 20 as they are.

The operation of the video call device 101 and the video call method of the first embodiment will be described using the flowchart shown in FIG. Steps S2 to S5, S11, and S12 in the flowchart shown in FIG. 2 may represent processing that the control program for the video call device of the first embodiment causes the computer included in the video call device 101 to execute.

In FIG. 2, when the video call processing by the video call system is started, the camera 1 acquires the captured image data of the speaker in step S1. In step S2, the image analysis unit 3 determines whether or not the line of sight of the speaker has been detected. If the line of sight of the speaker is not detected (NO), the image analysis unit 3, in step S4, detects murmurs based on the captured image, which is a value indicating that no sound such as murmurs has been detected. L” is generated and supplied to the audio control unit 4. After that, the process proceeds to step S12. "L" is, for example, "0".

If the line of sight of the speaker has been detected in step S2 (YES), the image analysis unit 3 determines whether or not the line of sight has left the camera 1 in step S3. If the line of sight does not deviate from the camera 1 (NO), the image analysis unit 3 generates murmur detection "L" based on the captured image and supplies it to the voice control unit 4 in step S4. After that, the process proceeds to step S12. As described above, in the first embodiment, the state in which the line of sight is not deviated from the camera 1 is regarded as the state in which no sound such as murmuring is detected.

If the line of sight is out of the camera 1 in step S3 (YES), the image analysis unit 3, in step S5, detects a murmur based on the captured image, which is a value indicating that a sound like murmur has been detected. A detection "H" is generated and supplied to the voice control unit 4. "H" is, for example, "1". Subsequently, in step S11, the audio control unit 4 cuts off the transmission of the audio data. The audio control unit 4 may reduce the volume of the audio data in step S11. After that, the process proceeds to step S12.

In step S12, the video call device 101 determines whether or not an instruction to end the video call has been issued. If no instruction to end the video call is given (NO), the video call device 101 repeats the processing of steps S1 to S12. If an instruction to end the video call is issued (YES), the video call device 101 ends the processing of the video call.

As described above, according to the first embodiment, it is possible to prevent or reduce the possibility that voices not intended to be transmitted to the called party are transmitted to and heard by the called party.

<Second embodiment>
In the video call device 102 of the second embodiment shown in FIG. 3, the same reference numerals are given to the same parts as in the video call device 101, and the description thereof may be omitted. The video call device 102 includes a voice analysis unit 8 as a component that the video call device 101 does not have. The audio control unit 4 may block transmission of audio data from the communication unit 5 to the network 20 based on both the analysis result of the image analysis unit 3 and the analysis result of the audio analysis unit 8 . The audio control unit 4 may reduce the volume of the audio data to be sent based on both the analysis result by the image analysis unit 3 and the analysis result by the audio analysis unit 8 .

The audio analysis unit 8 determines whether or not the sound pressure level of the input audio data is equal to or lower than a predetermined threshold. The voice analysis unit 8 performs discrete Fourier transform on the input voice data. Typically, the speech analysis unit 8 performs a discrete Fourier transform on speech data using a Fast Fourier Transform (FFT) algorithm.

Speech that is intended to be sent to participants on the second side is often voiced, and speech that is intended to be conveyed only to participants sitting on the first side, such as murmurs, is often unvoiced. Voiced sounds and unvoiced sounds differ in frequency bands in which they exist. The speech analysis unit 8 determines whether the sound is voiced or unvoiced based on the frequency of the frequency domain data obtained by performing the discrete Fourier transform of the speech data, which is the time domain data, to the participants on the second side. It is determined whether the sound is the intended sound or whether the sound such as muttering is unvoiced sound.

When the sound pressure level of the input audio data before the discrete Fourier transform is equal to or lower than a predetermined threshold and the data after the discrete Fourier transform is the frequency of unvoiced sound, the sound analysis unit 8 detects that the microphone 6 has picked up the sound. It is preferable to determine that the voice is murmur-like voice. The sound analysis unit 8 does not determine whether or not the sound pressure level is equal to or less than the threshold, and when the frequency of the frequency domain data is the frequency of unvoiced sound, the sound picked up by the microphone 6 is a murmur-like sound. It may be determined that there is The voice analysis unit 8 notifies the voice control unit 4 of the analysis result as to whether the voice picked up by the microphone 6 is voiced sound or unvoiced sound. Note that the speech analysis unit 8 may determine that a word in a foreign language containing unvoiced consonants is unvoiced when the number of unvoiced sounds is greater than or equal to a predetermined ratio so as not to erroneously determine that the words are murmurs.

When the image analysis unit 3 notifies the sound control unit 4 that the line of sight is off and the sound analysis unit 8 notifies the analysis result that the sound picked up by the microphone 6 is unvoiced sound, the sound control unit 4 converts the sound data. The communication unit 5 is controlled so as to cut off transmission to the network 20 . As a result, even if the speaker turns his/her face to the other participant and utters a voice such as muttering that is not intended to be transmitted to the other party, the voice is not transmitted to the other party. It is possible to prevent the participants on the other side from being heard carelessly.

When the image analysis unit 3 notifies the voice control unit 4 that the line of sight is off and the voice analysis unit 8 notifies the analysis result that the voice picked up by the microphone 6 is unvoiced sound, the voice control unit 4 adjusts the volume of the voice data. You may control the communication part 5 so that it may reduce. As a result, even if the speaker turns his/her face to the participant who is present and utters a voice such as murmuring that is not intended to be transmitted to the other party, the voice data will be transmitted at an extremely low volume. Therefore, it is possible to reduce the chances of the participants on the second side being unintentionally asked.

If the voice control unit 4 is not notified by the image analysis unit 3 that the line of sight is off, or if the voice analysis unit 8 is not notified of the analysis result that the voice picked up by the microphone 6 is unvoiced sound, the voice control unit 4 cannot reproduce the voice data. The communication unit 5 is not controlled to block transmission to the network 20. Also, the audio control unit 4 does not control the communication unit 5 to lower the volume of the audio data. Therefore, the communication unit 5 sends the captured image data and the audio data to the network 20 as they are.

The operation of the video call device 102 and the video call method of the second embodiment will be described using the flowchart shown in FIG. Steps S2 to S5, S7 to S11, and S20 to S22 of the flow chart shown in FIG. 4 may represent processing executed by the computer provided in the video calling device 101 by the control program of the video calling device of the second embodiment.

In FIG. 4, steps S1-S5 are the same as steps S1-S5 shown in FIG. The microphone 6 acquires the audio data of the voice uttered by the speaker in step S6. The speech analysis unit 8 performs discrete Fourier transform on the speech data in step S7. In step S8, the sound analysis unit 8 determines whether or not the sound pressure level is equal to or less than the threshold based on the sound data before discrete Fourier transform. If the sound pressure level is not equal to or lower than the threshold value (NO), the voice analysis unit 8 generates murmur detection "L" based on voice and supplies it to the voice control unit 4 in step S10. After that, the process proceeds to step S20.

If the sound pressure level is equal to or lower than the threshold in step S8 (YES), the speech analysis unit 8 determines whether or not the frequency indicates unvoiced sound in step S9 based on the frequency domain data after the discrete Fourier transform. do. If the frequency does not indicate unvoiced sound (NO), the voice analysis unit 8 generates a voice-based murmur detection “L” and supplies it to the voice control unit 4 in step S10. If the frequency indicates unvoiced sound (YES), the voice analysis unit 8 generates a murmur detection "H" based on voice and supplies it to the voice control unit 4 in step S11. Processing proceeds from step S10 or S11 to step S20.

Step S8 may be omitted and step S10 and step S11 may be selected only by the determination of step S9.

In step S20, the voice control unit 4 determines whether or not the murmur detection is "H" based on the captured image and the murmur detection is "H" based on the voice. If the murmur detection is "H" based on the captured image and the murmur detection is "H" based on the voice (YES), the voice control unit 4 cuts off the transmission of voice data in step S21. The audio control unit 4 may reduce the volume of the audio data in step S21. After that, the process proceeds to step S22. If the murmur detection is not "H" based on the captured image and the murmur detection is not "H" based on the voice (NO), the process proceeds from step S20 to step S22.

In step S22, the video call device 102 determines whether or not an instruction to end the video call has been issued. If no instruction to end the video call is issued (NO), the video call device 102 repeats the processing of steps S1 to S22. If an instruction to end the video call is given (YES), the video call device 102 ends the processing of the video call.

As described above, according to the second embodiment, it is possible to prevent or reduce the possibility that voices not intended to be transmitted to the called party are transmitted to and heard by the called party. According to the second embodiment, when the speaker only turns his/her face to the participant who is present and does not actually utter a voice such as muttering, the transmission of the voice data is not interrupted. It is possible to cut off the transmission of voice data when a voice such as muttering is uttered.

<Third Embodiment>
In the video call device 103 of the third embodiment shown in FIG. 5, the same reference numerals are given to the same parts as those of the video call device 102, and the description thereof may be omitted. The video call device 103 includes a recording unit 10, a learning unit 11, a display unit 12, a speaker 13, and an operation unit 14 as components that the video call device 102 does not have. The display unit 12 , the speaker 13 and the operation unit 14 may be externally attached to the video call device 103 .

The recording unit 10 records the captured image data output from the temporary storage memory 2 and the audio data output from the temporary storage memory 7 . For example, after finishing the video call, the speaker operates the operation unit 14 to reproduce the captured image data and voice data from the past video call recorded in the recording unit 10 . The display unit 12 displays a captured image based on the captured image data being reproduced. The speaker 13 outputs sound based on the reproduced sound data.

When the captured image data and audio data recorded in the recording unit 10 are reproduced, the image analysis unit 3 analyzes the captured image of the captured image data being reproduced, and the audio analysis unit 8 analyzes the reproduced audio. Analyze the audio of the data. The image analysis unit 3 detects the line of sight of the speaker. The speech analysis unit 8 determines whether the sound is voiced or unvoiced based on the frequency of the sound, determines the sound pressure level of the sound, and further determines whether the sound is voiced or unvoiced based on the frequency. judge.

While confirming the captured image displayed on the display unit 12 and the sound output from the speaker 13, the speaker designates a section during which the sound that should not be transmitted to the network 20 is emitted by the communication unit 5. . The learning unit 11 learns a segment in which speech that should not have been transmitted to the network 20 is generated based on the segment specified by the speaker. At this time, the learning unit 11 combines the section specified by the speaker, the analysis result of the captured image analyzed by the image analysis unit 3 in the section specified by the speaker, and the analysis result of the sound analyzed by the sound analysis unit 8. are associated and learned.

The speaker causes the learning unit 11 to learn the voice that should not be sent to the network 20 multiple times after the end of the video call as described above. Then, the learning unit 11 determines whether or not there is a section in which a sound that should not be transmitted to the network 20 is emitted, based on the analysis result of the captured image by the image analysis unit 3 and the analysis result of the sound by the sound analysis unit 8. It is possible to acquire the judgment ability to judge

After the learning unit 11 acquires the determination ability, the learning unit 11 extracts a section in which voices that should not be transmitted to the network 20 by the communication unit 5 are emitted during a new video call. The learning unit 11 is based on the learning result, the analysis result of the image captured by the new video call analyzed by the image analysis unit 3, and the analysis result of the sound of the new video call analyzed by the sound analysis unit 8. , to extract a section in which a sound that should not be transmitted is emitted. Information indicating the interval extracted by the learning unit 11 is supplied to the voice control unit 4 . The audio control unit 4 controls the communication unit 5 so as to cut off the transmission of the audio in the section extracted by the learning unit 11 or reduce the volume of the audio.

As described above, according to the third embodiment, it is possible to prevent or reduce the possibility that voices not intended to be transmitted to the called party are transmitted to and heard by the called party. According to the third embodiment, the learning unit 11 learns in advance the section in which the sound that should not be transmitted to the network 20 is emitted, and acquires the determination ability to determine whether or not it is such a section. Therefore, voices that should not be transmitted to the network 20 can be blocked with high accuracy.

The present invention is not limited to the first to third embodiments described above, and various modifications can be made without departing from the gist of the present invention.

1 camera 2, 7 temporary storage memory 3 image analysis unit 4 audio control unit 5 communication unit 6 microphone 8 audio analysis unit 10 recording unit 11 learning unit 12 display unit 13 speaker 14 operation unit 20 network 30 video call server 40, 101 to 103 video calling device

Claims (5)

a camera that captures the person making the video call;
a microphone for picking up the voice uttered by the speaker;
a communication unit that transmits to a network the captured image of the speaker captured by the camera and the speaker's voice collected by the microphone;
an image analysis unit that detects the line of sight of the speaker based on an image captured by the camera of the speaker and detects whether or not the line of sight is off the camera;
audio control for controlling the communication unit to cut off transmission of the audio or reduce the volume of the audio at least when the image analysis unit detects that the line of sight of the speaker is out of the camera; Department and
A video call device with further comprising a speech analysis unit that analyzes whether the speech of the speaker picked up by the microphone is voiced or unvoiced,
When the image analysis unit detects that the line of sight of the speaker is out of the camera and the audio analysis unit analyzes that the speaker's voice is unvoiced sound 2. The video call device according to claim 1, wherein the communication unit is controlled to block transmission of the sound or reduce the volume of the sound. a camera that captures the person making the video call;
a microphone for picking up the voice uttered by the speaker;
an image analysis unit that analyzes an image captured by the camera of the speaker;
a voice analysis unit that analyzes the speaker's voice picked up by the microphone;
a communication unit that transmits to a network the captured image of the speaker captured by the camera and the speaker's voice collected by the microphone;
a recording unit that records the captured image and the sound picked up by the microphone;
A section during which the voice, which should not be transmitted to the network by the communication unit, is emitted and which is specified by the speaker when reproducing the captured image and voice of the past video call recorded in the recording unit, and the image The analysis result of the reproduced photographed image analyzed by the analysis unit and the analysis result of the reproduced sound analyzed by the audio analysis unit are associated and learned, and when a new video call is made, the learning result and the image are learned. Based on the analysis result of the image captured by the new video call analyzed by the analysis unit and the analysis result of the sound of the new video call analyzed by the sound analysis unit, the communication unit should send it to the network. A learning unit that extracts a section in which a voice that is not
a voice control unit that controls the communication unit so as to cut off the transmission of the voice in the section where the voice that should not be transmitted to the network extracted by the learning unit or to reduce the volume of the voice;
A video call device with Take a picture of the speaker who makes a video call with a camera,
picking up the voice uttered by the speaker with a microphone;
an image analysis unit that detects the line of sight of the speaker based on an image captured by the camera of the speaker, and detects whether or not the line of sight is off the camera;
If the image analysis unit does not detect that the line of sight of the speaker is out of the camera, the communication unit detects the captured image of the speaker captured by the camera and the speech captured by the microphone. and send the voice of the person to the network,
A video call method, wherein at least when the image analysis unit detects that the line of sight of the speaker is out of the camera, transmission of the sound by the communication unit is cut off or the volume of the sound is reduced. to the computer,
a step of detecting a line of sight of the speaker based on a photographed image of the speaker, and detecting whether or not the line of sight is off the camera;
If it is not detected that the line of sight of the speaker is out of the camera, the communication unit transmits the captured image of the speaker captured by the camera and the voice of the speaker collected by the microphone to a network. sending to
blocking transmission of the voice by the communication unit or reducing the volume of the voice at least when it is detected that the line of sight of the speaker is out of the camera;
A video call device control program that runs

Images