JP2015012557A - Video audio processor, video audio processing system, video audio synchronization method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform lip-sync of a video and audio without using additional information such as a time stamp.SOLUTION: A video audio processor includes: a video processing part for inputting a video; an audio processing part for inputting an audio; a video conversation determination part for detecting the face from the video input to the video processing part, and for performing the dynamic/static determination of the lip area of the detected face to detect that a conversation has started on the video; a voice conversation determination part for performing the volume determination of the audio input to the audio processing part to detect that a conversation has started on the audio; and a video audio synchronization part for, on the basis of a difference between the on-video conversation start timing in which it is detected that the conversation has started on the video by the video conversation determination part and the on-voice conversation start timing in which it is detected that the conversation has started on the audio by the audio conversation determination part, adjusting the output timing of the video and audio.

Description

  The present invention relates to a technique for synchronizing video and audio (lip sync).

  In recent years, customers who use video conferencing have a need to use a satellite communication network (bandwidth: 64 kbps) as a part of BCP (Business Continuity Plan) for disaster countermeasures on a specific agenda. However, the bandwidth of the satellite communication network was clearly insufficient to satisfy the communication quality required by customers for both video and audio.

  Here, the above problem can be solved by preparing separate networks for video and audio transmission / reception. For example, it is conceivable to use a satellite communication network as originally scheduled for video and to use a satellite telephone network for audio.

  However, when video and audio transmission / reception networks are prepared separately, there is a time difference in video and audio input from each network on the receiving side due to different communication delays on each network. There is a case.

  For this reason, when video and audio transmission / reception networks are separately prepared in a video conference, the receiving side synchronizes the video and audio (lip sync) so that the movement and audio of the speaker's mouth on the video are synchronized. It is necessary to match.

  As related technologies for synchronizing video and audio, there are technologies described in Patent Documents 1 to 3.

  In the technique described in Patent Document 1, time stamps added to video and audio are compared to synchronize the playback timing of video and audio. The technique of using additional information such as a time stamp for synchronization as described above is a general technique.

  In the technique described in Patent Document 2, when movement of the mouth is detected in an image obtained by capturing the face of a participant in a video conference, an audio signal is generated based on a signal from a microphone corresponding to the participant. And the enlarged image of the participant is displayed.

  In the technique described in Patent Document 3, the transmitting side has valid audio data, a predetermined amount of movement is detected in the lip video data, and the speaker holds the right to speak. In this case, the audio data and utterance information are transmitted, and the video data encoding method is changed. The receiving side decodes and displays the video data, and selectively synthesizes only the audio data from the terminal device indicated by the utterance information and outputs the audio.

JP 2000-324163 A JP 2004-118314 A JP 2002-158983 A

  However, since the technique described in Patent Document 1 needs to compare time stamps of video and audio, the processing device that processes video and audio is the same, or the processing device that processes video and audio are the same. It is necessary to prepare a processing device for processing separately and exchange time stamps between the processing devices in real time.

  Further, the technique described in Patent Document 1 cannot be applied to a satellite telephone network that cannot add a time stamp to voice.

  The technique described in Patent Document 2 synchronizes the timing when a participant speaks and the timing when the participant's image is enlarged and displayed, and does not perform lip sync between video and audio.

  In the technique described in Patent Document 2, the synchronization timing is the timing before video and audio are transmitted to the network, and therefore communication delays occurring on the network cannot be absorbed.

  The technique described in Patent Document 3 synchronizes the timing of speaking by a speaker holding the right to speak and the timing of changing the encoding method of video data, and does not perform video and audio lip synchronization. Absent.

  As described above, the technique described in Patent Document 1 has a problem that it is necessary to use additional information such as a time stamp, and the techniques described in Patent Documents 2 and 3 provide a lip sync between video and audio. There is a problem that it cannot be done.

  Accordingly, an object of the present invention is to solve the above-described problems and provide a technique capable of performing lip sync between video and audio without using additional information such as a time stamp.

The video / audio processing apparatus of the present invention comprises:
A video processing unit for inputting video;
A voice processing unit for inputting voice;
A video conversation determination unit for detecting a face from the video input to the video processing unit and detecting a conversation on the video by performing a motion determination of the detected lip area;
A voice conversation determination unit that detects the start of a conversation on the voice by performing a volume determination of the voice input to the voice processing unit;
The difference between the on-video conversation start timing when the video conversation determination unit detects that the conversation has started on the video and the on-speech conversation start timing when the voice conversation determination unit detects that the conversation has started on the voice And a video / audio synchronization unit that adjusts the output timing of the video and the audio.

The video / audio processing system of the present invention includes:
A video processing unit for inputting video;
A voice processing unit for inputting voice;
A video conversation determination unit for detecting a face from the video input to the video processing unit and detecting a conversation on the video by performing a motion determination of the detected lip area;
A voice conversation determination unit that detects the start of a conversation on the voice by performing a volume determination of the voice input to the voice processing unit;
The difference between the on-video conversation start timing when the video conversation determination unit detects that the conversation has started on the video and the on-speech conversation start timing when the voice conversation determination unit detects that the conversation has started on the voice And a video / audio synchronization unit that adjusts the output timing of the video and the audio.

The video / audio synchronization method of the present invention includes:
A video / audio synchronization method by a video / audio processing apparatus,
A video processing step for inputting video;
A voice processing step for inputting voice;
A video conversation determination step of detecting a face from the video input in the video processing step and detecting a conversation on the video by performing a motion determination of the detected lip region;
A voice conversation determination step of detecting that a conversation is started on the voice by performing a volume determination of the voice input in the voice processing step;
A conversation start timing on the video when the conversation start on the video is detected in the video conversation determination step, and a voice conversation start timing when the conversation is started on the voice in the voice conversation determination step; And a video / audio synchronization step for adjusting an output timing of the video and the audio based on the difference between the video and audio.

The program of the present invention
On the computer,
Video processing procedure for inputting video,
Audio processing procedure for inputting audio,
A video conversation determination procedure for detecting a face from the video input in the video processing procedure and detecting a conversation on the video by performing a motion determination of the detected lip area;
A voice conversation determination procedure for detecting that a conversation is started on the voice by performing a volume determination of the voice input in the voice processing procedure;
A conversation start timing on the video when the conversation start on the video is detected in the video conversation determination procedure, and a voice conversation start timing when the conversation is started on the voice in the voice conversation determination procedure; And a video / audio synchronization procedure for adjusting the output timing of the video and the audio based on the difference between the video and the audio.

  According to the present invention, it is possible to perform an lip sync between video and audio without using additional information such as a time stamp.

It is a figure which shows the structure of the video / audio processing system of the 1st Embodiment of this invention. 2 is a block diagram showing a configuration of a video conference transceiver 200. FIG. FIG. 11 is a sequence diagram showing an operation of the video conference transceiver 200. It is a figure which shows the synchronizing operation | movement in the audio | video audio | voice synchronizer 233 in case audio | voice input and video input are simultaneous. It is a figure which shows the synchronizing operation | movement in the video audio | voice synchronizer 233 when the input delay of the image | video with respect to the audio | voice input is settled in the range of a buffer length. It is a figure which shows the synchronizing operation | movement when the video input delay with respect to the audio | voice input in the video / audio synchronizer 233 does not fit in the buffer length range. It is a figure which shows the synchronizing operation | movement when the video input delay with respect to the audio | voice input in the video / audio synchronizer 233 does not fit in the buffer length range. It is a figure which shows the synchronizing operation | movement when the video input delay with respect to the audio | voice input in the video / audio synchronizer 233 does not fit in the buffer length range.

EMBODIMENT OF THE INVENTION Below, the form for implementing this invention is demonstrated with reference to drawings.
(1) First Embodiment This embodiment is an example in which the present invention is applied to a video conference.
(1-1) Configuration of First Embodiment First, the configuration of the present embodiment will be described.

  FIG. 1 shows an overview of the entire video / audio processing system of this embodiment.

  The video / audio processing system of this embodiment includes video conference transceivers 100 and 200 as video / audio processing apparatuses.

  The video conference transceivers 100 and 200 transmit and receive video and audio of a video conference in real time in both directions. The video network 1 is used for video transmission and reception, and the audio network 2 is used for audio transmission and reception.

  For example, in the transmission of video and audio from the video conference transceiver 100 to the video conference transceiver 200, the video conference transceiver 200 inputs video from the video network 1 and inputs audio from the audio network 2. It detects that a time difference may have occurred, and adjusts the output timing of video and audio.

  In the present embodiment, a fixed telephone network having a relatively small communication delay is used for the audio network 2, and an IP (internet protocol) network having a wide bandwidth but a large communication delay is used for the video network 1. Shall be used. Under this condition, it is considered that video tends to be input later than audio.

  Also, in this embodiment, it is considered that audio responsiveness is important in video conferencing, and when video input is greatly delayed with respect to audio input, avoid adjusting the video and audio output timing. Assume that only audio is output first.

  FIG. 2 shows a configuration of the video conference transceiver 200. Note that FIG. 2 shows only components that perform video and audio input processing from the network and subsequent components among the components of the video conference transceiver 200, and other components are omitted. The configuration of the video conference transceiver 100 is the same as that of the video conference transceiver 200.

  The video conference transceiver 200 includes a video processing unit 210, an audio processing unit 220, and a video / audio synchronization processing unit 230.

  The video processing unit 210 has a role of terminating communication with the video conference transmitter / receiver 100 via the video network 1 and inputting the video of the video conference transmitted from the video conference transmitter / receiver 100 from the video network 1.

  The audio processing unit 220 has a role of terminating communication with the video conference transceiver 100 via the audio network 2 and inputting the audio of the video conference transmitted from the video conference transceiver 100 from the audio network 2.

  The video / audio synchronization processing unit 230 includes three components: a conversation determination unit (video) 231, a conversation determination unit (sound) 232, and a video / audio synchronization unit 233.

  The conversation determination unit (video) 231 detects that a conversation has started on the video input to the video processing unit 210.

  The conversation determination unit (voice) 232 detects that a conversation has started on the voice input to the voice processing unit 220.

  The video / audio synchronization unit 233 has a timing (conversation start timing on video) when the conversation determination unit (video) 231 detects that a conversation is started on the video and a conversation determination unit (sound) 232 starts conversation on the audio. On the basis of the difference from the timing at which this is detected (speech start timing on speech), the video and audio output timings are adjusted so that these conversation start timings match.

  Here, the video / audio synchronization unit 233 includes a video buffer and an audio buffer for temporarily storing video and audio in order to adjust the output timing.

Note that the video / audio synchronization unit 233 outputs video to a video playback device such as a monitor, and outputs audio to a voice playback device such as a speaker.
(1-2) Operation of the First Embodiment Next, the operation of the present embodiment will be described.
(A) Operation Sequence of Video Conference Transceiver 200 First, the operation sequence of the video conference transceiver 200 will be described.

FIG. 3 shows an operation sequence of the video conference transceiver 200. FIG. 3 shows an operation when a video is input with a delay compared to audio.
<A: Video>
First, an operation from when a video is input to when it is detected that conversation has started on the video will be described.

Step a-1:
First, the video processing unit 210 inputs the video of the video conference transmitted from the video conference transceiver 100 from the video network 1.

Step a-2:
Next, the video processing unit 210 passes the video input above to the conversation determination unit (video) 231.

Step a-3:
Next, the conversation determination unit (video) 231 performs conversation determination on the content of the video passed from the video processing unit 210 and detects that the conversation is started on the video.

  The conversation determination is realized by detecting a face from the video and determining the movement of the lip area of the detected face. A state where there is a movement exceeding the threshold in the lip region static determination is regarded as a state in which the speaker is speaking, and it is detected that the conversation is started when the threshold is exceeded.

Step a-4:
When the conversation determination unit (video) 231 detects that the conversation is started on the video, it notifies the video / audio synchronization unit 233 of the fact. This notification is expressed by one parameter (0/1 flag) of a function to be called when the video is transferred from the conversation determination unit (video) 231 to the video / audio synchronization unit 233. Regardless of the presence or absence of this notification, the conversation determination unit (video) 231 outputs video to the video / audio synchronization unit 233 as needed.
<B: Voice>
Next, an operation from when a voice is input until it is detected that a conversation is started on that voice will be described.

Step b-1:
First, the audio processing unit 220 inputs the audio of the video conference transmitted from the video conference transceiver 100 from the audio network 2.

Step b-2:
Next, the voice processing unit 220 passes the voice input above to the conversation determination unit (voice) 232.

Step b-3:
Next, the conversation determination unit (speech) 232 performs conversation determination on the content of the voice passed from the voice processing unit 220 and detects that the conversation is started on the voice.

  In the conversation determination, the sound volume is determined, the state where the volume exceeds the threshold value is regarded as the state where the speaker is speaking, and it is detected that the conversation is started when the threshold value is exceeded.

  In addition, in order to prevent false detection of surrounding sounds and noise as speech, human speech is extracted in advance using the frequency band of human speech, and then the above conversation determination is performed on the extracted speech. Shall be performed.

Step b-4:
When the conversation determining unit (sound) 232 detects that the conversation is started on the sound, the conversation determining unit (sound) 232 notifies the video / audio synchronizing unit 233 to that effect. This notification is expressed by one parameter (0/1 flag) of a function to be called when the voice is transferred from the conversation determination unit (voice) 232 to the video / audio synchronization unit 233. Regardless of the presence or absence of this notification, the conversation determination unit (audio) 232 outputs audio to the video / audio synchronization unit 233 as needed.
<C: Audio / video synchronization>
Next, an operation for synchronizing video and audio will be described.

Step c-1:
The audio / video synchronization unit 233 inputs the audio from the conversation determination unit (audio) 232 as needed, accumulates the input audio in the audio buffer, and inputs the video from the conversation determination unit (video) 231 as needed. Is stored in the video buffer.

  The video / audio synchronization unit 233 adjusts the video and audio output timing when the video input delay with respect to the audio input falls within a preset allowable time (for example, T seconds).

  Therefore, the buffer lengths of the audio buffer and the video buffer are set to a length corresponding to the above-described allowable time (for example, set to 6 frames for 60 fps).

  In addition, the video / audio synchronization unit 233 receives a notification from the conversation determination unit (audio) 232 that the conversation has started on the voice, and the video / audio synchronization unit 233 receives the video from the conversation determination unit (video) 231. Get notified that the conversation has started.

  Here, the video / audio synchronization unit 233, when the video input delay with respect to the audio input (that is, the delay in the video conversation start timing with respect to the voice conversation start timing) falls within the buffer length range, Adjust the output timing, and simultaneously output the video frame (conversation start video frame) when the conversation starts on the video and the voice frame (conversation start audio frame) when the conversation starts on the audio. To do.

  On the other hand, if the video input delay with respect to the audio input does not fall within the buffer length range, the video / audio synchronization unit 233 outputs the audio of the conversation start audio frame first without adjusting the output timing. To.

Step c-2:
The video / audio synchronization unit 233 adjusts the video and audio output timing when the video input delay with respect to the audio input falls within the buffer length range, and simultaneously outputs the audio of the conversation start audio frame and the video of the conversation start video frame. Output. At this time, the video of the video frame from the top of the video buffer to immediately before the conversation start video frame is skipped.

On the other hand, when the video input delay with respect to the audio input does not fall within the buffer length range, the video / audio synchronization unit 233 outputs the audio of the conversation start audio frame first without adjusting the output timing.
(B) Specific Example of Video and Audio Synchronization Operation Next, the video and audio synchronization operation in the video and audio synchronization unit 233 will be described with a specific example.

  4 to 8 show the contents of the video buffer and the audio buffer, and the contents of the synchronization operation in this state.

  Here, the video / audio synchronization unit 233 treats the same number of frames per second for video and audio (in this case, frame conversion is performed to make the number of frames the same if necessary), and the video frame and audio Both frames start from the first frame and the conversation starts at the Nth frame.

4 to 8, the upper part shows the contents of the audio buffer and the lower part shows the contents of the video buffer. Each buffer receives (stores) data from the left and outputs from the right (FIFO (First In First Out)). Assume that it is a buffer. A square in each buffer indicates frame data. Moreover, the character described in each data means what frame the data is, for example, the data with N is the Nth frame. Further, the display is shaded on the Nth frame where the conversation is started, and the frame of the frame output at that time is displayed thickly.
(B-1) When Audio Input and Video Input are Simultaneous FIG. 4 shows a synchronization operation when audio input and video input are simultaneous.

  In FIG. 4, both the audio buffer and the video buffer store Nth frame data at the beginning of the conversation.

  Therefore, the video / audio synchronization unit 233 detects that there is no video input delay.

  Therefore, the video / audio synchronization unit 233 outputs data of the first N frames of the buffer for both audio and video.

In this output result, audio and video are synchronized.
(B-2) When video input is delayed with respect to audio input (B-2-1) When video input delay is within the buffer length range Video input is delayed with respect to audio input However, if the video input delay falls within the buffer length range, the audio and video can be synchronized.

  FIG. 5 shows the synchronization operation in this case.

  In FIG. 5, the Nth frame of audio data when conversation is started is stored at the beginning of the audio buffer. However, in the video buffer, the Nth frame of video data is stored in addition to the beginning.

  Therefore, the video / audio synchronization unit 233 detects a video input delay.

  Therefore, the video / audio synchronization unit 233 outputs the audio data of the first N frames of the audio buffer, and skips the video data from the N-3th frame to the (N-1) th frame of the first buffer. The video data of the Nth frame is output.

In this output result, audio and video are synchronized.
(B-2-2) When the video input delay is not within the buffer length range When the video input is delayed with respect to the audio input, and the video input delay is not within the buffer length range, The video input delay exceeds the allowable time, and the audio and video cannot be synchronized.

  6 to 8 show the synchronization operation in this case.

  FIG. 6 shows an operation when the conversation is started on the voice but the conversation is not yet started on the video.

  Since the audio data of the Nth frame when the conversation is started is stored at the head of the audio buffer, the video / audio synchronization unit 233 knows that the conversation is started on the audio, but the video buffer No video data for the Nth frame has been stored yet.

  Therefore, the video / audio synchronization unit 233 does not detect a video input delay.

  Therefore, the video / audio synchronization unit 233 outputs the first N frames of audio data in the audio buffer, and outputs the first N−8th frame of video data in the video buffer.

  Here, the reason why the video / audio synchronization unit 233 does not detect the delay of the video is when the start of the conversation on the audio is erroneously detected or when the start of the conversation on the video is not erroneously detected. It is for preparing.

  In this output result, the audio and the video are not synchronized, and the video is delayed by 8 frames from the audio.

  FIG. 7 shows a case where conversation is started even on the video after the operation of FIG. 6 (when the video data of the Nth frame when the conversation is started on the video is input to the video / audio synchronization unit 233). The operation is shown.

  The video / audio synchronization unit 233 detects the video input delay at the timing when the video data of the Nth frame is input from the conversation determination unit (video) 231.

  Therefore, the video / audio synchronization unit 233 outputs the audio data of the first N + 3th frame of the audio buffer, and skips the video data from the first N-5th frame to the N−1th frame of the video buffer. The video data of the Nth frame is output.

  In this output result, the audio and the video cannot be synchronized, and the video is delayed by 3 frames from the audio (however, the delay of 5 frames is improved compared to FIG. 6).

  FIG. 8 shows an operation when a plurality of video frames are collected and input to the video / audio synchronization unit 233 after the operation of FIG. 7 to improve the video delay.

  Since the video / audio synchronization unit 233 detects the video input delay, it tries to improve the video input delay as much as possible.

  Therefore, the video / audio synchronization unit 233 outputs the first N + 10th frame of audio data in the audio buffer, skips the first N + 7th frame to the N + 8th frame of the video buffer, and outputs N + 9th frame. Outputs the eye image data.

  In this output result, the audio is not synchronized with the video, and the video is delayed by one frame from the audio (however, the delay of two frames is improved as compared with FIG. 7).

  The video / audio synchronization unit 233 skips the video data of the first video frame among the plurality of video frames when the plurality of video frames are input next time. Thereby, the sound and the video are synchronized.

In this manner, the video / audio synchronization unit 233 receives X (X is a natural number of 2 or more) video frames as a whole until the video and audio are synchronized, and among the X video frames, When the video of the first video frame is accumulated at the top of the video buffer, the video of Y (Y is a natural number less than X-1) video frames from the top of the video buffer is skipped and the remaining video frames are skipped. The video of the first video frame among the video frames is output.
(1-3) Effects of the First Embodiment Next, effects of the present embodiment will be described.
(1-3-1) It is not necessary to use additional information such as a time stamp. In the present embodiment, the video / audio synchronization processing unit 230 detects a face from the video, and determines the movement of the detected lip region of the face. A conversation determination unit (video) 231 that detects that the conversation has started on the video, and a conversation determination unit (video) 231 that detects that the conversation has started on the voice by performing the sound volume determination. Voice) 232 and conversation determination unit (video) 231 detects the start of conversation on the video and conversation determination unit (voice) 232 detects that conversation has started on the voice. And a video / audio synchronizer 233 that adjusts the video and audio output timing based on the difference from the voice conversation start timing.

  As described above, in this embodiment, since the conversation start timing is detected from the video and audio itself, additional information such as a time stamp is not necessary.

  Therefore, without using additional information, the video and audio can be synchronized (lip sync), and the movement of the mouth of the speaker on the video can be matched with the audio.

  In addition, since additional information is not used, the network and processing device to be used can be applied regardless of the type.

For example, even when a failure occurs in the network or processing device to be used in the event of a disaster, lip sync can be realized simply by changing the processing device.
(1-3-2) A point where an existing processing device can be used without modification In the present embodiment, as described above, a video / audio synchronization process which is a third party other than the video processing unit 210 and the audio processing unit 220. The unit 230 realizes lip sync.

  Therefore, the video processing unit 210 and the audio processing unit 220 can be physically separated from each other, and both need not be aware of each other.

  As a result, it is possible to use an existing processing device (a fixed phone, a mobile phone, etc.) without modification.

In addition, by using an existing processing device, there is no need to newly acquire a network usage authorization or the like.
(1-3-3) The point that the combination of a network and a processing apparatus is free In this embodiment, since the existing processing apparatus can be diverted as above-mentioned, the existing processing apparatus which can utilize a specific network By combining and using, it is possible to cope with a plurality of network types.

  For example, in a telephone network, ISDN (Integrated Services Digital Network), NGN (Next Generation Network), satellite telephone network, etc., protocols and communication procedures are limited, but these networks can also be supported.

  Further, although the protocol format is generalized in the IP network, it can be applied to the IP network.

In addition, since it is possible to cope with a plurality of network types, system availability is also improved.
(1-3-4) A point where synchronization is possible with respect to real-time delay fluctuations in the network In the present embodiment, the start of conversation is a trigger for synchronization, and synchronization is re-established every time a synchronization trigger occurs. It is also possible to cope with real-time delay fluctuations on the network.

For example, in mobile communications and communications with foreign countries, there are cases where the delay amount varies due to the quality of the network, but it is possible to cope with such a variation in delay amount.
(2) Other Embodiments Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

  For example, in the above embodiment, the case where the present invention is applied to a video conference has been described as an example, but the present invention is not limited to this. The following is an example to which the present invention can be applied.

・ Web conference ・ Video distribution (VOD (Video On Demand) etc.)
・ Video editing on multimedia processing device In multimedia processing devices (video player, PC (Personal Computer), etc.), when editing a video file (such as tethering or TV commercial message cut) If a part or all of the time stamp of the file is lost, it becomes impossible to synchronize video and audio using the time stamp during playback. Even in such a case, the present invention makes it possible to synchronize video and audio.

  Thus, the present invention is also useful for a processing apparatus that does not use a network.

  In the above-described embodiment, an example in which a fixed telephone network is used for the audio network 2 and an IP network is used for the video network 1 has been described. However, the present invention can be applied regardless of the network type. is there.

  For example, a bandwidth guarantee type network or a network having a path control function such as OpenFlow can be used. Even if audio and video are communicated on these same networks, audio that requires higher real-time performance due to priority control and autonomous control of routing, etc. is handled with priority over video, There may be a time difference. Also in this case, it is possible to synchronize video and audio according to the present invention. In addition, since the configuration and operation in this case are exactly the same as those in the above embodiment, a detailed description is omitted here.

  In the above embodiment, the video processing unit 210, the audio processing unit 220, and the video / audio synchronization processing unit 230 are arranged in the same processing device. However, these processing units may be provided as separate processing devices. good.

  For example, an existing fixed telephone, a mobile phone, or the like can be used for the audio processing unit 220, which makes it possible to effectively use an existing video / audio related processing device.

  In the above embodiment, video and audio are synchronized when the conversation starts, but in addition to this, when the conversation ends, sentences and word breaks are analyzed, and these are also triggered as a video. And audio may be synchronized.

  By increasing the trigger of synchronization in this way, it is possible to prevent the synchronization once taken due to the fluctuation of the network or the like from shifting with time.

  In the above embodiment, one-to-one communication has been described as an example. However, the present invention can also be applied to one-to-many and many-to-many communication.

  In the above embodiment, the processing device that terminates video and audio of a video conference is taken as an example, but video and audio are relayed as in a video conference MCU (Multi point control unit). The present invention can also be applied to a processing apparatus.

  Moreover, although the example which divided | segmented the network utilized by each of an image | video and an audio | voice was given in the said embodiment, this invention is applicable also when using the same network for an image | video and an audio | voice.

  In the above embodiment, an example in which the video frame is skipped in step c-2 in FIG. 3 has been described. Instead, the video frame is discarded or the video frame is temporarily fast-forwarded. Also good.

  In the above embodiment, a flag is set in the calling function in steps a-4 and b-4 of FIG. 3, but instead of one of video and audio data (H.264, AAC, etc.). Using a region, data at a specific location may be handled as a flag.

  In the above embodiment, the number of frames per second for video and audio is considered to be the same in FIGS. 4 to 8, but this need not necessarily be the case. That is, the output timing is not adjusted based on the frame, but the output timing is adjusted based on the time length from the start of the moving image, so that finer adjustment is possible.

  The method performed by the video / audio processing apparatus of the present invention may be applied to a program for causing a computer to execute the method. In addition, the program can be stored in a storage medium and can be provided to the outside via a network.

In addition, although a part or all of said embodiment can be described also as the following additional remarks, it is not restricted to the following.
(Appendix 1)
A video processing unit for inputting video;
A voice processing unit for inputting voice;
A video conversation determination unit for detecting a face from the video input to the video processing unit and detecting a conversation on the video by performing a motion determination of the detected lip area;
A voice conversation determination unit that detects the start of a conversation on the voice by performing a volume determination of the voice input to the voice processing unit;
The difference between the on-video conversation start timing when the video conversation determination unit detects that the conversation has started on the video and the on-speech conversation start timing when the voice conversation determination unit detects that the conversation has started on the voice And a video / audio synchronization unit that adjusts an output timing of the video and the audio based on the video and audio.
(Appendix 2)
The video / audio synchronization unit includes:
The video according to appendix 1, wherein the video and the audio output timing are adjusted when a difference between the video conversation start timing and the voice conversation start timing falls within a preset allowable time range. Audio processing device.
(Appendix 3)
The video / audio synchronization unit includes:
A video buffer that is set to a buffer length corresponding to the allowable time, and temporarily stores the video;
An audio buffer that is set to a buffer length corresponding to the allowable time and temporarily stores the audio;
The voice of the conversation start voice frame corresponding to the voice conversation start timing is accumulated at the beginning of the voice buffer, and the video of the conversation start video frame corresponding to the video conversation start timing is other than the head of the video buffer. If stored, the audio of the conversation start audio frame is output, and the video of the video frame from the beginning of the video buffer to the immediately before the conversation start video frame is skipped, deleted, or fast-forwarded to start the conversation The video / audio processing apparatus according to appendix 2, which outputs video of a video frame.
(Appendix 4)
The video / audio synchronization unit includes:
When the audio of the conversation start audio frame is accumulated at the head of the audio buffer and the video of the conversation start video frame is not accumulated in the video buffer, the audio of the conversation start audio frame is output, and Output the video of the first video frame in the video buffer,
When the video of the conversation start video frame is accumulated in the video buffer after outputting the audio of the conversation start audio frame, the audio of the first audio frame of the audio buffer is output and the video buffer Skips, deletes, or fast forwards the video frame video from the beginning of the video frame immediately before the conversation start video frame, and outputs the video of the conversation start video frame,
Thereafter, X (X is a natural number of 2 or more) video frames are input until the video and audio are synchronized, and the video of the first video frame among the X video frames is used for the video. When accumulated at the beginning of the buffer, the audio of the first audio frame of the audio buffer is output and Y (Y is a natural number equal to or less than X-1) video frames from the beginning of the video buffer. 4. The video / audio processing apparatus according to appendix 3, wherein the video is skipped, deleted, or fast-forwarded and the video of the first video frame among the remaining video frames is output.
(Appendix 5)
A video processing unit for inputting video;
A voice processing unit for inputting voice;
A video conversation determination unit for detecting a face from the video input to the video processing unit and detecting a conversation on the video by performing a motion determination of the detected lip area;
A voice conversation determination unit that detects the start of a conversation on the voice by performing a volume determination of the voice input to the voice processing unit;
The difference between the on-video conversation start timing when the video conversation determination unit detects that the conversation has started on the video and the on-speech conversation start timing when the voice conversation determination unit detects that the conversation has started on the voice And a video / audio synchronization system that adjusts an output timing of the video and the audio based on the video and audio.
(Appendix 6)
The video / audio synchronization unit includes:
The video according to appendix 5, wherein the video and the audio output timing are adjusted when a difference between the video conversation start timing and the voice conversation start timing falls within a preset allowable time range. Voice processing system.
(Appendix 7)
The video / audio synchronization unit includes:
A video buffer that is set to a buffer length corresponding to the allowable time, and temporarily stores the video;
An audio buffer that is set to a buffer length corresponding to the allowable time and temporarily stores the audio;
The voice of the conversation start voice frame corresponding to the voice conversation start timing is accumulated at the beginning of the voice buffer, and the video of the conversation start video frame corresponding to the video conversation start timing is other than the head of the video buffer. If stored, the audio of the conversation start audio frame is output, and the video of the video frame from the beginning of the video buffer to the immediately before the conversation start video frame is skipped, deleted, or fast-forwarded to start the conversation The video / audio processing system according to appendix 6, which outputs video of a video frame.
(Appendix 8)
The video / audio synchronization unit includes:
When the audio of the conversation start audio frame is accumulated at the head of the audio buffer and the video of the conversation start video frame is not accumulated in the video buffer, the audio of the conversation start audio frame is output, and Output the video of the first video frame in the video buffer,
When the video of the conversation start video frame is accumulated in the video buffer after outputting the audio of the conversation start audio frame, the audio of the first audio frame of the audio buffer is output and the video buffer Skips, deletes, or fast forwards the video frame video from the beginning of the video frame immediately before the conversation start video frame, and outputs the video of the conversation start video frame,
Thereafter, X (X is a natural number of 2 or more) video frames are input until the video and audio are synchronized, and the video of the first video frame among the X video frames is used for the video. When accumulated at the beginning of the buffer, the audio of the first audio frame of the audio buffer is output and Y (Y is a natural number equal to or less than X-1) video frames from the beginning of the video buffer. The video / audio processing system according to appendix 7, wherein the video is skipped, deleted, or fast-forwarded and the video of the first video frame among the remaining video frames is output.
(Appendix 9)
A video / audio synchronization method by a video / audio processing apparatus,
A video processing step for inputting video;
A voice processing step for inputting voice;
A video conversation determination step of detecting a face from the video input in the video processing step and detecting a conversation on the video by performing a motion determination of the detected lip region;
A voice conversation determination step of detecting that a conversation is started on the voice by performing a volume determination of the voice input in the voice processing step;
A conversation start timing on the video when the conversation start on the video is detected in the video conversation determination step, and a voice conversation start timing when the conversation is started on the voice in the voice conversation determination step; And a video / audio synchronization step of adjusting an output timing of the video and the audio based on a difference between the video and audio.
(Appendix 10)
In the video / audio synchronization step,
The video according to appendix 9, wherein the output timing of the video and the audio is adjusted when a difference between the video conversation start timing and the voice conversation start timing falls within a preset allowable time range. Audio processing method.
(Appendix 11)
The video / audio processing apparatus includes:
A video buffer that is set to a buffer length corresponding to the allowable time, and temporarily stores the video;
An audio buffer that is set to a buffer length corresponding to the allowable time and temporarily stores the audio;
In the video / audio synchronization step,
The voice of the conversation start voice frame corresponding to the voice conversation start timing is accumulated at the beginning of the voice buffer, and the video of the conversation start video frame corresponding to the video conversation start timing is other than the head of the video buffer. If stored, the audio of the conversation start audio frame is output, and the video of the video frame from the beginning of the video buffer to the immediately before the conversation start video frame is skipped, deleted, or fast-forwarded to start the conversation The video / audio processing method according to appendix 10, wherein the video of the video frame is output.
(Appendix 12)
In the video / audio synchronization step,
When the audio of the conversation start audio frame is accumulated at the head of the audio buffer and the video of the conversation start video frame is not accumulated in the video buffer, the audio of the conversation start audio frame is output, and Output the video of the first video frame in the video buffer,
When the video of the conversation start video frame is accumulated in the video buffer after outputting the audio of the conversation start audio frame, the audio of the first audio frame of the audio buffer is output and the video buffer Skips, deletes, or fast forwards the video frame video from the beginning of the video frame immediately before the conversation start video frame, and outputs the video of the conversation start video frame,
Thereafter, X (X is a natural number of 2 or more) video frames are input until the video and audio are synchronized, and the video of the first video frame among the X video frames is used for the video. When accumulated at the beginning of the buffer, the audio of the first audio frame of the audio buffer is output and Y (Y is a natural number equal to or less than X-1) video frames from the beginning of the video buffer. The video / audio processing method according to attachment 11, wherein the video is skipped, deleted, or fast-forwarded, and the video of the first video frame among the remaining video frames is output.
(Appendix 13)
On the computer,
Video processing procedure for inputting video,
Audio processing procedure for inputting audio,
A video conversation determination procedure for detecting a face from the video input in the video processing procedure and detecting a conversation on the video by performing a motion determination of the detected lip area;
A voice conversation determination procedure for detecting that a conversation is started on the voice by performing a volume determination of the voice input in the voice processing procedure;
A conversation start timing on the video when the conversation start on the video is detected in the video conversation determination procedure, and a voice conversation start timing when the conversation is started on the voice in the voice conversation determination procedure; And a video / audio synchronization procedure for adjusting the output timing of the video and the audio based on the difference between them.
(Appendix 14)
In the video / audio synchronization procedure,
The program according to appendix 13, wherein the output timing of the video and the audio is adjusted when a difference between the video conversation start timing and the voice conversation start timing falls within a preset allowable time range. .
(Appendix 15)
The computer
A video buffer that is set to a buffer length corresponding to the allowable time, and temporarily stores the video;
An audio buffer that is set to a buffer length corresponding to the allowable time and temporarily stores the audio;
In the video / audio synchronization procedure,
The voice of the conversation start voice frame corresponding to the voice conversation start timing is accumulated at the beginning of the voice buffer, and the video of the conversation start video frame corresponding to the video conversation start timing is other than the head of the video buffer. If stored, the audio of the conversation start audio frame is output, and the video of the video frame from the beginning of the video buffer to the immediately before the conversation start video frame is skipped, deleted, or fast-forwarded to start the conversation The program according to appendix 14, which outputs a video of a video frame.
(Appendix 16)
In the video / audio synchronization procedure,
When the audio of the conversation start audio frame is accumulated at the head of the audio buffer and the video of the conversation start video frame is not accumulated in the video buffer, the audio of the conversation start audio frame is output, and Output the video of the first video frame in the video buffer,
When the video of the conversation start video frame is accumulated in the video buffer after outputting the audio of the conversation start audio frame, the audio of the first audio frame of the audio buffer is output and the video buffer Skips, deletes, or fast forwards the video frame video from the beginning of the video frame immediately before the conversation start video frame, and outputs the video of the conversation start video frame,
Thereafter, X (X is a natural number of 2 or more) video frames are input until the video and audio are synchronized, and the video of the first video frame among the X video frames is used for the video. When accumulated at the beginning of the buffer, the audio of the first audio frame of the audio buffer is output and Y (Y is a natural number equal to or less than X-1) video frames from the beginning of the video buffer. The program according to appendix 15, wherein the video is skipped, deleted, or fast-forwarded and the video of the first video frame among the remaining video frames is output.

  The present invention can be used in general multimedia fields using video and audio.

  In addition, the present invention can be used regardless of the necessity of bidirectional communication or the necessity of real-time performance.

DESCRIPTION OF SYMBOLS 1 Video network 2 Audio network 100,200 Video conference transmitter / receiver 210 Video processing unit 220 Audio processing unit 230 Video / audio synchronization processing unit 231 Conversation determination unit (video)
232 Conversation determination unit (voice)
233 Video / audio synchronization unit

Claims (10)

A video processing unit for inputting video;
A voice processing unit for inputting voice;
A video conversation determination unit for detecting a face from the video input to the video processing unit and detecting a conversation on the video by performing a motion determination of the detected lip area;
A voice conversation determination unit that detects the start of a conversation on the voice by performing a volume determination of the voice input to the voice processing unit;
The difference between the on-video conversation start timing when the video conversation determination unit detects that the conversation has started on the video and the on-speech conversation start timing when the voice conversation determination unit detects that the conversation has started on the voice And a video / audio synchronization unit that adjusts an output timing of the video and the audio based on the video and audio. The video / audio synchronization unit includes:
The output timing of the video and the audio is adjusted when the difference between the video conversation start timing and the audio conversation start timing falls within a preset allowable time range. Video / audio processor. The video / audio synchronization unit includes:
A video buffer that is set to a buffer length corresponding to the allowable time, and temporarily stores the video;
An audio buffer that is set to a buffer length corresponding to the allowable time and temporarily stores the audio;
The voice of the conversation start voice frame corresponding to the voice conversation start timing is accumulated at the beginning of the voice buffer, and the video of the conversation start video frame corresponding to the video conversation start timing is other than the head of the video buffer. If stored, the audio of the conversation start audio frame is output, and the video of the video frame from the beginning of the video buffer to the immediately before the conversation start video frame is skipped, deleted, or fast-forwarded to start the conversation The video / audio processing apparatus according to claim 2, wherein the video / audio processing apparatus outputs video of a video frame. The video / audio synchronization unit includes:
When the audio of the conversation start audio frame is accumulated at the head of the audio buffer and the video of the conversation start video frame is not accumulated in the video buffer, the audio of the conversation start audio frame is output, and Output the video of the first video frame in the video buffer,
When the video of the conversation start video frame is accumulated in the video buffer after outputting the audio of the conversation start audio frame, the audio of the first audio frame of the audio buffer is output and the video buffer Skips, deletes, or fast forwards the video frame video from the beginning of the video frame immediately before the conversation start video frame, and outputs the video of the conversation start video frame,
Thereafter, X (X is a natural number of 2 or more) video frames are input until the video and audio are synchronized, and the video of the first video frame among the X video frames is used for the video. When accumulated at the beginning of the buffer, the audio of the first audio frame of the audio buffer is output and Y (Y is a natural number equal to or less than X-1) video frames from the beginning of the video buffer. The video / audio processing apparatus according to claim 3, wherein the video is skipped, deleted, or fast-forwarded, and the video of the first video frame among the remaining video frames is output. A video processing unit for inputting video;
A voice processing unit for inputting voice;
A video conversation determination unit for detecting a face from the video input to the video processing unit and detecting a conversation on the video by performing a motion determination of the detected lip area;
A voice conversation determination unit that detects the start of a conversation on the voice by performing a volume determination of the voice input to the voice processing unit;
The difference between the on-video conversation start timing when the video conversation determination unit detects that the conversation has started on the video and the on-speech conversation start timing when the voice conversation determination unit detects that the conversation has started on the voice And a video / audio synchronization system that adjusts an output timing of the video and the audio based on the video and audio. The video / audio synchronization unit includes:
6. The output timing of the video and the audio is adjusted when a difference between the on-video conversation start timing and the on-voice conversation start timing falls within a preset allowable time range. Video / audio processing system. The video / audio synchronization unit includes:
A video buffer that is set to a buffer length corresponding to the allowable time, and temporarily stores the video;
An audio buffer that is set to a buffer length corresponding to the allowable time and temporarily stores the audio;
The voice of the conversation start voice frame corresponding to the voice conversation start timing is accumulated at the beginning of the voice buffer, and the video of the conversation start video frame corresponding to the video conversation start timing is other than the head of the video buffer. If stored, the audio of the conversation start audio frame is output, and the video of the video frame from the beginning of the video buffer to the immediately before the conversation start video frame is skipped, deleted, or fast-forwarded to start the conversation The video / audio processing system according to claim 6, wherein the video / audio processing system outputs video of a video frame. The video / audio synchronization unit includes:
When the audio of the conversation start audio frame is accumulated at the head of the audio buffer and the video of the conversation start video frame is not accumulated in the video buffer, the audio of the conversation start audio frame is output, and Output the video of the first video frame in the video buffer,
When the video of the conversation start video frame is accumulated in the video buffer after outputting the audio of the conversation start audio frame, the audio of the first audio frame of the audio buffer is output and the video buffer Skips, deletes, or fast forwards the video frame video from the beginning of the video frame immediately before the conversation start video frame, and outputs the video of the conversation start video frame,
Thereafter, X (X is a natural number of 2 or more) video frames are input until the video and audio are synchronized, and the video of the first video frame among the X video frames is used for the video. When accumulated at the beginning of the buffer, the audio of the first audio frame of the audio buffer is output and Y (Y is a natural number equal to or less than X-1) video frames from the beginning of the video buffer. The video / audio processing system according to claim 7, wherein the video is skipped, deleted, or fast-forwarded and the video of the first video frame among the remaining video frames is output. A video / audio synchronization method by a video / audio processing apparatus,
A video processing step for inputting video;
A voice processing step for inputting voice;
A video conversation determination step of detecting a face from the video input in the video processing step and detecting a conversation on the video by performing a motion determination of the detected lip region;
A voice conversation determination step of detecting that a conversation is started on the voice by performing a volume determination of the voice input in the voice processing step;
A conversation start timing on the video when the conversation start on the video is detected in the video conversation determination step, and a voice conversation start timing when the conversation is started on the voice in the voice conversation determination step; And a video / audio synchronization step of adjusting an output timing of the video and the audio based on a difference between the video and audio. On the computer,
Video processing procedure for inputting video,
Audio processing procedure for inputting audio,
A video conversation determination procedure for detecting a face from the video input in the video processing procedure and detecting a conversation on the video by performing a motion determination of the detected lip area;
A voice conversation determination procedure for detecting that a conversation is started on the voice by performing a volume determination of the voice input in the voice processing procedure;
A conversation start timing on the video when the conversation start on the video is detected in the video conversation determination procedure, and a voice conversation start timing when the conversation is started on the voice in the voice conversation determination procedure; And a video / audio synchronization procedure for adjusting the output timing of the video and the audio based on the difference between them.

Images