JP6016277B2 - Audiovisual processing system, audiovisual processing method, and program - Google Patents

Description

  The present invention relates to an audiovisual processing system, an audiovisual processing method, and a program.

  In video communication devices and the like, there is often a demand for viewing by paying attention to a specific object (such as a person or an object) from an audiovisual signal. Regarding attention processing on a specific object, attention processing in a video signal is referred to as video attention processing, and attention processing in an audio signal is referred to as acoustic attention processing.

  An example of the video attention process will be described with reference to FIG. Assume that the original video frame 71 includes four objects (object A74, object B75, object C76, and object D77). An object is an object constituting a photographed video space, such as a person, a car, or a building. In the original video frame 71, rectangles indicating the positions of these objects are displayed by dotted lines. Here, the attention area in the original video frame 71 is a solid line rectangle indicated by the attention area 73. The attention processing image 72 is an example in which the image attention processing is performed on the attention region 73. The attention processing image 72 is an example in which the attention region is enlarged and displayed so that the display width is maximized.

  As an example of the acoustic attention process, there is a method of reproducing only the acoustic signal corresponding to the object of interest. In this case, in the original video frame 71, when the attention area 73 is designated, only the sound of the object D77 included therein is reproduced.

  By the attention processing as described above, it is possible to observe in detail only the region in which the viewer is interested.

  In order to realize attention processing for a specific object that emits sound in a video, it is necessary to extract a video object as a sound source and separate the sound and video generated by each object. An example relating to such a technique is described in Patent Document 1.

  According to Patent Document 1, when a specific camera image is selected as a region of interest using a plurality of cameras and a plurality of microphones in a TV conference system, only a microphone located near the image is operated, so that it is suitable for the region of interest. Recording and playback of recorded audio is described. That is, by fixing a specific camera and microphone to one object, the video object and the sound object are associated with each other and separated.

  As a related technique, there is an object separation method using sound source direction detection. FIG. 19 shows an object separation apparatus related to FIG.

  This object separation device includes a video object separation unit 901 and a sound source direction detection unit 902. The video signal is input to the video object separation unit 901, and the audio signal is input to the sound source direction detection unit 902.

  The sound source direction detection unit 902 detects the sound source direction using a multi-channel acoustic signal. As an example of the sound source direction detection method, there is a method in which signals from a plurality of directional microphones are compared, and the direction in which the loudest microphone is facing is set as the sound source direction. As another sound source direction detection method, there is an acoustic beam forming technique. This is a known technique in which the direction in which the phase difference between the plurality of microphone signals is the smallest is regarded as the sound source direction, and the sound source direction is estimated by signal processing. Note that one sound source direction is obtained by the sound source direction detection unit 902.

  The video object separating unit 901 uses the direction information obtained by the sound source direction detecting unit 902 to separate the video object in the video frame from the video signal. Examples of video objects include person objects such as object A74, object B75, and object C76 in FIG. In addition, an object that constitutes a space, such as an automobile, a building, or a plant, can be regarded as a video object.

  As an example of the video object separation unit 901, there is an object detection method using pattern recognition. A video object image template is created in advance, and template matching is performed on the entire video frame using this template. If the correlation value with the template is equal to or greater than the threshold value, it is determined that a desired video object exists. The video object separation unit 901 selects and outputs one object existing in the direction obtained by the sound source direction detection unit 902 from the object candidates detected by pattern recognition or the like. Since the video is a projection of a limited area in space, the video object closest to the sound source direction is selected and output accurately.

  As described above, the object separation apparatus of FIG. 19 can separate the object signal in which the sound and the video are associated by separating the video object in the sound source direction.

JP 2005-45779 A

  However, the technique of Patent Literature 1 needs to acquire a video and audio signal by appropriately arranging a camera and a microphone for each object. As a result, there has been a problem that the cost of video production / storage / transmission increases.

  Moreover, other related techniques require only one sound source direction in sound source direction detection. As a result, there is a problem that when a plurality of sound sources are uttered at the same time, correspondence with the video object cannot be obtained.

  Therefore, the present invention has been invented in view of the above problems, and an object of the present invention is to associate at least one sound object and video object by using a correlation among a plurality of objects. To provide an audiovisual processing system, an audiovisual processing method, and a program.

  The present invention that solves the above-described problems includes a video object separation unit that separates a video object from an input video signal, a sound object separation unit that separates a sound object from an input sound signal, and a correlation between the video object and the sound object. In other words, the present invention is a video / audio processing system having a correlation associating unit that associates at least one of the video object and the audio object.

  The present invention that solves the above problems separates a video object from an input video signal, separates a sound object from an input audio signal, obtains a correlation between the video object and the audio object, Is a video and audio processing method for associating at least one or more.

  The present invention for solving the above-described problems is a process of separating a video object from an input video signal, a process of separating a sound object from an input audio signal, obtaining a correlation between the video object and the acoustic object, A program for causing an information processing apparatus to execute processing for associating at least one or more acoustic objects.

  The present invention can associate a video object and an audio object even if a plurality of objects are mixed in the signal.

FIG. 1 is a block diagram of the present embodiment. FIG. 2 is a diagram for explaining the present embodiment. FIG. 3 is a block diagram of the first embodiment. FIG. 4 is a block diagram of the second embodiment. FIG. 5 is a block diagram of the correlation association unit 32 according to the second embodiment. FIG. 6 is a diagram for explaining the operation of the category association unit 323. FIG. 7 is a block diagram of the third embodiment. FIG. 8 is a block diagram of the video object separation unit 13 according to the third embodiment. FIG. 9 is a block diagram of the correlation association unit 33 according to the third embodiment. FIG. 10 is a diagram for explaining the operation of the AV signal correlation association unit 333. FIG. 11 is a diagram for explaining the operation of the AV signal correlation association unit 333. FIG. 12 is a diagram for explaining the operation of the AV signal correlation association unit 333. FIG. 13 is a diagram for explaining the operation of the AV signal correlation association unit 333. FIG. 14 is a diagram for explaining the operation of the AV signal correlation association unit 333. FIG. 15 is a block diagram of the fourth embodiment. FIG. 16 is a block diagram of the correlation association unit 34 according to the fourth embodiment. FIG. 17 is a diagram for explaining the operation of the AV signal correlation association unit 343. FIG. 18 is a diagram for explaining the related art. FIG. 19 is a diagram for explaining the related art.

  An outline of an embodiment of the present invention will be described.

  Referring to FIG. 1, the present invention includes a video object separation unit 1, an acoustic object separation unit 2, and a correlation association unit 3.

  The video object separation unit 1 separates the video object in the video frame from the video signal. Examples of video objects include person objects such as object A74, object B75, and object C76 in FIG. In addition, an object that constitutes a space, such as an automobile, a building, or a plant, can be regarded as a video object. There may be a plurality of video objects to be separated.

  The acoustic object separation unit 2 is a unit that separates an input acoustic signal into a plurality of sound source signals. Here, the separated sound source signal is called an acoustic object.

  The correlation associating unit 3 receives a plurality of video objects and a plurality of audio objects, obtains a correlation between the video objects and the audio objects, and corresponds to the video object at which the audio object corresponds in the video frame. Is identified.

Embodiments of the present invention will be described below in detail with reference to the drawings.
<First Embodiment>
A first embodiment will be described.

  Referring to FIG. 3, the first embodiment includes a video object separation unit 11, an acoustic object separation unit 21, and a correlation association unit 31.

  The video object separation unit 11 separates the video object in the video frame from the video signal. Examples of video objects include person objects such as object A74, object B75, and object C76 in FIG. In addition, an object that constitutes a space, such as an automobile, a building, or a plant, can be regarded as a video object. An example of the video object separation unit 1 is an object detection method using pattern recognition. A video object image template is created in advance, and template matching is performed on the entire video frame using this template. If the correlation value with the template is equal to or greater than the threshold value, it is determined that a desired video object exists, and the corresponding partial area is separated as a video object signal. There may be a plurality of video objects to be separated.

  The acoustic object separation unit 21 is a unit that separates an input multi-channel acoustic signal into a plurality of sound source signals. Here, the separated sound source signal is called an acoustic object. As a method of generating object separation information in the acoustic object separation unit 2, a technique called blind source separation (Independent Component Analysis) or a method called independent component analysis can be used. Non-Patent Document 1 (2005, “Speech Enhancement”, Springer, (Speech Enhancement, Springer, 2005, pp. 271-369), page 271, describes the technology related to blind source separation and independent component analysis. 369 pages). By performing appropriate parameter settings, the acoustic object separation unit 2 can automatically separate the sound source signal from the input audio signal.

  The correlation associating unit 31 inputs a plurality of video objects and a plurality of audio objects, obtains a correlation between the video objects and the sound objects, and indicates at which position in the video frame the sound object corresponds to the video object. Identify. That is, it is determined from where in the video frame the acoustic object (sound source) is generated. The association processing is realized by extracting a feature vector from each object of video and sound, and obtaining a correlation having the highest correlation value by taking a correlation between them. Examples of feature vectors include time frequency features and category attribution.

In this way, the video object and the sound object are associated with each other.
<Second Embodiment>
A second embodiment will be described.

  Referring to FIG. 4, the second embodiment includes a video object separation unit 12, an acoustic object separation unit 22, and a correlation association unit 32.

  The video object separation unit 12 and the audio object separation unit 22 are the same as the video object separation unit 11 and the video object separation unit 11 according to the first embodiment, and thus detailed description thereof is omitted.

  As shown in FIG. 5, the correlation association unit 32 includes a video category determination unit 321, an acoustic category determination unit 322, and a category association unit 323.

  The video category determination unit 321 is a unit that specifies the category of the video object or calculates the degree of belonging to the category. Examples of object categories include male faces, female faces, children's faces, male whole bodies, female whole bodies, child whole bodies, cars, trains, PCs, displays, and the like. The determined category of the object is used for specifying an acoustic object existing in the video frame by subsequent processing.

  An example of the operation of the video category determination unit 321 will be described below. Several video categories are determined in advance, and typical image groups corresponding to the respective categories are prepared as templates. The attribute category is specified by performing pattern matching between the video object region pixel and the template and classifying the video object into the category having the highest similarity. There is also a method of calculating the similarity as the degree of belonging to each category. As a pattern matching method, a known technique such as a normalized correlation method can be used.

  The acoustic category determination unit 322 is a unit that specifies the category of the acoustic object or calculates the degree of belonging to the category. Examples of object categories include male voices, female voices, child voices, car sounds, train sounds, air conditioning sounds, keyboard sounds, mouse click sounds, ambient noise, and the like.

  An example of the operation of the acoustic category determination unit 322 will be described below. Several acoustic categories are determined in advance, and typical sound source data corresponding to each category is prepared. The belonging category is specified by matching the waveform of the audio object with the waveform of the sound source data and classifying the acoustic object into the category having the highest similarity. There is also a method of calculating each similarity as the degree of belonging to each category.

The category association unit 323 associates the category of the video object with the category of the audio object, and associates the video object with the audio object. An example of the operation of the category association unit 323 will be described with reference to FIG.
In the video frame 111, a male face 112, a female face 113, and a car 114 exist as video object categories. A video object group is shown in the object list 115.

  In the video frame 111, car sounds, female voices, male voices, and noises are classified as acoustic object categories. The acoustic object group is shown in the acoustic object list 116. It can be easily determined that a car corresponds to the sound of a car, a male voice corresponds to a male face, and a female voice corresponds to a female face.

  However, there is no corresponding video object only for the noise acoustic object.

  The object correspondence table 117 can be generated by the above processing. From the object correspondence table 117, it is possible to determine which video object each acoustic object corresponds to and the coordinate value in the video frame of the object.

  Note that the object correspondence table 117 is an example of uniquely identifying a category. As an example of a method for realizing the category association unit 323, a category attribute is used as a feature amount, and a correlation between feature amounts of objects is obtained and associated. There is also a method. The category attribution is a feature vector composed of the similarity to each category, and is represented by a value such as (male, female, car) = (1.0, 0.5, 0.2). By taking the closest combination of the feature vectors, the association between the video object and the sound object can be realized.

After the correlation and association unit 32 associates the video with the sound, the video object signal and the acoustic object signal are output.
<Third Embodiment>
A third embodiment will be described.

  Referring to FIG. 7, the third embodiment includes a video object separation unit 13, an acoustic object separation unit 23, and a correlation association unit 33.

  The acoustic object separation unit 23 operates in the same manner as the acoustic object separation unit 21, and separates and outputs the acoustic object.

  The video object separation unit 13 includes a person detection unit 131. The person detection unit 131 extracts a person area as a video object using pattern recognition. As a method for extracting a person area, there is a method for detecting a face area of a person. 586-591 (1991)).

  As shown in FIG. 8, the video object separation unit 13 detects a person area using a person detection unit 131 and separates the person area as a video object.

  As shown in FIG. 9, the correlation association unit 33 includes a motion detection unit 331, a voice segment detection unit 332, and an AV signal correlation unit 333.

  The motion detection unit 331 pays attention to a partial region such as a lip in the person region, obtains a difference between frames of the partial region over the past t time, and outputs a video motion pattern. The video motion pattern represents the temporal change of the pixel value in the area.

  The voice section detection unit 332 obtains, for each acoustic object, whether a voice section exists over the past t time, and outputs a voice section pattern.

  The AV signal correlation associating unit 333 associates the video motion pattern from the motion detecting unit 331 with the audio segment pattern from the audio segment detecting unit 332, and obtains a combination having a high correlation. Identify the corresponding acoustic object.

  A specific operation of the AV signal correlation association unit 333 will be described with reference to FIG.

  For example, in the video frame 121, the object detection unit 131 detects the object J122 and the object K123. The inter-frame difference in the lip portion which is a partial area in these person areas is shown in the lip area inter-frame difference 124. The motion pattern 125 is obtained by binarizing the lip region inter-frame difference 124 with an appropriate threshold value.

  Also, a result of voice segment detection performed on the acoustic object is a voice segment pattern 126. When the movement pattern 125 is compared with the voice segment pattern 126, it can be seen that there is a high correlation between the motion pattern of the object J and the first voice segment pattern 127, so it is determined that these objects are the same. To do. Further, since there is a high correlation between the motion pattern of the object K and the second voice segment pattern 128, it is determined that these objects are the same as well. In this way, it can be seen that the sound source signal of the object J is the first acoustic object and the sound source signal of the object K is the second acoustic object.

  Then, after the video and sound are associated by the AV signal correlation associating unit 333, the video object signal and the acoustic object signal are output.

  Next, a specific correlation value calculation method in the AV signal correlation association unit 333 will be described.

  FIG. 11 shows an example of a time-series motion pattern bx201 obtained by binarizing the integral value of the inter-frame difference of video into 0 and 1, and an audio section pattern by202 binarized into 0 and 1. Show. Here, the motion pattern bx201 corresponds to the motion pattern 125 described above, and the speech segment pattern by202 corresponds to the speech segment pattern 126 described above.

  Using a predetermined time interval T, the correlation value S from time a to time T can be calculated using equation (1).

そして、音と映像の組み合わせの中から、相関値Ｓが大きい組み合わせを選択することによって対応付けを行なう。

Then, association is performed by selecting a combination having a large correlation value S from the combination of sound and video.

  A correlation value calculation method in another AV signal correlation association unit 333 will be described.

  FIG. 12 shows a time-series motion pattern bx211 obtained by binarizing the integral value of the inter-frame difference of video into 0 and 1, and an audio section pattern by212 binarized into 0 and 1. . Here, the motion pattern bx211 corresponds to the motion pattern 125 described above, and the voice segment pattern by 212 corresponds to the voice segment pattern 126 described above.

  The time when the motion pattern bx is a star (time when it changes from 0 to 1) is t1x, and the end time (time when it changes from 1 to 0) is t2x. In addition, the time when the voice section pattern by rises (time when changing from 0 to 1) is t1y, and the end time (time when changing from 1 to 0) is t2y. Then, the time difference is calculated by Td of Equation 2. In the combination of sound and video, it is considered that the smaller the time difference Td is, the more the sound and video are associated.

尚、スタート時間だけを比較することで対応付けを行なうことも可能である。この場合には、式１０３に示す時間差Ｔｄ２を用いて時間差を算出する。

It is also possible to perform association by comparing only the start times. In this case, the time difference is calculated using the time difference Td2 shown in Expression 103.

  A correlation value calculation method in another AV signal correlation association unit 333 will be described.

  The AV signal correlation associating unit 333 receives the video motion pattern (difference between frames) from the motion detecting unit 331 and the audio section pattern from the audio section detecting unit 332. Then, the time-series motion pattern M221 of the integral value of the inter-frame difference of the video and the audio signal power J222 of the acoustic object are obtained. FIG. 13 shows an example of the time-series motion pattern M221 of the integral value of the inter-frame difference of the video and the audio signal power J222 of the acoustic object.

  At this time, the correlation value S2 between the video object and the sound object from time a to time T can be calculated using Equation 3 using a predetermined time interval T.

また、数４のＳ３のように、ＭとＪとの相関係数を相関値として算出することもできる。

Further, the correlation coefficient between M and J can be calculated as a correlation value, as in S3 of Equation 4.

そして、音と映像の組み合わせの中から、相関値Ｓが大きい組み合わせを選択することによって対応付けを行なう。

Then, association is performed by selecting a combination having a large correlation value S from the combination of sound and video.

  In the above-described correlation value calculation method in the AV signal correlation association unit 333, the motion pattern may be calculated from the motion vector of the video object.

In this case, as shown in FIG. 14, the motion vector of the video object from time t to time t + 1 is obtained. A motion vector calculation method includes a template matching method. This is a method in which a partial region image occupied by an object at time t is used as a template, and a position where a similar pattern exists is searched from the t + 1 video by template matching. Thereby, the motion vector of the video object from time t to t + 1 can be calculated. Next, the length of the motion vector 233 is obtained. In the present embodiment, the motion vector is generated by replacing the length of the motion vector with the integral value of the inter-frame difference described above, and the correlation value is calculated.
<Fourth embodiment>
A fourth embodiment will be described.

  Referring to FIG. 15, the fourth embodiment includes a video object separation unit 14, an acoustic object separation unit 24, and a correlation association unit 34.

  The video object separation unit 14 operates in the same manner as the video object separation unit 11 and separates and outputs the video object. The acoustic object separation unit 24 operates in the same manner as the acoustic object separation unit 21, and separates and outputs the acoustic object.

  As shown in FIG. 16, the correlation association unit 34 includes a video motion detection unit 341, a motion sound section detection unit 342, and an AV signal correlation correlation unit 343.

  The video motion detection unit 341 pays attention to the partial area where the video object exists, obtains the inter-frame difference of the partial area over the past t time, and outputs a motion pattern. The motion pattern represents a temporal change in pixel values in the region.

  The motion sound section detection unit 342 determines whether a motion sound exists for the past t hours for each acoustic object, and outputs a motion sound section pattern. As an example of the operation sound, there are an engine sound of a car and a footstep sound of a person walking.

  The AV signal correlation associating unit 343 compares the video motion pattern and the motion sound interval pattern to obtain a highly correlated combination, and identifies an acoustic object corresponding to the video object.

  The operation of the AV signal correlation association unit 343 will be described with reference to FIG.

  In the video frame 131, the object L132 and the object M133 are detected by the video object separation unit. The video motion detection unit 341 calculates the inter-frame difference of the partial area where these objects exist (the object area inter-frame difference 134 in FIG. 17), and 2 for the object area inter-frame difference 134 by an appropriate threshold value. The motion pattern 135 is calculated by digitizing.

  Further, the motion sound section detection unit 342 sets a motion sound section pattern 136 as a result of performing the motion sound section detection on the acoustic object.

  Since the AV signal correlation associating unit 343 compares the motion pattern 135 and the motion sound interval pattern 136, it can be seen that there is a high correlation between the motion pattern of the object L and the first audio interval pattern 137. , It is determined that these objects are the same. In addition, since there is a high correlation between the movement pattern of the object M and the second voice segment pattern 138, it is similarly determined that these objects are the same. In this way, it can be seen that the sound source signal of the object L is the first acoustic object and the sound source signal of the object M is the second acoustic object.

  The AV signal correlation associating unit 343 outputs the video object signal and the audio object signal after associating the video with the sound.

  The correlation value can be calculated using the correlation value calculation method described in the third embodiment.

  In the above-described embodiment, each unit is configured by hardware, but may be configured by an information processing apparatus such as a CPU that operates by a program. In this case, the program causes the CPU or the like to execute the above-described operation.

  Although the present invention has been described with reference to the preferred embodiments, the present invention is not necessarily limited to the above-described embodiments, and various modifications can be made within the scope of the technical idea.

DESCRIPTION OF SYMBOLS 1 Video object separation part 2 Acoustic object separation part 3 Correlation correlation part 131 Person detection part 321 Video category discrimination | determination part 323 Acoustic category discrimination | determination part 323 Category correlation part 331 Motion detection part 332 Audio | voice section detection part 333 AV signal correlation part 341 Video Motion detection unit 342 Motion sound interval detection unit 343 AV signal correlation association unit

Claims (6)

A video object separation unit that separates the video of each object from the input video signal as a video object;
An acoustic object separation unit that separates each sound source signal from the input acoustic signal as an acoustic object using blind signal source separation;
Correlation between each of the video objects and each of the acoustic objects, a correlation association unit that associates at least the video objects and the acoustic objects;
An output unit that outputs a signal of the video object and a signal of the acoustic object after the correlation associating unit associates the video object and the acoustic object ;
The video object is a partial area of the input video,
The video sound processing system , wherein the output unit outputs a signal of the video object corresponding to the partial area and a signal of the audio object corresponding to the video object . The video / audio processing system according to claim 1 , wherein the correlation associating unit generates an object correspondence table indicating to which video object each acoustic object separated from an input audio signal corresponds. Separate the video of each object from the input video signal as a video object,
Separate each sound source signal from the input acoustic signal using blind signal source separation as an acoustic object,
Obtaining a correlation between each video object and each acoustic object, and associating at least the video object with the acoustic object;
After associating the video object with the acoustic object, the signal of the video object and the signal of the acoustic object are output ,
The video object is a partial area of the input video,
A video and audio processing method for outputting the video object signal corresponding to the partial area and the audio object signal corresponding to the video object in outputting the video object signal and the audio object signal . The video / audio processing method according to claim 3 , wherein an object correspondence table indicating which video object corresponds to each audio object separated from the input audio signal is generated. A process of separating the video of each object from the input video signal as a video object;
Separating each sound source signal from the input acoustic signal as an acoustic object using blind signal source separation;
Correlation between each of the video objects and each of the acoustic objects, and an association process for associating at least the video objects and the acoustic objects;
After the video object and the acoustic object are associated with each other by the correlation association process, the information processing apparatus executes an output process for outputting the signal of the video object and the signal of the acoustic object ,
The video object is a partial area of the input video,
The output process is a program for outputting a signal of the video object corresponding to the partial area and a signal of the acoustic object corresponding to the video object . The program according to claim 5 , wherein the association processing generates an object correspondence table indicating which video object corresponds to each acoustic object separated from the input acoustic signal.

Images