JP2008118591A - Moving image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a moving image processor capable of removing effects of panning and hand shaking, and adding an index to a frame representative of a moving image. <P>SOLUTION: The moving image processor comprises a moving image acquisition means for acquiring the moving image; a detection means for detecting motion vectors for the moving image; an average/spread calculation means for calculating an average value and a spread value from the detected motion vectors; a complexity calculation means for calculating a motion complexity evaluation value from the calculated average value and the spread value; an index determination means for comparing the complexity evaluation value with a stored determination standard, and determining if the moving image corresponds to an index frame; and an index image display means for displaying an index image in accordance with a determination result. Therefore, the index can be added to a frame in the moving image with complex motion, and the image representing a characteristic of the moving image is shown as the index. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that detects a motion vector in a moving image and assigns an index according to a feature amount calculated from the motion vector. In particular, an index is assigned to a video camera, a DVD recorder, or the like. By displaying a list of images as thumbnails, the searchability of moving images is improved. Also, this is to realize a zapping function such as a moving image view.

  When a user searches for a target content from a plurality of moving images, the target moving image is selected while referring to an index image representing each moving image. However, unlike a still image, a moving image is composed of many still images that are arranged in time series, and therefore it is not easy to select an image that represents moving image content for display as an index image.

  In general, when an index image of a moving image is presented to a user, a method is often employed in which the first frame of the moving image is presented as a thumbnail image. However, the first frame of a moving image often starts with a fade and becomes a white or black image, or is often a less important scene because there are many states before some event starts.

  Thus, as a method for assigning an index to an image representing a moving image, Patent Document 1 discloses a technique for automatically adding an index by detecting a scene change based on a motion in the moving image. In this method, a portion where the correlation is hardly observed in the preceding and succeeding frames is determined as a scene change, and an index is assigned to the frame immediately after the scene change. This function works effectively in the case of a moving image that frequently undergoes scene changes such as news, but the above method cannot be applied if a scene change does not occur in one moving image, such as a moving image shot with a video camera.

In Patent Document 2, a moving image is divided into several spatial blocks, each motion vector is obtained, and an index is added to a frame whose size and number of motion vectors exceed a preset threshold. The technique of doing is disclosed. That is, a frame having a large motion in the moving image is detected as an index. In this case, an index is assigned to a frame in which a moving image is moving greatly regardless of a scene change.
JP 9-312827 A JP 2003-187216 A

  However, in the above method, when a large movement occurs on the entire screen due to panning, camera shake, zooming, or the like, the condition considered as an index frame is applied, and an index is assigned to the frame. That is, since only the magnitude and number of motion vectors are used as index detection determination materials, there is a problem that a frame applicable to the above algorithm is detected as an index regardless of the direction of the motion vector.

  Therefore, in the present invention, a motion vector is detected from a moving image, a motion complexity is calculated from the size and direction of the motion vector, and a frame having a large complexity is used as an index image, so that the motion is large but complicated. An object of the present invention is to provide an index image in which the effects of panning, camera shake, and zooming that are small in degree are eliminated.

  In order to achieve the above object, a moving image processing apparatus of the present invention includes a moving image acquisition unit that acquires a moving image, a movement detection unit that detects a motion vector for the moving image, and an average from the detected motion vectors. Average / variance calculation means for calculating values and variance values, motion complexity calculation means for calculating motion complexity evaluation values from the calculated average values and variance values, and determination to hold the complexity evaluation values Index determination means for determining whether or not a corresponding moving image corresponds to an index frame, a determination result storage means for storing the determination result, and an index for displaying an index image based on the determination result Image display means.

  With such a configuration, both the average value and variance of the motion vector become large, that is, it is possible to add an index to a frame in which the motion in the moving image is complicated, eliminating the effects of panning, camera shake, and zoom. An image that clearly shows the characteristics of the moving image can be presented as an index.

  With the above configuration, only frames with complex motion can be used as indexes, eliminating the problem that frames such as panning, camera shake, and zoom are large, but frames such as movement in a single direction are erroneously detected as indexes. be able to. In addition, an index can be assigned to a frame that characterizes moving image content, which is a frame with complicated movement, and the moving image search performance of the user is improved.

  Hereinafter, embodiments of the present patent will be described with reference to the drawings.

  FIG. 1 shows the configuration of the moving image processing apparatus of this patent. 108 is a moving image acquisition unit that acquires video from a digital video camera or the like, 101 is a motion detection unit that detects a motion vector for an image captured by the moving image acquisition unit 108, and 102 is detected by the motion detection unit 101. Average / variance calculating means for calculating the average value and variance of the magnitude from the motion vector, 103 is a motion complexity calculating means for calculating the complexity of the motion from the average value and variance calculated by the average / variance calculating means 102, Reference numeral 104 denotes index determination means for determining whether or not an index frame can be obtained based on the motion complexity obtained from the motion complexity calculation means 103, and 105 denotes a reference value serving as a determination criterion in the index determination means 104. The determination criterion storage means 106 is determined as an index by the index determination means 104. Determination result storage means for storing a frame of information and the complexity of that time, 107 based on the index frame information stored in the determination result storage unit 106, an index image display means for displaying an index image.

  The operation of the noise reduction apparatus of the present invention configured as described above will be described with reference to the flowchart of FIG. This configuration shows a procedure for creating one index image in one moving image. A moving image obtained from a digital video camera is a processing target. The image capturing means 108 is an image pickup apparatus including an analog circuit and a CCD in a digital video camera. First, initialization processing (S01) is performed. In the initialization process, 0 is stored in the complexity evaluation value storage register of the determination result storage means 106, and the first frame of the moving image captured in the index frame storage register is stored. Next, an image for one frame or one field is captured by the image capturing means 108 (S02). Next, a motion vector is detected by the motion detection means for the captured image (S03). The motion vector is obtained by calculating the shift amount for each block between successive frames as a vector. In other words, the spatial relative position of the reference area when the difference between the image area in the image to be processed and the reference value is the smallest with the image area (block) in the image as the reference value It is. Hereinafter, a block matching method which is a typical method for obtaining a motion vector will be described.

  In the block matching method, an image is divided into small areas (blocks) having a certain size, and a motion vector of each block is obtained. A search is made as to where in the previous frame each block corresponds, and the spatial relative position of the position of the corresponding block is used as a motion vector. When obtaining the motion vector of an N × N block of the t-th frame, if the position of the pixel at the upper left corner of the block is (i0, j0), the image of this block and the corresponding position of the immediately preceding frame ( The difference from the image of the block shifted by p, q) is evaluated. For the evaluation of matching, the sum of absolute values of differences in pixel values (SAD: Sum of Absolute Difference) as shown in the following equation is used.

The SAD (p, q) is calculated by changing the block position (p, q) of the correspondence candidate, and the minimum (p, q) is set as the motion vector (pm, qm). When the search range of the motion vector is Rs, the following expression is used.

Here, the above expression means (p, q) that minimizes SAD (p, q).

  In this embodiment, a block matching method is used as a method for obtaining a motion vector. The block matching method has an advantage that even a portion where the luminance changes abruptly has little vector error and is resistant to noise.

  In addition to the block matching method, a gradient method may be used as a method for obtaining the motion vector. The gradient method is a technique for estimating a flow vector from a condition based on a constraint equation of spatiotemporal differentiation of an image, and has an advantage that processing can be performed at high speed because it is not necessary to perform corresponding point search.

  Next, with respect to the motion vector obtained by the motion detection means 101, the average value of the magnitudes and the variance of the motion vector are calculated (S04). The formula for obtaining the average value is shown below.

Furthermore, the formula for obtaining the variance is shown below.

A motion complexity is obtained from the average and variance of the obtained motion vectors (S05). The complexity of motion is a value that represents the complexity of motion in which the value of a frame with a large motion in the moving image is large, and conversely, the value of a frame in which there is almost no motion in the moving image is small. The complexity is calculated from the average E and variance V obtained by the average / variance calculation means 102 by the following formula.
E × V
A value obtained by multiplying the average value of the motion vector and the variance is defined as complexity, and an index frame is detected using this as an evaluation value. In E × V, the movement of the entire screen is large, and the value increases when the direction of each vector is in a different direction. For example, as shown in FIG. 4, when the same direction of motion occurs on the entire screen, such as panning or camera shake, the average value of the motion vector is large, but the vector is in the same direction. The variance is a value close to 0. At this time, the aforementioned complexity does not become a large value. On the other hand, when considering a moving image obtained by photographing a subject crossing the background, as shown in FIG. 5, the motion vector is not so large, but its variance takes a large value, and the complexity also becomes high. During zooming, as shown in FIG. 6, a radial motion vector having a direction in the center direction on the entire screen or a radial motion vector having a direction outward from the center is detected. At this time, the average value is 0 because the motion vectors at the opposite positions cancel each other. Therefore, the value of the above-described evaluation formula becomes small, and as a result, the influence of zoom can be eliminated. Similarly, in rotation, as shown in FIG. 7, since the average value of motion vectors rotating around the center of the screen becomes 0, the influence can be eliminated.

  After calculating the motion complexity, which is an evaluation value, it is necessary to determine whether or not to add an index to the frame. Considering the case where one index is assigned to one moving image, the frame in which the above-described evaluation formula is the maximum among all the frames in the moving image is used as the index. The evaluation value obtained in each frame is compared with the evaluation value stored in the complexity evaluation value storage register of the determination result storage means 106 (S06). If the evaluation value is larger than the stored value, a new evaluation of that frame is performed. The value is stored in the complexity evaluation value storage register. The frame number at that time is stored in the index frame storage register (S07).

  The above processing is executed in the same manner up to the final frame, and after the final frame processing is finished, an index is assigned to the frame whose evaluation value is stored in the index frame storage register of the determination result storage means 106. In the case of a video camera, a list of index images is displayed and used as thumbnail images.

  With such a configuration, both the average value and variance of the motion vector become large, that is, it is possible to add an index to a frame in which the motion in the moving image is complicated, eliminating the effects of panning, camera shake, and zoom. The frame image characterizing the moving image content can be presented as an index.

Note that the complexity evaluation formula E × V may be a sum obtained by weighting the average value and the variance instead of multiplication. The formula is shown below.
aE + bV
Here, the constants a and b are weighting factors for the average value and the variance, respectively.

  Furthermore, although the case where only one index is assigned in one moving image was considered in the above embodiment, a configuration in which a plurality of indexes are added in one moving image can be easily realized. The flowchart in that case is shown in FIG. The processes of S11 to S14, S17, and S18 are the same as the processes of S02 to S05, S08, and S09 in the flowchart of FIG. The difference from the flowchart of FIG. 2 is that the criteria for determining whether or not to use an index is recorded in the frame storage register as an index for all frames in which the motion complexity exceeds a certain threshold (S15) (S16). Yes, it is possible to add a plurality of indexes by taking out a frame having a large evaluation value from the top as many as the number set in advance by the user as an index.

  Further, the index image display unit 107 has been described with respect to an embodiment in which one or a plurality of indexes are assigned to a moving image and the corresponding index image is displayed. It is also possible to display a list of one index image corresponding to the moving image for each moving image, and display thumbnail images for selecting moving images to be reproduced.

  Also, by displaying the moving images before and after the index on the index image display means 107, only a part representing the moving image content can be reproduced, and it can be used for confirming the contents of the moving image in a short time. It is.

  According to the moving image processing apparatus of the present invention, when adding an index to a moving image, it is possible to eliminate the influence of panning and camera shake, efficiently detect a frame moving in a complicated manner, and add an index. In addition, by presenting the index to the user, it is possible to improve the searchability of moving image files, which is useful in moving image processing apparatuses such as video cameras and DVD recorders.

The block diagram which shows the structure of this invention The flowchart which shows operation | movement of the moving image processing apparatus in one embodiment of this invention. The flowchart which shows operation | movement of the moving image processing apparatus in other embodiment of this invention. Diagram showing motion vector during panning and camera shake A diagram representing the motion vector of a frame with complex motion Diagram showing motion vector during zooming Diagram showing motion vector during rotation

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Motion detection means 102 Average / variance calculation means 103 Motion complexity calculation means 104 Index determination means 105 Judgment reference storage means 106 Determination result storage means 107 Index image display means 108 Moving image acquisition means

Claims (6)

Moving image acquisition means for acquiring a moving image;
Motion detection means for detecting a motion vector for a moving image;
An average / variance calculating means for calculating an average value and a variance value from the detected motion vector;
A motion complexity calculating means for calculating a motion complexity evaluation value from the calculated average value and variance value;
An index determination unit that compares the complexity evaluation value with a determination criterion to be retained and determines whether or not the corresponding moving image corresponds to an index frame;
A determination result storage means for storing the determination result;
A moving image processing apparatus comprising index image display means for displaying an index image based on the determination result. The moving image processing apparatus according to claim 1, wherein the index determining unit assigns an index to a frame having the largest complexity evaluation value in the moving image. The moving image processing apparatus according to claim 1, wherein the index determination unit assigns an index to all frames in which the complexity evaluation value exceeds a threshold value. The moving image processing according to claim 1, wherein the index determining unit selects the number of the complexity evaluation values from the top in the number set by the user, and assigns an index to each corresponding frame. apparatus. The moving image processing apparatus according to claim 1, wherein the index image display unit displays a list of corresponding index images for a plurality of moving images, and displays thumbnail images for selecting moving images. 6. The moving image processing apparatus according to claim 1, wherein the index image display means reproduces the frame before and after the indexed frame in order to confirm the contents of the moving image.