JPH09297850A - Device and method for analyzing dynamic picture - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform processing fast without using any special image processing hardware by tracking a body over frame images applicable to dynamic picture editing from one to another, and specifying the application to dynamic picture editing. SOLUTION: A key frame detection part 13 takes out a frame image containing a scenes change or a largely moving body on a screen as a key frame image. An area detection part 14 generates an area divided image from the key frame image on the basis of color and edge information. A moving vector detection part 15 acquires two key frame images which succeed with time, detects moving vectors in block units by using a block matching method, and also detects a moving vector in an area on the basis of the area divided image. An area correspondence part 16 makes an area which is considered to indicate the same body to correspond between frames according to the moving vector in the area and the area divided image.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving image analyzing apparatus and a moving image analyzing method, and more particularly to a moving image analyzing apparatus and a moving image analyzing method used for editing moving images.

[0002]

2. Description of the Related Art A conventional moving image analyzing apparatus and a conventional moving image analyzing method are mainly used to automatically recognize a person or a vehicle in a surveillance system. As an example of these, an analysis method for detecting a pedestrian from a moving image is disclosed in a document (Salt
Others, "Detection of pedestrians from moving images", Information Processing Research Group Report, CV75-5, 1991. ). In the moving image analysis device and the moving image analysis method described in this document, an object area is extracted by generating a background image for a frame image, and a motion vector is detected by using a block matching method, and the same motion is detected. Regions are divided based on the assumption that the regions that indicate are likely to be the same object.

Further, in recent years, there has been a great movement to use moving image analysis for editing moving images. As an example of this, a moving image analysis device and a moving image analysis method based on the link method between same subjects are known (Ueda, Nagasaka, Miyatake, “Automatic link between objects based on video analysis result and its application”, The Institute of Electrical Communication National Convention, 1993.). When the user specifies an object, the moving image analysis device and the moving image analysis method described in this paper automatically detect objects having the same color combination from the remaining frames and generate hyperlinks between them. To do.

[0004]

Among the conventional moving image analyzing apparatus and moving image analyzing method described above, in the moving image analyzing apparatus and moving image analyzing method applied to the former field of automatic recognition, the object area It is intended for accurate detection and fine tracking. Therefore, the detection of the object region and the tracking by associating the frame images of the regions estimated to represent the same object across the frame images are performed by finely processing the frame images of the moving image without spacing. For this reason, the processing takes time, and special image processing hardware is often required for speeding up.

On the other hand, in the latter moving image analyzing apparatus and moving image analyzing method applied to the moving image editing, although the processing is simple, the relation between the frame images is not considered when detecting the same object, Since the frame images are not continuously examined temporally and the objects are not associated with each other, an error is likely to occur and there is a problem in accuracy.

[0006] The present invention has been made in view of the above points,
Video that can be applied to video editing, can track objects across frame images, and specializes in video editing to perform high-speed processing without using special image processing hardware. An object is to provide an image analysis device and a moving image analysis method.

[0007]

In order to achieve the above-mentioned object, a moving image analysis apparatus of the present invention is a key frame detecting section for extracting a key frame image which is a key frame accurately representing moving image contents from moving image data. An area detection unit that generates an area-divided image from the key frame image, and a motion vector detection unit that detects a motion vector of the area from the key frame image acquired from the key frame detection unit and the area divided image acquired from the area detection unit. And the motion of the object is derived from the region-divided image acquired from the region detection unit and the motion vector of the region acquired from the motion vector detection unit, and the correspondence between the frames of the regions estimated to point to the same object And a storage device that stores at least a key frame image, a region division image, and a motion vector. It is.

Further, according to the moving image analysis method of the present invention, a key frame image that is a key frame that accurately represents the moving image content is extracted from the moving image data, an area division image is generated from this key frame image, and then the key frame image is generated. The motion vector of the area is detected from the image and the area-divided image, and the motion of the object is further derived from the motion vector of this area and the area-divided image, and between the frames of the area estimated to point to the same object. Correspond.

In the moving image analysis apparatus and the moving image analysis method of the present invention, since the area detection and the motion vector detection for object tracking are performed by using only the key frame image, the number of processing symmetric frames can be reduced. it can. Further, the frame image to be processed can be specified as a key frame image that accurately represents the image content.

Further, the moving image analysis apparatus of the present invention calculates a contrast value for each of the divided blocks, and a block dividing unit for dividing one of two time-continuous key frame images into a plurality of blocks. , The calculated contrast value is compared with the threshold value and the motion vector of the block whose contrast value is lower than the threshold value is indefinite, and the other key for all the blocks whose contrast value is higher than the threshold value. A motion vector detection unit for a block configured to inspect a color difference from a block in the vicinity of a frame image, and a block matching unit that uses a point having the smallest color difference as a corresponding point and a change in the position as a motion vector Since there are few features, it is possible to exclude the motion vector of the block where the corresponding points are prone to errors due to the small number of features. Kill.

Further, in the moving image analysis apparatus of the present invention, a block division unit for dividing one of two temporally consecutive key frame images into a plurality of blocks, and the other key frame for all the divided blocks. A block matching unit that inspects a color difference with a block in the vicinity of an image and uses a point where the color difference is the minimum as a corresponding point as a motion vector, and a block motion calculated by the block matching unit. Since the color error of the vector is compared with the threshold value, the motion vector detecting section for the block is configured to include the color error comparing section that makes the motion vector of the block with the color error larger than the threshold value indeterminate. Eliminates the influence of illegal vectors that occur when the corresponding points disappear because the frames are spaced apart because only keyframes are processed. It can be.

Further, in the moving image analysis apparatus of the present invention, the bias near the minimum value of the color error distribution of the motion vector of the block calculated by the block matching unit is calculated,
Since the configuration has the motion vector detection unit of the block including the color error distribution inspection unit that makes the motion vector of the block of the color error distribution in which the bias is larger than the threshold value indefinite, there are many blocks with similar colors, It is possible to remove an incorrect motion vector that occurs when one of the two happens to match.

Further, in the moving image analysis apparatus of the present invention, the area division image is obtained from the area detection unit, the motion vector is obtained from the motion vector detection unit of the block, and the number of motion vectors of the blocks existing in the area division image is calculated. When the number is less than the threshold, the motion vector number inspecting unit that makes the motion vector of the region indefinite, and when the motion vector number inspecting the region divided image by the motion vector number inspecting unit is greater than the threshold value, A histogram creation unit that creates a histogram of motion vectors existing in the area-division image based on the area-division image acquired from the area detection unit and the motion vector of the block acquired from the block motion vector detection unit. And a peak detection unit that detects the motion vector with the highest appearance frequency as the motion vector of that region. Due to a configuration in which a motion vector detecting portion of the free region, it is possible to remove the motion vector of the less accurate area caused when the number of blocks present in the area was small.

Furthermore, in the moving image analysis apparatus of the present invention,
The histogram creation unit and the peak detection unit, and a frequency inspection unit that compares the ratio of the frequency of the motion vector detected by the peak detection unit to the whole with a threshold value and makes the motion vector of the ratio equal to or higher than the threshold value indefinite Since the configuration is provided with the motion vector detecting unit for the region including, the motion vector of the region with low accuracy that occurs when the values of the motion vectors of the blocks in the region are dispersed can be removed.

[0015]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Next, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a block diagram of a first embodiment of the present invention and corresponds to the invention described in claim 1. The moving image analysis apparatus according to this embodiment includes a data processing apparatus 1 that operates under program control and a storage apparatus 2 that stores information.

The data processing apparatus 1 includes a moving image reproducing section 11, a frame acquiring section 12, a key frame detecting section 13, an area detecting section 14, a motion vector detecting section 15, and an area associating section 16.
It is composed of

The moving image reproducing section 11 reproduces the digitized moving image data. The frame acquisition unit 12 extracts frames one by one as still images from the moving image data being reproduced. The key frame detection unit 13 measures the color histogram change and the inter-frame difference of the frame image acquired by the frame acquisition unit 12 to accurately determine the content of the moving image of the frame image in which the scene changes or the object in the image moves greatly. It is extracted as the key frame image shown in. Large screen changes can be detected by color histogram changes,
By using the difference between the frames together, the color change can be detected at least when the object in the moving image largely moves, and the accuracy can be further improved.

The area detector 14 is the key frame detector 1
The color and edge information is detected from the key frame image detected in 3, and the area division image is generated from the key frame image based on the detected color and edge information. The motion vector detection unit 15 acquires two temporally consecutive key frame images detected by the key frame detection unit 13 and detects a block-based motion vector using the block matching method. further,
The motion vector of the area is detected based on the area division image detected by the area detection unit 14. Based on the motion vector of the area detected by the motion vector detecting section 15 and the area division image detected by the area detecting section 14, the area associating section 16 detects the same object between frames of the area estimated to point to the same object. Correspond.

The storage device 2 is composed of a key frame storage unit 21, an area storage unit 22, a motion vector storage unit 23, and a related area storage unit 24. Key frame storage unit 2
Reference numeral 1 manages the key frame image detected by the key frame detection unit 13. The area storage unit 22 includes the area detection unit 1
The area division image detected by 4 is managed. The motion vector storage unit 23 manages the motion vector of the area detected by the motion vector detection unit 15. Related area storage 24
Stores information about the area estimated to point to the same object by the area association unit 16.

FIG. 2 is a more detailed block diagram of the motion vector detecting unit 15 according to the first embodiment of the present invention, which corresponds to the invention described in claim 7. Motion vector detection unit 1
5 is composed of a block motion vector detection unit 151 and a region motion vector detection unit 152, and corresponds to the invention described in claims 2, 3 and 7. The block motion vector detection unit 151 further includes a block division unit 151.
1 and the block matching unit 1512, two temporally continuous key frame images are acquired, and the motion vector is detected using the block matching method.

The area motion vector detecting section 152 is composed of a histogram creating section 1521 and a peak detecting section 1522. The motion vector detecting unit 152 of the area detects the motion of the block existing in the area by the histogram creating unit 1521 based on the area divided image detected by the area detecting unit 14 and the motion vector of the block detected by the motion vector detecting unit 151 of the block. A vector histogram is created, and the peak detection unit 1522 detects the motion vector having the maximum frequency in the histogram to detect the motion vector of the area.

FIG. 3 shows a more detailed block diagram of the area associating unit 16. As shown in FIG. 3, the area associating unit 1
6 is a region prediction unit 161 and a corresponding region matching unit 162.
It is composed of The area prediction unit 161 includes the area detection unit 1
For each area of the area-divided image detected by 4,
The area is predicted by calculating the area whose position is moved based on the motion vector.

The corresponding area matching unit 162 checks the degree of overlap of the areas predicted by the area prediction unit 161 with all the areas of the next frame, and determines the area with the largest overlapping area as the same object. Treated as the corresponding area. The association storage unit manages a group of sets of frame numbers and area IDs for areas estimated to represent the same object.

Next, the operation of the first embodiment of the present invention will be described with reference to FIGS. 4 is shown in FIG.
6 is a flowchart illustrating the operation of the first exemplary embodiment of the present invention shown in FIG.

First, the moving image reproducing section 11 reproduces the digitized moving image data (step ST1). The frame acquisition unit 12 takes out frames one by one as still images from the moving image data being reproduced (step ST
2). The key frame detection unit 13 uses the frame acquisition unit 12
The color histogram change and the inter-frame difference of the frame image (still image) acquired by are measured (step ST
3) Then, it is determined whether the frame is a key frame or not depending on whether each is a preset threshold value or more (step ST4). Here, if the inter-frame difference value is less than the threshold value, it is determined that the acquired frame is not a key frame, and if it is greater than or equal to the threshold value, the acquired frame is detected as a key frame. To do.

The area detector 14 is the key frame detector 1
The color and edge information is detected from the frame image of the key frame detected in 3 (hereinafter referred to as the key frame image),
Based on this, an area division image corresponding to the key frame image is generated (step ST5). Motion vector detection unit 15
Detects a motion vector from two temporally consecutive key frame images detected by the key frame detection unit 13, and detects the motion vector of the area based on the area division image generated by the area detection unit 14. Is derived (step ST
6).

Then, it is checked whether all the frames existing in the moving image file have been processed (step ST
7) If all the frames have not been processed, the process returns to step ST2, the frame acquisition unit 12 acquires the next frame, and then the processes of steps ST3 to ST6 are executed again. When it is determined in step ST7 that all the frames are processed, the area matching unit 16
Is based on the motion vector of the region detected by the motion vector detection unit 15 and the region-divided image generated by the region detection unit 14 and associates the regions estimated to point to the same object between frames. Perform (step ST8).

Next, details of the motion vector detection processing by the motion vector detection unit 15 in step ST6 will be described with reference to the flowcharts of FIGS. 5 and 6. First, the processing of the block motion vector detecting unit 151 shown in FIG. 2 will be described with reference to the flowchart of FIG. 5. The block dividing unit 1511 detects that two time-sequential keys detected by the key frame detecting unit 13 are consecutive. Frame images are acquired (step ST11), and the first key frame image is divided into blocks (step ST1).
2).

Then, the block matching unit 1512 in the motion vector detecting unit 151 of the block shown in FIG. 2 determines that the block having the first key frame image is all blocks in the vicinity of the second key frame image. And inspect the difference in the colors of (steps ST13, ST14),
A position vector to a block in which the color difference is the smallest is derived as a motion vector (step ST1).
5). It is determined whether or not this motion vector has been derived for all blocks of the first key frame image (step ST16), and steps are performed until motion vectors have been derived for all blocks of the first key frame image. The processing of ST13 to ST16 is repeated.

Next, regarding the processing of the motion vector detection unit 152 of the area shown in FIG.
Description will be made with reference to the flowchart of FIG. The area motion vector detection unit 152 is a block motion vector detection unit 1.
The motion vector of the block is acquired from 51 (step S
(T21), and further acquires a region division image from the region detection unit 14 (step ST22).

The histogram creating unit 1521 creates a histogram of motion vectors of blocks existing in a certain area based on these (step ST23). Then, the peak detection unit 1522 detects the motion vector having the maximum frequency from the created histogram as the motion vector of the region and accumulates it (step ST2).
4). Then, the processes of steps ST23 and ST24 described above are executed for all the regions (step ST2).
5).

Next, details of the processing of the area associating unit 16 in the embodiment of FIG. 1 will be described with reference to the flowchart of FIG. Area mapping unit 16 shown in FIGS. 1 and 3
Acquires the area information read from the area storage unit 22 (step ST31), and further acquires the motion vector read from the motion vector storage unit 23 (step ST3).
2) The position of each area and its motion vector are acquired based on the acquired information (step ST3).
3).

Then, it is judged whether or not the obtained motion vector of one area is indefinite (step ST34),
When the motion vector is indefinite, the position and motion vector of the next area are acquired and it is determined whether or not the motion vector of the area is indefinite (steps ST33, ST34).
When the motion vector is not indefinite, the region predicting unit 161 predicts a region in which the position of the region is moved based on the motion vector by calculation (step ST35), and then the corresponding region matching unit 162 predicts the predicted region and the next frame. The degree of area overlap with all areas is checked (step ST36), and the area with the largest area overlap is registered in the related area storage unit 24 as a corresponding area representing the same object (step ST37).

The above steps ST31 to ST37 are repeated for all areas (step ST3
8), and the processing of steps ST31 to ST38 is repeated for all frames (step ST3).
9).

As described above, in this embodiment, since the area detection and the motion vector detection for object tracking are performed only for the key frame, the number of frames to be processed can be reduced and the speed can be increased. You can Moreover, although the number of frames to be processed is reduced, the frames that accurately express the contents are processed, so that it is possible to suppress the decrease in accuracy within a reasonable range.

Next, a second embodiment of the present invention will be described. FIG. 8 is a detailed block diagram of the motion vector detecting unit according to the second embodiment of the present invention, and corresponds to the invention described in claim 4. 8, those parts that are the same as those corresponding parts in FIG. 2 are designated by the same reference numerals. The second embodiment of the present invention is characterized by the motion vector detection unit 153 of the block in the motion vector detection unit 15 and the motion vector detection unit 15 of the area.
2 has exactly the same configuration as that of the first embodiment.

The block motion vector detector 153
Block division unit 1511 and block matching unit 151
2 and a contrast inspection unit 1513. The contrast inspection unit 1513 calculates the contrast value of all the blocks of the first key frame image, compares the calculated value with a predetermined threshold value, and obtains a calculated value lower than the threshold value. The frame image is treated as having low reliability and the motion vector is indefinite.

Next, the operation of the second embodiment of the present invention will be described with reference to FIGS. FIG. 9 shows a flowchart showing the operation of the second embodiment of the present invention. In FIG. 9, the block division unit 1511 of FIG.
Two temporally consecutive key frame images detected by the key frame detection unit 13 are acquired (step ST4
1) Among them, the first key frame image is divided into blocks (step ST42).

Subsequently, the contrast inspection unit 1513 shown in FIG. 8 calculates the contrast value of each block of the first key frame image (step ST43), and sets the contrast value as a predetermined threshold value. The comparison is made (step ST44). As a result of this comparison, a block having a contrast value equal to or lower than the threshold value is regarded as having low reliability, and the motion vector of the block is treated as indefinite (step ST45).

On the other hand, as for the block of the first key frame image having the contrast value equal to or higher than the threshold value by the above comparison, the block matching unit 1512 colors the blocks of all the neighboring blocks of the second key frame image. Are compared and inspected (steps ST46 and ST47), and a position vector to a block in which the color difference is the smallest is derived as a motion vector (step ST48), and this motion vector is used for the motion of the block with high reliability. It is stored in the motion vector storage unit 23 as a vector (step ST4).
9).

Then, it is judged whether or not this motion vector has been derived for all blocks of the first key frame image (step ST50), and motion vectors of all blocks of the first key frame image have been calculated. The processes of steps ST43 to ST50 are repeated until the derivation.

According to the second embodiment, a block which is likely to have an error at a corresponding point due to a small number of features and which is expected to have a low motion vector accuracy is excluded. Therefore, a decrease in accuracy is prevented. effective. Further, since the block matching is performed only on the blocks having the contrast value equal to or more than the threshold value, the number of times of the block matching processing which requires a calculation cost can be reduced, and thus the processing speed can be increased.

Next, a third embodiment of the present invention will be described. FIG. 10 is a detailed block diagram of the motion vector detecting unit according to the third embodiment of the present invention, and corresponds to the invention described in claim 5. 10, those parts that are the same as those corresponding parts in FIG. 2 are designated by the same reference numerals. The third embodiment of the present invention is characterized by the block motion vector detection unit 154 in the motion vector detection unit 15, and the region motion vector detection unit 152 is exactly the same as in the first and second embodiments. It is a composition.

The block motion vector detector 154
Block division unit 1511 and block matching unit 151
2 and a color error comparison unit 1514. Color error comparison unit 15
Reference numeral 14 compares the color error when the motion vector is calculated by the block matching method in the block division unit 1511 and the block matching unit 1512 with a predetermined value, and the motion vector of the block whose error is larger than the threshold value is Indefinite

Next, the operation of the third embodiment of the present invention will be described with reference to FIGS. FIG.
Shows a flowchart for explaining the operation of the third embodiment of the present invention. 11, the same processing steps as those in FIG. 5 are designated by the same reference numerals, and the description thereof will be omitted.

In FIG. 11, when the position vector to the block having the smallest color difference is derived as the motion vector in step ST15, the color error comparison unit 1514 shown in FIG. The minimum value of the color error when the motion vector is calculated by the block matching unit 1512 by the block matching method is compared with a predetermined value, and it is determined whether or not the minimum value of the color error is smaller than the threshold value ( Step ST55).

When the minimum value of the color error is equal to or more than the threshold value, it is determined that the reliability of the motion vector is low, and the motion vector of the block is indefinite (step ST56). On the other hand, when the minimum value of the color error is smaller than the threshold value, this motion vector is stored in the motion vector storage unit 23 as the motion vector of the block with high reliability (step S).
T57).

Then, it is judged whether or not this motion vector has been derived for all blocks of the first key frame image (step ST58). Steps ST13 to ST15, ST until derivation
The processes of 55 to ST58 are repeated.

According to the third embodiment, when the interval between two key frames is widened and the corresponding points of the second key frame image disappear, the color error of the corresponding points is reduced. Since the minimum value is greater than or equal to the threshold value, the motion vector is not fixed, that is, the motion vector is not judged, and thus the influence of the illegal vector generated for processing only the key frame can be removed.

Next, a fourth embodiment of the present invention will be described. 12 is a block diagram showing a configuration of a motion vector detecting unit according to a fourth embodiment of the present invention.
Corresponds to the invention described in (1). 12, those parts that are the same as those corresponding parts in FIG. 2 are designated by the same reference numerals. The fourth embodiment of the present invention is characterized by the block motion vector detection unit 155 in the motion vector detection unit 15, and the region motion vector detection unit 152 is exactly the same as in the first to third embodiments. It is a composition.

The block motion vector detection unit 155
Block division unit 1511 and block matching unit 151
2 and a color error distribution inspection unit 1515. The color error distribution inspection unit 1515 makes the motion vector of the block, in which the color error distribution when the motion vector is calculated by the block matching method in the block division unit 1511 and the block matching unit 1512 is biased near the minimum value, to be indefinite.

Next, the operation of the fourth embodiment of the present invention will be described with reference to FIGS. FIG.
Shows a flowchart for explaining the operation of the fourth embodiment of the present invention. In FIG. 13, the same processing steps as those in FIG. 5 are designated by the same reference numerals, and the description thereof will be omitted.

In FIG. 13, when the position vector to the block having the smallest color difference is derived as the motion vector in step ST15, the color error distribution inspection unit 1515 shown in FIG. And the block matching unit 1512 checks the bias of the color error distribution near the minimum value when the motion vector is calculated by the block matching method (step ST61), and compares whether the calculated bias is smaller than the threshold value (step ST61). ST62).

As a result of the comparison, when the calculated bias is equal to or more than the threshold value, it is determined that the reliability of the motion vector of the block is low, and the motion vector of the block is undefined (step ST63). On the other hand, when the calculated bias is smaller than the threshold value, it is determined that the reliability of the motion vector is high, and the motion vector is accumulated as the motion vector of the block (step ST64).

Subsequently, it is judged whether or not this motion vector has been derived for all blocks of the first key frame image (step ST65). Steps ST13 to ST15, ST until derivation
The processing from 61 to ST65 is repeated.

According to the fourth embodiment, it is possible to eliminate an illegal motion vector which occurs when there are a large number of blocks having similar colors and it happens that one of them coincides.

Next explained is the fifth embodiment of the invention. 14 is a block diagram showing a configuration of a motion vector detecting unit according to a fifth embodiment of the present invention.
Corresponds to the invention described in (1). 14, those parts that are the same as those corresponding parts in FIG. 2 are designated by the same reference numerals. The fifth embodiment of the present invention is characterized by the motion vector detection unit 156 of the area within the motion vector detection unit 15, and the block motion vector detection unit 151 has exactly the same configuration as that of the first embodiment. .

The area motion vector detection unit 156 includes a histogram creation unit 1521, a peak detection unit 1522, and a motion vector number inspection unit 1523. When the number of blocks in which a motion vector belonging to an area is detected is small, the motion vector number inspection unit 1523 makes the motion vector of the area indefinite. If not, the area detection unit 14
Based on the region information acquired from the block and the block motion vector detection unit 151, the histogram creation unit 1521 creates a histogram indicating the distribution of the motion vector of the block, and the peak detection unit 152.
The motion vector as a region is derived by examining the peak according to 2.

Next, the operation of the fifth embodiment of the present invention will be described with reference to FIGS. FIG.
Shows a flowchart for explaining the operation of the fifth embodiment of the present invention. In FIG. 15, the same processing steps as those in FIG. 6 are designated by the same reference numerals, and the description thereof will be omitted.

In FIG. 15, when the area-divided image is acquired from the area detection unit 14 in step ST22, the process shown in FIG.
The motion vector number inspection unit 1525 shown in FIG. 4 takes out a block in which a highly reliable motion vector belonging to the area is detected (step ST71) and checks whether the number of blocks is larger than a threshold value (step ST7).
2).

When the number of blocks is equal to or less than the threshold value, the motion vector of the area is undefined (step ST
73). On the other hand, when the number of blocks is larger than the threshold value, the histogram creation unit 1521 creates a histogram of motion vectors of the blocks existing in the area (step ST23) and the peak detection unit, as in the first embodiment. Accumulation of the motion vector having the highest frequency from the histogram of 1522 (step ST24) is sequentially performed. Then, the above-mentioned steps ST72 and ST
The processing of 73 or ST23 and ST24 is executed for all areas (step ST74).

According to the fifth embodiment, there is an effect of removing the motion vector of the region having low accuracy which occurs when the motion vector of the block existing in the region is small.

Next, a sixth embodiment of the present invention will be described. 16 is a block diagram showing the configuration of a motion vector detection unit according to the sixth embodiment of the present invention.
Corresponds to the invention described in (1). 16, those parts that are the same as those corresponding parts in FIG. 2 are designated by the same reference numerals. The sixth embodiment of the present invention is characterized by the motion vector detection unit 157 of the area within the motion vector detection unit 15, and the block motion vector detection unit 151 is exactly the same as in the first and fifth embodiments. It is a composition.

The area motion vector detecting unit 157 includes a histogram creating unit 1521, a peak detecting unit 1522, and a frequency checking unit 1524. The frequency inspection unit 1524 makes the motion vector of the region undefined when the ratio of the frequency of the maximum motion vector in the histogram detected by the peak detection unit 1522 is lower than the threshold value.

Next, the operation of the sixth embodiment of the present invention will be described with reference to FIGS. FIG.
9 is a flowchart for explaining the operation of the sixth embodiment of the present invention. In FIG. 17, the same processing steps as those in FIG. 6 are designated by the same reference numerals, and the description thereof will be omitted.

In FIG. 17, when the motion vector with the highest frequency is derived from the histogram by the peak detection unit 1522 in step ST23, the frequency detection unit 1524 shown in FIG. The ratio of the frequency of the maximum motion vector in the detected histogram to the entire frequency is calculated (step ST
81) and compares the ratio with a predetermined threshold value (step ST82).

As a result of the above comparison, when the above ratio is equal to or more than the threshold value, the motion vector of the area is made indefinite (step ST83). On the other hand, when the above ratio is smaller than the threshold value, it is stored as the motion vector of the area as in the first embodiment (step ST24). Then, with the above steps ST23, ST81, ST82,
The processing of ST83 or ST24 is executed for all areas (step ST84).

According to the sixth embodiment, there is an effect of removing a motion vector of a region with low precision which occurs when the motion vector values of blocks in the region are dispersed.

Next, a seventh embodiment of the present invention will be described. FIG. 18 is a block diagram showing the configuration of the motion vector detection unit according to the seventh embodiment of this invention. FIG.
The same components as those in FIGS. 2, 8, 10, 12, 14 and 16 are designated by the same reference numerals. The seventh embodiment is a combination of all the configurations in the motion vector detection unit 15 of the first to sixth embodiments.

That is, as shown in FIG. 18, the motion vector detecting section 15 includes a block motion vector detecting section 1
58 and an area motion vector detection unit 159. The block motion vector detection unit 158 includes a block division unit 1511, a contrast inspection unit 1513, a block matching unit 1512, a color error comparison unit 1514, and a color error distribution inspection unit 1515. Region motion vector detection unit 1
59 includes a motion vector number inspection unit 1523, a histogram creation unit 1521, a peak detection unit 1522, and a frequency inspection unit 1524.

Next, the operation of the seventh embodiment of the present invention will be described with reference to FIGS. 18, 19 and 20. FIG. 19 is a flowchart for explaining the operation of the block motion vector detection unit 158 according to the seventh embodiment of the present invention.
FIG. 20 is a flowchart for explaining the operation of the area motion vector detection unit 159 according to the seventh embodiment. In FIG.
The same processing steps as those in FIG. 5, FIG. 9, FIG. 11 and FIG. 13 are given the same reference numerals, and in FIG.
The same processing steps as those in No. 7 are denoted by the same reference numerals, and the description thereof will be appropriately omitted.

In the seventh embodiment, the block motion vector detection unit 158 divides the key frame image acquired by the block division unit 1511 into blocks (steps ST11 and ST12), and then the contrast inspection unit 1513.
Checks the contrast of each block for the first key frame image (steps ST43 and ST44), and the block matching unit 1512 detects all the neighborhoods of the second key frame image for the block of the first key frame image. The color difference from the block is compared and inspected (steps ST13 and ST14), and the position vector to the block having the smallest color difference is derived as a motion vector (step ST15).

Then, the color error when the color error comparing section 1514 calculates the motion vector is compared with a predetermined value, and the motion vector of the block having the color error larger than the threshold value is undefined (step ST55, ST6
3) The color error distribution inspection unit 1515 makes the motion vector of the block in which the color error distribution of the motion vector whose color error is smaller than the threshold value is biased near the minimum value is indefinite (steps ST61 to ST63).

Further, as shown in the flowchart of FIG. 20, the motion vector detecting unit 159 of the area determines that the motion vector number inspecting unit 1523 determines that the number of blocks in which the motion vector belonging to the area is detected is less than or equal to the threshold value. , Make the motion vector of the area indefinite (steps ST71, ST7
2, ST83), otherwise, the area detection unit 14
Based on the area information acquired from the block and the motion vector of the block acquired from the block motion vector detection unit 158, the histogram creation unit 1521 creates a histogram showing the distribution of the motion vector of the area, and the peak detection unit 152
The motion vector as a region is derived by examining the peak in step 2 (step ST23).

Then, the frequency checking unit 1524 determines the motion vector of the region in the histogram detected by the peak detecting unit 1522 when the ratio of the frequency of the maximum motion vector to the whole is lower than the threshold value. Indefinite (steps ST81 to ST83). When the ratio is smaller than the threshold value, the motion vector of the area is stored (step ST24), as in the first embodiment. Then, the above steps ST71, ST
The processing of 72, ST23, ST81, ST82 and ST83 or ST24 is executed for all the areas (step ST84).

According to the seventh embodiment, as described in the first to sixth embodiments, illegal motion vector removal can be performed in block units and region units, so that keyframes can be removed. It is possible to reduce the decrease in accuracy due to processing only the data.

[0078]

Next, the operation of one example of the first embodiment of the present invention will be described in detail. A fish image is used as an example of the moving image to be processed.

As already described with reference to the flowchart of FIG. 4, the data processing device 1 shown in FIG. 1 has the frame acquisition unit 12 for each frame of the digitized moving image data reproduced by the moving image reproduction unit 11. The key frame detection unit 13 measures the color histogram change and the inter-frame difference of the frame image, and inspects whether each is greater than or equal to a preset threshold value.

Here, the above-mentioned reproduced moving image data is shown in FIG.
It is assumed that the image data is such that the frame images of the fish can be temporally arranged and expressed. In the example of FIG. 21,
The screen is switched from the full black image 31 to the fish frame image 32, the fish frame images are changed as indicated by 33, 34 and 35, and then the screen is switched to the full black image 36. Here, if it is assumed that the screen is switched due to the color histogram change, the key frame detection unit 13 detects that the screen is switched from the images 31 to 32, and one frame image 32 of the fish is detected. By detecting the key frame image 32 ′ of the eye and further examining the inter-frame difference, the frame image 34 in which the amount of movement of the fish is large with respect to the frame image 32 is detected, and this is used as the second key frame image 34 ′. To detect. The detected key frame images 32 ′ and 34 ′ are 96 × 96.
It is sampled in pixels.

The area detection unit 14 detects color and edge information from the above 96 × 96 pixel key frame image, and generates an area division image corresponding to the key frame image based on the information. That is, the area detection unit 14 uses the 96 × 96 pixel key frame image shown in FIG.
The region-divided image of 24 × 24 pixels shown in is generated.

The motion vector detection unit 15 receives two temporally consecutive key frame images of 96 × 96 pixels from the key frame detection unit 13 and detects the array of 24 × 24 pixel motion vectors. Furthermore, a motion vector as a region is derived based on the 24 × 24 pixel region division image acquired from the region detection unit 14. The motion vector detection unit 15 sequentially processes all frames existing in the moving image file.

Details of the motion vector detecting process in the motion vector detecting unit 15 will be described with reference to FIG. FIG.
(A) shows a block motion vector detection process, and (b) shows a region motion vector detection process. The block division unit 1511 shown in FIG.
The first piece of the 96 × 96 pixel key frame image acquired in step 3 is 24 blocks in the vertical direction and 24 blocks in the horizontal direction (that is, 24 × 24 blocks, one block is 4 ×).
4 pixels).

That is, in FIG. 23, for convenience of illustration, the 24 × 24 blocks are actually shown as 8 × 8 blocks, but the block division unit 1511, as shown in FIG. The first key frame image (the key frame image on the left side in FIG. 23A) acquired by (3) is divided into blocks.

The block matching unit 1512 inspects all the blocks of the first key frame image for color differences from the blocks in the vicinity of the second key frame image, and determines the point where the color difference is the minimum as the corresponding point. The change in position is the motion vector. FIG. 23A shows the fish-eye block of the first keyframe image, and the corresponding point at which the color difference of the second keyframe image is the smallest with respect to this block is Since there are 3 blocks in the negative direction and 2 blocks are moved in the negative direction on the y-axis, the motion vector of the block is derived as (-3, -2).

Further, the area motion vector detecting unit 152
As described above, the block motion vector detection unit 1
Sequence of motion vector of block from 51 (24 × 24)
Then, the area division image (24 × 24) detected by the area detection unit 14 is acquired. Here, FIG. 23 (b)
As shown on the left side of, the arrow indicates the motion vector of the block. Note that, in the left-hand diagram of FIG. 23B, among the shaded portions, the blocks for which motion vectors are not shown indicate the portions in which the motion vectors have been deleted because the motion vectors are undefined.

The histogram creating section 1521 is shown in FIG.
In the diagram on the left side of (b), a histogram of motion vectors of blocks existing in the area is created. Peak detector 1
522 detects that the motion vector with the maximum frequency in the created histogram is the motion vector in the diagonally left upper direction, and as shown schematically in the diagram on the right side of FIG.
The motion vector in the shaded area is detected as diagonally upward to the left. The area associating unit 16 associates the motion vector of the area acquired from the motion vector detecting unit 15 with the area estimated to point to the same object based on the area information acquired from the area detecting unit 14. I do.

The present invention is not limited to the above embodiment, but the block motion vector detecting unit 15
It is also possible to combine any one of 1, 153 to 155 and any one of the area motion vector detecting units 152, 156, 157.

[0089]

As described above, according to the present invention,
Since the number of frames to be processed is reduced by performing area detection and motion vector detection for object tracking only on key frames, the object extraction processing can be sped up and the frames to be processed are processed. Although the number is reduced, keyframes that accurately express the content are processed, so the reduction in accuracy can be suppressed within a reasonable range.

Further, according to the present invention, the contrast value of the block is calculated and the motion vector of the block whose value is lower than the threshold value is made indefinite, so that the corresponding points are likely to have an error due to the small number of features. Since the motion vector of the block is excluded, it is possible to reduce the decrease in accuracy, and thus it is possible to reduce the decrease in the accuracy of object extraction due to processing only the key frame.

Further, according to the present invention, the color error when the motion vector is calculated by the block matching method is compared with a predetermined value, and the motion vector of the block whose error is larger than the threshold value is indefinite. This eliminates the influence of the incorrect vector that occurs when the corresponding points disappear because the intervals between the frames become empty because only the keyframes are processed. It is possible to reduce a decrease in the accuracy of object extraction due to processing only the object.

Further, according to the present invention, by making the motion vector of the block in which the color error distribution when the motion vector is calculated by the block matching method is biased near the minimum value is indefinite, blocks having similar colors can be determined. Illegal motion vectors that occur when there are a lot of them that happen to match one of them are eliminated, so the reduction in accuracy can be reduced, and by doing this, object extraction by processing only key frames It is possible to reduce the decrease in the accuracy of.

Further, according to the present invention, when the number of blocks in which a motion vector belonging to a region is detected is small, the motion vector of that region is made indefinite, so that the motion vector of the block existing in the region is small. In this case, since the motion vector of the low precision region is removed, it is possible to reduce the reduction in precision, and thus it is possible to reduce the reduction in the precision of object extraction due to processing only the key frame.

Furthermore, according to the present invention, when the ratio of the frequency of the motion vector having the maximum frequency in the histogram to the whole is lower than the threshold value, the motion vector of the region is made indefinite, and Since the motion vector of the low precision area that occurs when the motion vector values of the inside blocks are dispersed is removed,
The reduction in accuracy can be reduced, and thus the reduction in the accuracy of object extraction due to processing only the key frame can be reduced.

[Brief description of drawings]

FIG. 1 is a block diagram illustrating a configuration of a first exemplary embodiment of the present invention.

FIG. 2 is a detailed block diagram of a motion vector detection unit according to the first embodiment of the present invention.

FIG. 3 is a detailed block diagram of an area associating unit in FIG.

FIG. 4 is a flowchart for explaining the operation of FIG.

5 is a flowchart for explaining the operation of the motion vector detection unit of the block of FIG.

FIG. 6 is a flowchart for explaining the operation of the motion vector detection unit in the area of FIG.

FIG. 7 is a flowchart for explaining the operation of the area associating unit according to the first embodiment of this invention.

FIG. 8 is a detailed block diagram of a motion vector detection unit according to the second embodiment of the present invention.

FIG. 9 is a flowchart for explaining the operation of FIG. 8;

FIG. 10 is a detailed block diagram of a motion vector detection unit according to the third embodiment of the present invention.

11 is a flowchart for explaining the operation of FIG.

FIG. 12 is a detailed block diagram of a motion vector detection unit according to the fourth embodiment of the present invention.

13 is a flowchart for explaining the operation of FIG.

FIG. 14 is a detailed block diagram of a motion vector detection unit according to a fifth embodiment of the present invention.

15 is a flowchart for explaining the operation of FIG.

FIG. 16 is a detailed block diagram of a motion vector detection unit according to the sixth embodiment of the present invention.

17 is a flowchart for explaining the operation of FIG.

FIG. 18 is a detailed block diagram of a motion vector detecting unit according to a seventh embodiment of the present invention.

FIG. 19 is a detailed block diagram of a motion vector detection unit of the area in FIG.

20 is a detailed block diagram of a motion vector detection unit for the area in FIG.

FIG. 21 is an explanatory diagram of an example of the present invention.

FIG. 22 is an operation explanatory diagram of the area detection unit.

FIG. 23 is an explanatory diagram of an embodiment of the present invention.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 Data processing device 2 Storage device 11 Video reproduction part 12 Frame acquisition part 13 Key frame detection part 14 Area detection part 15 Motion vector detection part 16 Area association part 21 Key frame storage part 22 Area storage part 23 Motion vector storage part 24 Related Region storage unit 151, 153-155, 158 Block motion vector detection unit 152, 156, 157, 159 Region motion vector detection unit 161 Region prediction unit 162 Corresponding region matching unit 1511 Block division unit 1512 Block matching unit 1513 Contrast inspection unit 1514 Color error comparison unit 1515 Color error distribution inspection unit 1521 Histogram creation unit 1522 Peak detection unit 1523 Motion vector number inspection unit 1524 Frequency inspection unit

Claims (12)

[Claims] 1. A key frame detection unit that extracts a key frame image that is a key frame that accurately represents moving image content from moving image data, an area detection unit that generates an area divided image from the key frame image, and the key frame. A motion vector detection unit that detects a motion vector of a region from a key frame image acquired from a detection unit and a region division image acquired from the region detection unit, and the region division image and the motion vector detection obtained from the region detection unit Area mapping unit that derives the motion of the object from the motion vector of the area acquired from the part, and associates the areas estimated to point to the same object between frames, the key frame image, the area A moving image analysis apparatus comprising: a divided image and a storage device that stores at least a motion vector. 2. The motion vector detecting unit detects a motion vector of a block unit using a block matching method from two temporally consecutive key frame images acquired from the key frame detecting unit. A vector detection unit, a motion vector detection unit for a region that detects a motion vector of the region based on a motion vector of the block detected by the motion vector of the block and the region division image acquired from the region detection unit, The moving image analysis apparatus according to claim 1, further comprising: 3. The motion vector detection unit of the block divides one of the two temporally consecutive key frame images into a plurality of blocks, and the other of all of the divided blocks. A block matching unit that inspects a color difference with a block in the vicinity of a key frame image, and uses a point having a minimum color difference as a corresponding point and a change in the position as a motion vector. 2. The moving image analysis device described in 2. 4. The block motion vector detection unit divides one of the two temporally consecutive key frame images into a plurality of blocks, and a contrast value for each of the divided blocks. A contrast inspection unit that calculates and compares the calculated contrast value with a threshold value to make the motion vector of a block having a contrast value lower than the threshold value indefinite, and all blocks having a contrast value higher than the threshold value. With respect to the other key frame image, the difference in color from a nearby block is inspected, and a block matching unit that uses a point having the minimum color difference as a corresponding point and a change in the position as a motion vector is included. The moving image analysis apparatus according to claim 2. 5. The motion vector detection unit of the block divides one of the two temporally consecutive key frame images into a plurality of blocks, and the other of all the divided blocks. The color difference with the block in the vicinity of the key frame image is inspected, and the point where the color difference is minimized is used as the corresponding point, and the change in the position is used as the motion vector, and the block matching section calculates the difference. 3. A color error comparison unit that compares a color error of a motion vector of a block with a threshold value and makes a motion vector of a block having a color error larger than the threshold value indefinite. Video analysis device. 6. The motion vector detection unit of the block divides one of the two temporally consecutive key frame images into a plurality of blocks, and the other of all the divided blocks. The color difference with the block in the vicinity of the key frame image is inspected, and the point where the color difference is minimized is used as the corresponding point, and the change in the position is used as the motion vector, and the block matching section calculates the difference. A color error distribution inspecting unit that calculates a bias near the minimum value of the color error distribution of the motion vector of the block, and makes the motion vector of the block of the color error distribution in which the bias is larger than a threshold value indefinite. The moving image analysis apparatus according to claim 2, which is characterized in that. 7. The motion vector detection unit of the area acquires the area division image from the area detection unit and the motion vector from the motion vector detection unit of the block, and the motion vector existing in the area division image. 7. The method according to claim 2, further comprising: a histogram creating unit that creates the histogram, and a peak detecting unit that detects a motion vector having a maximum appearance frequency in the created histogram as a motion vector of the region. The moving image analysis apparatus according to any one of the above. 8. The area motion vector detection unit acquires the area division image from the area detection unit, acquires the motion vector from the motion vector detection unit of the block, and acquires the motion vector of the block existing in the area division image. When the number of motion vectors is less than or equal to the threshold value, the motion vector number inspection unit that makes the motion vector of the region indefinite, and the number of motion vectors of the block existing in the region divided image by the motion vector number inspection unit is as described above. If there are more than the threshold,
A histogram creation unit that creates a histogram of the motion vectors existing in the region division image based on the region division image obtained from the region detection unit and the motion vector of the block obtained from the motion vector detection unit of the block; 7. The moving image according to any one of claims 2 to 6, further comprising: a peak detection unit that detects a motion vector having a maximum appearance frequency in the created histogram as a motion vector of the region. Analyzer. 9. The motion vector detection unit of the area acquires the area division image from the area detection unit and the motion vector from the motion vector detection unit of the block, and the motion vector existing in the area division image. Of the motion vector detected by the peak detection unit, and a peak generation unit that detects a motion vector with the highest appearance frequency in the created histogram, 7. The moving image analysis apparatus according to claim 2, further comprising a frequency inspection unit that makes a motion vector having a ratio equal to or higher than the threshold value indeterminate. 10. The key frame detection unit respectively measures a color histogram change and an inter-frame difference of the frame image acquired by the frame acquisition unit, and at least one of the color histogram change and the inter-frame difference is a threshold value. The moving image analysis apparatus according to claim 1, wherein a frame image larger than the frame image is extracted as the key frame image. 11. A key frame image that is a key frame that accurately represents the content of the moving image is extracted from the moving image data,
After generating a region segmented image from this keyframe image,
A motion vector of a region is detected from the key frame image and the region-divided image, and the motion of the object is derived from the motion vector of the region and the region-divided image, and the motion of the region estimated to point to the same object is detected. A moving image analysis method characterized in that correspondence is made between frames. 12. The motion vector of the area is detected by detecting a block-by-block motion vector from the two temporally consecutive key frame images by using a block matching method. The moving image analysis method according to claim 11, wherein a motion vector of the region is detected based on the region divided image.