JP3332166B2 - Video search device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing device such as a multimedia processing system or a video device, and a technique for retrieving a moving image in such a device. a scene desired by the search equipment suitable moving image to efficiently search are those relating <br/>.

[0002]

2. Description of the Related Art In recent years, the importance of moving image information as information media has been dramatically increasing, and an editing support environment that allows ordinary people to easily create and utilize moving image information is required. In editing moving images, there is a need for a function of efficiently searching a scene required by an editor from many moving image materials.

In general, a moving image search is performed by creating an index of a moving image based on a keyword or the like.
In this search, designation of a search target can be relatively easily performed by inputting text information serving as an index. However, there is an essential problem that a search cannot be performed using a keyword that is not registered as an index. This is a major dissatisfaction for the user, especially in creative activities such as video editing.
There is also a problem that the work of registering the index must be performed by a human.

[0004] For the purpose of solving such a problem, for example, a technique by the present inventors has been proposed by the Information Processing Society of Japan, “Transactions of Information Processing Society of Japan, Vol.
o. Pp. 41992 ”. "Automatic Indexing Method and Object Retrieval Method for Color Video Images" at pages 543-550
(Nagasaka, Tanaka). With this technology, you only need to specify a single color image that contains the object you are trying to find, and all frames in which the object appears (a series of still images that form a moving image. 30
(Which consists of a single frame) from a color moving image. Hereinafter, the details of this technique will be described.

As a technique for determining whether or not a specified target exists in an image, there are various algorithms including pattern matching. However, in the search for a moving image, the size and shape of the object in the frame change every moment because the posture and position of the object change. Therefore, it must be robust to such deformation. Further, in order to process a huge number of frames of a moving image, high speed is indispensable.

[0006] The object retrieval techniques described in the above-mentioned documents have been devised so as to satisfy the above-mentioned conditions. The algorithm is roughly divided into the following two basic processes. (A) A process of extracting characteristic information from an image of a designated target object (reference image). (B) A process of determining whether or not the extracted information is included in an arbitrary frame.

In the process (a), pairs of adjacent colors (color pairs) on the color image are extracted as information characterizing the image of the object. Such a color pair is local characteristic information of the object, and is information unique to the object image that is invariable in the size and orientation of the object. In the process (b), it is checked whether or not the reference color pair selected in the process (a) exists in the frame to be determined. At this time, if N (for example, about N = 16) reference color pairs, which are characteristic information of the object, are simultaneously present in the frame, it is determined that the frame contains the object. . Hereinafter, the details of the process (a) will be described with reference to FIG. 17, and the details of the process (b) will be described with reference to FIG.

FIG. 17 is a flowchart showing an example of a conventional method for selecting characteristic information of a color moving image object.
The feature extraction process from the color reference image is performed as follows. First, the searcher specifies one color image (hereinafter simply referred to as an image) in which the object appears (step 17).
01) Only the image area of the object is cut out as a reference image (step 1702). Next, the reference image is divided into rectangular areas (hereinafter referred to as cells) (step 170).
3) The following processing is performed for each cell (step 170).
4). First, a color histogram is obtained (step 170).
5) Extract all colors (referred to as cell colors) whose frequency in the histogram is larger than the set threshold and register them in the list CC (steps 1706 to 1708). Table 2 shows an example of the data structure of the list CC.

[Table 2]

Next, the following processing is performed for each cell on the reference image (step 1709). First, a pair (color pair) is created by selecting one color from the cell color of a cell and the cell color of each cell adjacent thereto (step 1710). Create this color pair by the number of cell colors of the two cells,
Register all combinations in list CP (step 171)
1). At this time, the combination of cell colors in each cell is also registered in the CP. Further, from the list CP, the N highest-frequency color pairs are selected and registered in the list RCP (step 17).
12). These N color pairs are used as information characterizing the object (referred to as reference color pairs). Examples of the data structures of the list CP and the list RCP are shown in the following Tables 3 and 4, respectively.

[Table 3]

[Table 4]

FIG. 18 is a flowchart showing an example of a conventional color moving image object search method. This flowchart determines whether or not the frame to be determined includes the object characterized in FIG.
The frame to be determined is divided into cells (1801), and the following processing is performed on each cell to determine the cell color of each cell (step 1802). First, a color histogram is obtained (step 1803), and all colors (cell colors) whose frequency in the histogram is larger than a set threshold are extracted and registered in the list CC (steps 1804 to 1806).

Next, the following processing is performed for each color pair registered in the list RCP to determine the number of appearances in the above-mentioned frame (step 1807). First, each of the cells on the frame (step 1808) has one of the two colors constituting the color pair as a cell color (step 18).
09), as for the other color of the color pair, all the cells (hereinafter referred to as valid cells) which the cell itself or any one of its adjacent eight cells has as a cell color are extracted (step 1810). , The total number of valid cells in the above-described frame is obtained (step 1811).

If the number of effective cells exceeds the set threshold value (step 1812), the color pair is counted as an effective color pair (step 1813), and the total number of the effective color pairs is compared with the set threshold value (step 1813). Step 1814), and it is determined whether or not the object exists in the frame (steps 1815 and 1816).

FIG. 19 is an explanatory diagram showing a search example using the algorithm shown in FIG. In this figure, 19
Reference numeral 1 denotes a frame including a target object (girl) serving as a reference for search, and a partial area surrounded by a frame therein is a reference image of the target object; 192, a frame including the search target object; A frame that does not contain an object,
94 and 195 are distribution maps in which the presence of 16 types of reference color pairs has been detected. In the example of this drawing, the number of reference color pairs determined to be present is 14 in the distribution map 194 and 8 in the distribution map 195. Therefore, if the threshold is set between 8 and 14, it can be correctly determined that the target exists in the distribution map 194 and does not exist in the distribution map 195.

However, as a result of experiments on various video samples, the above-mentioned conventional algorithm includes the following Table 1.
As can be seen from the data in the column described as "Conventional," it was found that detection omission and excessive detection occurred to a considerable extent. Visual observation during the experiment suggests that the following two points are the biggest causes of these malfunctions. (B) How to select a reference color pair. (B) Color change of the object due to color temperature change of the light source. Hereinafter, these two points will be considered in detail.

The effect of (a) how to select a reference color pair will be described. Many of the reference images of the object can be divided into a plurality of partial regions having a uniform color. Taking the woman shown in the image of FIG. 20 as an example, it is possible to consider a partial region mainly having a black, flesh-color or green color for each component such as hair, face and clothes. At this time, a pair of colors with low similarity, such as a pair of skin color and green (hereinafter, referred to as
The unique pair is the boundary between the elements (in this case, the boundary between the face and the clothes)
It exists only in a very small area.

However, a number of similar color pairs such as pairs of green colors (same color pairs) exist depending on the size of each partial region. Therefore, when only the color pair having the highest appearance frequency is used as the reference color pair, the same color pair occupies the majority of the reference color pairs. Then, as will be described with reference to FIG. 20, the same color pair often appears on the frame even when the target object does not appear, as compared with the different color pair. In this case, in order to prevent omission of detection, the number of effective color pairs must be determined with a small threshold, and excessive detection increases.

FIG. 20 is an explanatory diagram showing an example of a reference color pair extracted from a reference image. In the figure, for each of the 16 reference color pairs extracted from the reference image 291, two pairs of RGB color space coordinates 292, 293 of two colors forming a pair
And the color difference 294 between the two colors. RGB
Each coordinate value of the color space coordinates 292 and 293 represents the luminance of each of the RGB components and takes a value from 0 to 7. The color difference 294
Is simply the distance between the RGB coordinates. The color pair with the large color difference 294 is the fourth, third, fifth, eighth and tenth in the order.
The third color pair is the same achromatic color pair, and the color difference is close to human perception. From this result, it can be seen that the same color pair occupies most of the reference color pair.

The effect of the change in the color temperature of the light source (b) will be described. Even in the image where the same object appears, if the objects are under light sources with different color temperatures,
The images of the object are represented in different colors. Therefore, it is not appropriate to use a reference color pair extracted from a reference image captured under a certain light source as it is for determining a frame captured under a light source having a different color temperature. This is one of the major causes of missed detection.

On the other hand, as a moving image search technique, there is a technique for detecting a cut transition based on, for example, a change in correlation between adjacent frames of a moving image. And
Utilizing such a technique, as shown in FIG.
Systems for improving the efficiency of video editing have been developed.

FIG. 21 is a block diagram showing the configuration of a conventional color moving image cut search and display system. In the figure, reference numeral 2101 denotes a moving image reproducing apparatus which includes devices such as a video tape recorder and a write-once laser disc player, and converts a video signal from these devices into digital image data (moving image) and outputs the digital image data (moving image). The device, 2102,
A cut detection device 2103 that detects a transition between cuts of a moving image sent from the moving image reproducing device 2101 includes:
Based on the cut detection operation of the cut detection device 2202, an icon screen for each cut is generated from the moving image transmitted from the moving image reproduction device 2101, and the display 210
The cut display control device 2105 to be displayed on the display 2104 is generated by the cut display control device 2103 and displayed on the display 2104.
It is a cut display screen consisting of a group of icons displayed above.

When a moving image is supplied from the moving image reproducing device 2101, the cut detecting device 2202 detects a change in cut based on a change in correlation between adjacent frames of the moving image. The cut display control device 2103 displays reduced images of the frame at the beginning of each cut as icons, in order, based on the detection operation of each cut change by the cut detection device 2202, and
The time length of each cut is recorded, and the time length is displayed as a thickness in the depth direction of each icon.

In this way, by displaying the time length of each cut and the icon composed of images in order, the temporal structure of the video can be visualized, and the editing for each cut can be performed efficiently. It becomes possible. However, this technique is a search that uses a cut as a minimum unit, and cannot support more detailed editing work for characters and the like.

[0023]

The problem to be solved is that, in the prior art, in a technique for determining a frame including a specific object, a color change of the object due to a method of selecting a reference color pair and a change in color temperature of a light source. Is not taken into account, it is not possible to prevent omission of detection of frames containing objects from color moving images, and in the color moving image cut search and display system, editing of color moving images requires operations in cut units. It is limited and cannot support more detailed editing work. An object of the present invention is to solve the problems of the related art, accurately detect a frame including an object from a moving image, and improve the reliability and performance of the technology related to the moving image detection, and is to <br/> provide search equipment of the moving image to allow more support detailed editing of the moving image.

[0024]

In order to achieve the above object, a moving image search apparatus according to the present invention comprises the following steps: (1) For a plurality of regions included in an object on a reference screen, each image region has a similar color. Means for creating a group (color group) for each color, means for combining the different color groups based on the adjacency of the corresponding partial areas to form a color group pair, and means for creating the color group The pair includes means for detecting a frame (specific frame) having a value equal to or more than a predetermined value from the moving image. In addition, (2) the above (1)
In the moving image search device described in the above, the cut of the moving image is displayed as a cube with the depth of the time length, and further, a position corresponding to the time length of the specific frame group that is temporally continuous is colored. It is characterized by displaying things. (3) In the moving image search device described in (2), the temporally first specific frame included in the cut is arranged and displayed at a temporally corresponding depth position of the cube. It is characterized by:

[0025]

In the present invention, a group (color group) in which similar colors are put together is paired (color group pair), and based on this color group pair, detection of a frame including an object from a color moving image is performed. Do. Further, after performing color correction of a color (unique color) unique to the image or the object, the determination for the frame is performed. This makes it possible to stably detect a pair of color groups even with a difference in the color temperature of the light source, and to greatly reduce omission in detection. Also, paying attention to the distribution characteristics of the different color group pairs and the same color group pairs, the same color group pairs and the different color group pairs are treated separately. By setting, excess detection can be reduced. Further, in determining whether or not an object exists in a frame, excessive detection can also be reduced by taking into account the adjacent direction between color group distribution areas. By using such a technique for determining the presence / absence of a specific object in a color image, a search for a color moving image can be performed in a frame unit that is more detailed than a cut unit.

[0026]

Embodiments of the present invention will be described below in detail with reference to the drawings. 1 and 2 are flowcharts showing a first embodiment of a method for determining the presence / absence of a specific object in a color image according to the present invention, and FIG. 3 is a block diagram showing a configuration of an apparatus used for carrying out the method. is there. In FIG. 3, reference numeral 1 denotes a computer for determining the presence or absence of a specific object in a color image of the present invention, and 2 denotes a CR for displaying an output screen of the computer 1.
Display such as T, 3 is a moving image reproducing device such as an optical disk or a video deck, 4 is an A / D converter for converting an analog signal into a digital signal, 5 is a control signal between the moving image reproducing device 3 and the computer 1 6 is an external storage device such as a hard disk; 7 is an input device such as a mouse; 8a to 8e are interfaces for controlling connection between the computer 1 and peripheral devices (indicated as I / F in the figure); Is a CPU that controls the processing of the computer 1
(Central processing unit) 10 is a memory directly accessed by the CPU 9.

The video signal output from the moving image reproducing device 3 is sequentially converted into digital image data by the A / D converter 4 and sent to the computer 1. In the computer 1, digital image data enters the memory 10 via the interface 8b, and is processed by the CPU 9 according to a program stored in the memory 10.

Each frame of a moving image handled by the moving image reproducing device 3 is numbered (frame number) sequentially from the beginning of the moving image. By transmitting the frame number from the computer 1 to the moving image reproducing device 3 via the control line 5,
The moving image of the scene is reproduced. Note that various information is stored in the external storage device 6 as necessary for processing. Memory 10
Stores various lists created by the processing described below, and is referred to as necessary.

With such a configuration, in consideration of the following points, processing relating to the method for determining the presence / absence of a specific object in a color image according to the present invention is performed to avoid omission in detection. That is,
When the object moves from the sun to the shade, the colors representing the object are generally low in brightness (shadow color), so similar colors are one regardless of the change in brightness. Grouped into two groups (color groups) and treated as the same color. As a result, a certain degree of frequency is always ensured as the total number of the color groups when the light source changes, such as sunshine or shade. Furthermore, pairs of such color groups (color group pairs) are also robust to such changes in the light source. Then, when the image area of the target object is divided into partial areas composed of colors regarded as being in the same group, the color group is a feature amount representing each partial area. In addition, the adjacency between the color groups is determined by the adjacency between the partial areas represented by the color group. Hereinafter, particularly the processing operation of the method for determining the presence or absence of a specific object in a color image of the present invention, which relates to a different color group pair,
This will be described with reference to FIG. 1 and FIG.

First, the process of forming a different color group pair for reference in FIG. 1 will be described. The searcher is designated one image showing the target object (step 101), and a plurality of sub-regions having substantially the same color are divided from the image region of the target object (step 102). For example, when the target object is a person, the target object is divided into three partial areas of three equal colors of hair, face, and clothes. The following processing is performed for each of these partial regions r (step 103).

First, a color histogram is obtained (step 1).
04), N colors having the highest frequency among similar colors are sequentially selected and grouped to form a color group, and as shown in the following Table 5,
Register in the list CG (r) (step 105).

[Table 5] Then, these color groups are combined based on the adjacency of the corresponding partial areas to form a pair (reference color group pair). That is, all pairs of adjacent partial regions (r1,
r2) (step 106), list CG of color groups
A pair of (r1) and CG (r2) is created and registered in a different color group pair list RCGP as shown in Table 6 below (step 107).

[Table 6] In this way, for example, two pairs of hair and face and face and clothes are made.

Next, a search process according to the present invention using the different color group pairs thus obtained will be described with reference to FIG.
The frame to be determined is divided into cells (step 20).
1) The following processing is performed on each cell to determine the cell color of each cell (step 202). First, a color histogram is obtained (step 203), and all colors (cell colors) in the histogram whose frequency is greater than a set threshold are extracted (steps 204 and 205) and registered in the list CC (step 206).

Next, the following processing is performed for each of the reference color group pairs registered in the list RCGP obtained in FIG. 1, and the reference color group pairs (CG (r1), C
G (r2)) is determined (step 207). First, of the cells C (x, y) on the frame (step 208), two color groups CG (r1), C
One of the colors belonging to one of the color groups G (r2) is used as a cell color (step 209), and any of the colors belonging to the other color group of the color group pair is assigned to the cell itself or its neighbor. The cell C (x, y) which any one of the eight cells has as a cell color is extracted as a valid cell (step 210), and the total number in the frame is obtained (step 211).

In each reference color group pair, if the number of valid cells in the frame is equal to or larger than the set threshold (step 21).
2) The color group pair (CG (r1), CG (r2)) is counted as an effective color group pair (step 213). If the total number of counted effective color group pairs is equal to or greater than the set threshold value (step 214), it is determined that the object is present in the frame (step 215), and the total number of valid color group pairs is set to the set threshold value. If it is below, it is determined that the object does not exist (step 216).

In the continuous determination experiments shown in FIGS. 1 and 2, new 2 results in Table 1 below were obtained.

[Table 1] As shown in Table 1, the upper and lower rows show the same tendency, and it can be seen that detection omission can be greatly reduced by grouping colors. However, in this determination method, the number of overdetections increases, while the number of detection omissions can be drastically reduced. In order to reduce such overdetection, a different color group pair (a pair of cells of a different color group) and a same color group pair (a pair of cells of the same color group) are used, and the distribution of the different color group pair and the same color group pair are used. It is effective to pay attention to the difference in the characteristics of the distribution and to reflect it in the determination processing.

That is, the effective cells of the different color group pair exist only around the boundary between the partial regions, but the effective cells of the same color group pair exist by the area of the partial region corresponding to the color group. For this reason, the number of effective cells of the same color group pair is much larger than that of the different color group pair, and if the same color group pair and the different color group pair are determined with the same set threshold, excessive detection occurs. Therefore, as shown in FIG. 4, in the case of the same color group pair, the determination of the effective color group pair is performed using a cell number threshold value higher than that in the case of the different color group pair.

FIG. 4 is a flowchart showing a second embodiment of the method for determining the presence or absence of a specific object in a color image according to the present invention. By the same processing as steps 201 to 205 in FIG. 2, all colors (cell colors) whose frequency of the color histogram is larger than the set threshold are extracted and registered in the list CC (step 401). Similarly, by the same processing as in steps 207 to 213 in FIG. 2, the effective color group pair (CG (r1), CG (r
2)) is obtained and counted (steps 402 and 40).
3). Then, the processing according to the second embodiment is performed.

That is, each color group CG (r1), CG (r
The following processing is performed for each of 2) (Step 4)
04). First, cells whose colors registered in the color group CG are registered in the list CC (x, y) are extracted from each cell, counted as valid cells, and the number of valid cells of the same color group pair is obtained. (Steps 405 to 407). Then, based on the set threshold value t2 that is larger than the set threshold value t1 used in the process of Step 403, it is determined whether or not the set is an effective color group pair (Step 408), and the color group CG determined to be an effective color group pair.
Is counted (step 409). If the total number of the effective color group pairs thus obtained is equal to or greater than the set threshold value t3 (step 410), it is determined that the object exists in the frame (step 411). If the total number of pairs is equal to or smaller than the set threshold, it is determined that the target object does not exist (step 412).

The results of the continuity determination experiment performed in this way are shown in the above-mentioned new 3 in Table 1. Previous New 2
In addition to the different color group pairs, three same color group pairs corresponding to hair, face, and clothes are included in the reference color group pairs. Thus, it can be seen that the number of overdetections has decreased while the number of detection omissions has been kept low. In other words, it is clear that the effect of reducing excessive detection is obtained by setting the threshold value of the effective cell high for the same color group pair.

Next, a technique for further reducing excessive detection will be described. In normal images, the object does not rotate upside down, so in the object image, the fluctuation in the adjacent direction between the partial regions corresponding to each color group is:
Almost subsides within about ± 45 degrees. Therefore, in the process of determining the existence of the reference color group pair, by adding the orientation between adjacent partial regions to the determination condition, the determination of the frame including the target object can be made more reliable. Note that in order to determine an area adjacent in a specific direction,
Although various methods are conceivable, here, a simple method is adopted as much as possible in order to suppress an increase in processing time.

For example, in the frame to be determined, the effective cells of the different color group pair exist only near the boundary between the distribution regions of the two color groups forming the different color group pair if they appear correctly. Conversely, no effective cell of the same color group pair should exist in a region other than the vicinity of the distribution region of the different color group pair. Therefore, by paying attention to the effective cells of the different color group pair and removing the effective cells of the same color group pair that are not in the vicinity thereof by masking them from the distribution map, it is possible to reduce noise that appears in an area unrelated to the object. When masking, the directionality of the color group pair is also taken into consideration in order to further enhance the effect. Less than,
A third embodiment of the method for determining the presence / absence of a specific object in a color image according to the present invention in consideration of the directionality of a color group pair will be described with reference to FIGS.

FIGS. 5 and 6 are flowcharts showing a third embodiment of the method for determining the presence / absence of a specific object in a color image according to the present invention. First, a process of extracting a reference color group pair from a reference image will be described with reference to FIG. By the same processing as in steps 101 to 107 in FIG. 1, the color group list C for each subregion r of the image region of the target object
G (r) is registered (step 501). And
A pair of color groups CG (r1) and CG (r2) is created for all adjacent partial region pairs (r1, r2) and registered in the list RCGP '(steps 502 and 503).
At this time, the direction (hereinafter referred to as a reference direction) of the area of the other color group when the area of one color group forming the color group pair (CG (r1), CG (r2)) is used as a base point is extracted. Is registered in the list RCGP 'as shown in the following Table 7 (step 5).
04). This step 504 is a newly added process,
This is a feature of the third embodiment. The reference direction is expressed in all eight directions, that is, up, down, left, and right. In this experiment using this embodiment, the direction was designated manually.

[Table 7]

Next, a description will be given, with reference to FIG. 6, of a search process using the reference color group pairs with a direction thus obtained. Similar to the processing of step 401 in FIG. 4, by the same processing as steps 201 to 205 in FIG. 2, all colors (cell colors) whose frequency of the color histogram is larger than the set threshold are extracted and registered in the list CC (step 601).
Similarly to the processing of steps 402 and 403 in FIG. 4, by the same processing as in steps 207 to 213 in FIG. 2, the effective color group pair (CG
(R1), CG (r2)) are obtained and counted (steps 602, 603). Further, steps 404 to 404 in FIG.
Similarly to the processing of step S407, the color groups CG (r1), CG
In each of (r2) (step 604), the colors registered in the color group CG are listed from each cell in the list CC (x,
The cells registered in y) are extracted and counted as valid cells, and the number of valid cells of the same color group pair is obtained (steps 605 to 607).

After calculating the number of valid cells for each cell C (x, y), in the third embodiment, the color group CG (r
1) The effective cell is removed by a mask in the vicinity of the distribution area on the CG (r2) frame (step 608).
Then, the total number of remaining valid cells is equal to the setting threshold value t2 (greater than the setting threshold value t1 used in the process of step 603).
If this is the case (step 609), the color group CG is determined to be an effective color group pair and counted (step 610).

If the total number of effective color group pairs obtained in this way is equal to or greater than the set threshold value t3 (step 611), it is determined that the object exists in the frame (step 612). The total number of effective color group pairs is equal to the set threshold value t3.
If it is below, it is determined that the object does not exist (step 613). In the third embodiment, Step 60 is performed.
8 is exactly the same as the algorithm of FIG. 4 except that the mask processing of FIG. 8 is added. Hereinafter, the mask processing in step 608 will be described with reference to FIG.

FIG. 7 is an explanatory view showing a fourth embodiment of the method for determining the presence or absence of a specific object in a color image according to the present invention. In the figure, reference numeral 71 denotes a reference color group pair obtained from the reference image in the processing in FIG. 5, 72 denotes a distribution map of the main color group in the reference color group pair 71, 73 denotes a distribution map of the sub color group, 74
Is a color group pair distribution map, 75 is a direction mask obtained by expanding the color group pair distribution map in the reverse direction of the reference direction, and 76 is a direction mask obtained by expanding the color group pair distribution map in the reference direction. , 77 are distribution maps of the main color group from which noise has been removed using the direction mask 75, and 78 is a distribution map of the subordinate color group from which noise has been removed using the direction mask 76.

The mask processing is performed as follows.
First, in the color group pair 74 (CG (r1), CG (r2)), starting from each of the effective cells, a length m cells (here, 4 cells) and a width in the reference direction are set in the reference direction. A rectangular area of n cells (here, 2 × 2 cells) in a direction orthogonal to the above is obtained for all effective cells. That is, the direction mask 76 is obtained. Then, of the color groups forming the color group pair, the color group CG (r2) that is not the reference point in the reference direction
For each valid cell, invalid cells that are not included in any of the rectangular areas are invalidated and subtracted from the total number of valid cells. That is, the distribution map 73 of the slave color group and the direction mask 7
6 is performed, and a distribution map 78 of the slave color group from which noise has been removed is obtained.

Next, the color group pair (CG (r1), CG (r
2)), with each effective cell as a starting point, a length of m cells (here, 4 cells) in a direction opposite to the reference direction, and a width of n cells (here, 2 × 2) in a direction orthogonal to the reference direction. Cell)). That is, the direction mask 75 is obtained. Then, among the color groups forming the color group pair, for each valid cell of the color group CG (r1) that is the reference point in the reference direction, the valid cells that are not included in any of the rectangular areas obtained by the number of valid cells are determined. Invalid and subtracted from the total number of valid cells. That is, AND processing is performed between the distribution map 72 of the main color group and the direction mask 75 to obtain the distribution map 77 of the sub color group from which noise has been removed.

As described above, the result of the continuous determination experiment that invalidates the same color group pair other than the vicinity of the region where the different color group pair is distributed is as follows.
As shown in New 4 in Table 1 above, compared to New 3,
In both the upper and lower stages, the detection omission remains 0, and the excess detection is greatly reduced. That is, it can be seen that excessive detection can be greatly suppressed by adding the determination of the adjacent direction to the reference color group pair.

The embodiment described with reference to FIGS. 1 to 7 is a specific object presence / absence determination process based on a color group pair.
As shown in the following FIG. 8, by using different color pairs, it is possible to determine the presence / absence of a specific target in a color image with higher reliability compared to the related art using only color pairs.

FIG. 8 is a flowchart showing a fifth embodiment of the method for determining the presence / absence of a specific object in a color image according to the present invention. In this embodiment, a list CP is obtained (step 801) according to the conventional technique shown in FIG.
It is characterized in that a different color pair is selected from this list CP and used as a reference color pair (step 802). For example, when a person is a target, the different color pair is selected so as to be a combination of colors among elements of hair, face, and clothes.

According to the processing steps of this experimental example, the result of the continuity determination for the same frame as that of the conventional method is shown in the new column 1 of Table 1 described above. It can be seen that the addition of the process of appropriately selecting a different color pair greatly reduces overdetection as compared with the conventional technology.

Although the description of the embodiment shown in FIGS. 1 to 8 has been made on the case where all the processes are executed by the CPU by a program, the respective processes are executed by electronic circuits as necessary. it can. Further, by performing color correction and freely selecting and combining the processing used in each embodiment according to the video, the reliability of the determination result can be further improved. Hereinafter, such an embodiment will be described with reference to FIGS. 9 and 10. FIG.

FIG. 9 is a block diagram showing an embodiment of the structure according to the present invention of the apparatus for determining the presence or absence of a specific object in a color image according to the present invention. The apparatus 90 for determining the presence / absence of a specific object in a color image according to the present embodiment performs a main control unit 91 for controlling the processing operation of the entire apparatus and input / output control of color image information with a moving image reproducing apparatus, a display apparatus, and the like. An image input / output unit 92, a color correction unit 93 that performs color correction (for example, hue, saturation, lightness, and the like) on an input image according to the present invention, and a specification specified by a user based on the input image. And a color group extraction unit 95 according to the present invention for extracting a plurality of similar color partial regions as color groups from the specified image region of the target object. A unique color selection unit 96 according to the present invention for selecting a pair of adjacent color groups indicating the characteristics of a specific object from a plurality of color groups extracted by the color group extraction unit 95; Color for registering the selected color group pair Versus the registration unit 97
The presence / absence determination unit 98 according to the present invention for determining whether or not a specific target object is included in any input image with reference to the color group pair registration unit 97; A determination method storage unit 99 stores processing contents selected and designated by the user, such as execution conditions and the respective determination processing methods described with reference to FIGS.

When an image of a specific object (reference image) is input to the image input / output unit 92, the color correction unit 93 performs color correction of this image based on the storage contents of the determination method storage unit 99. Or output without color correction. In this color correction, for example, an auto white balance method widely used in video cameras is applied, and the reference image is kept at a specific value or an approximate value regardless of the light source.

For the image from the color correction section 93, the object specifying section 94 cuts out the image area of the specific object specified by the user as a reference image. Then, the color group extraction unit 95
Extracts a plurality of partial regions of similar colors from the image region of the specified object as a color group, and sends the extracted color regions to the unique color selection unit 96. The unique color selection unit 96 selects a pair of adjacent color groups indicating the characteristics of the specific object from the color groups received from the color group extraction unit 95, and registers the pair in the color group pair registration unit 97. At this time, the unique color selection unit 96 registers the information in the color group pair registration unit 97 together with information such as the reference direction in the different color group pair based on the storage content of the determination method storage unit 99.

Next, when an image to be determined is input to the image input / output unit 92, the color correction unit 93 performs correction on the above-described reference image based on the storage contents of the determination method storage unit 99. Correspondingly, the image is subjected to color correction or output without color correction. Alternatively, the color correction unit 93 performs color correction of the color group pairs registered in the color group pair registration unit 97 corresponding to the light source of this image based on the storage contents of the determination method storage unit 99. Color group pair
It is sent to the presence / absence determination unit 98.

The presence / absence judging section 98 uses the color group pair registered in the color group pair registering section 97 or the color group pair color-corrected by the color correcting section 93 to convert the image from the color correcting section 93 into an image. It is determined whether or not a specific object is included. At this time, the presence / absence determination unit 98 performs determination using different set thresholds for, for example, the same color or different color group pair and a different color or color group pair based on the storage content of the determination method storage unit 99, Alternatively, the existence of a pair is determined by using the adjacent direction of the sense of the distribution area of the color or the color group to determine whether or not a specific object is included.

As described above, in the determination processing by the presence / absence determination section 98, color correction of images and color groups is performed by the color correction section 93. In the case where an attempt is made to search for the same object under, that is, when the hue is greatly changed due to a change in the color temperature of the light source, an accurate determination can be made. Specifically, after extracting a color unique to the object from the reference image, the unique color is shifted according to the difference between the light source of the image to be searched and the light source of the reference image. As a result, the unique color of the object can be correctly detected, and the presence or absence of the object under a light source different from that of the reference image can be correctly determined.

FIG. 10 is a flowchart showing the processing operation of the apparatus for determining the presence / absence of a specific object in a color image in FIG. 9 according to the present invention. First, when a reference image including a specific object is input to the image input / output unit 92 in FIG. 9, the input reference image is input to the color correction unit 93 in FIG.
Performs color correction (step 1001). Then, based on the color-corrected image, the object specifying unit 94 shown in FIG.
And a color group extraction unit 95 extracts a plurality of colors unique to a specific land-saving object (step 1002), collects similar colors into a color group, and sets this color group as a specific target specific color. 9 in the color group pair registration unit 98 (step 100).
3).

Next, when the image to be determined is input to the image input / output unit 92 of FIG. 9, the color correction unit 93 of FIG. 9 corrects the color of the input determination image or inputs the image. The color group pair registration unit 9 in FIG. 9 based on the brightness of the image for determination and the like.
8 is corrected (step 1).
004). Then, the presence / absence determination unit 98 in FIG. 9 performs determination using different set threshold values for the same color or color group pair and a different color or color group pair (step 1).
005), the existence of a pair is determined using the adjacent direction between the distribution regions of the color or the color group (step 1006).
In this way, it is determined whether or not the unique colors appear as a pair in the image to be determined (step 100).
7).

In this embodiment, the determination method storage unit 99 shown in FIG.
Can be selected on the basis of the stored contents of step 100, and the user can select step 100 enclosed by a double frame in the figure.
1, 1003 to 1006 can be arbitrarily combined. Steps 1001 and 1004
As described with reference to FIG. 9, the color correction by using the auto white balance method widely used in video cameras and the like, and applying the colors of the same object in the reference image and the image to be determined to the light source, respectively. Instead, it is kept at a specific value or its approximate value. This also makes it possible to correctly determine the presence or absence of an object under a light source different from that of the reference image.

As described above with reference to FIGS. 1 to 10, in the method and the apparatus for determining the presence / absence of a specific object in a color image according to the present embodiment, a pair is formed by a color group in which similar colors are put together. (Color group pair) Based on the color group pair, a frame including an object is detected from the color moving image. This makes it possible to stably detect a pair of color groups even when there is a difference in color temperature of a light source, etc., and greatly reduces detection omissions in determining whether an object is present in a frame. Can be done. In addition, for a pair of different color groups, a threshold for determining an effective color pair is set higher than a threshold for a pair of the same color group. This can reduce overdetection.
In addition, by taking into account the adjacent direction between the distribution areas of the color groups, overdetection can be further reduced.

The method and apparatus for determining the presence or absence of a specific object in a color moving image according to the present invention are not limited to the embodiment described with reference to FIGS. Can be variously changed. For example,
In extracting the above-described color group, only colors that actually exist in the partial area of the reference image are grouped. However, colors that are considered to be similar in various color spaces may be obtained by calculation and may be grouped. Next, the color moving image cut search and display system of the present invention will be described with reference to FIGS.

FIG. 11 is a block diagram showing a first embodiment of the configuration according to the present invention of the color moving picture cut search and display system of the present invention. In the figure, reference numeral 11 denotes a moving image reproducing apparatus such as an optical disk or a video deck, and 12 denotes a moving image reproducing apparatus 11 by the same processing as the apparatus 90 for judging the presence or absence of a specific target in a color moving image described with reference to FIG. The frame determination device 13 that determines whether or not a specific target is included in each frame of the continuous color moving image output from the Each of the cuts is divided into cuts, and a cut display screen 14 of a rectangular parallelepiped made up of each cut and a frame within the cut is displayed on a display 15.
The cut display control device 16 for displaying on the screen of the specific frame, based on the determination result by the frame determination device 12, a specific frame detection unit that detects a frame containing a specific object and a cut, 17 Detector 16
The specific frame image display unit generates and displays and outputs the cut display screen 14 and the target object screen 18 using the frame including the specific target detected in the step (a). 14a
Is a cut display screen displayed by the conventional color moving image cut search and display system, in which the image of the first frame of each cut is displayed. The time length of each cut is represented by the depth (thickness) of the cut display screen 14a.

The cut display control device 13 uses the result of the process of judging a specific object by the frame judging device 12 when displaying a continuous color moving image output from the moving image reproducing device 11 in cut units. . That is, the specific frame detection unit 16 detects whether or not each cut includes a frame (specific frame) in which the specific target object exists, based on the specification result of the frame determination device 12. Then, when generating a cut display screen for each cut, the specific frame image display unit 17 displays an image of the specific frame for a cut including the specific frame.

At this time, the specific frame is displayed at a position corresponding to the portion indicating the time length of the cut on the cut display screen. In the present embodiment, it is displayed at a depth m. For a cut that does not include a specific frame, the image of the first frame of the cut is displayed on the foreground, as in the related art. As described above, in the color moving image cut search and display system according to the present embodiment, the image of the specific frame including the specific target is preferentially displayed on the cut display screen. Can be easily searched for.

FIG. 12 is a block diagram showing a second embodiment of the configuration according to the present invention of the color moving image cut search and display system of the present invention. The second embodiment is different from the cut display control device 13 shown in FIG.
9, a cut display control device 13a, which displays a cut display screen 20 indicating the time length of a specific frame group including a specific target object continuously.
When displaying a continuous color moving image output from the moving image reproducing device 11 in units of cuts, the cut display control device 13a uses the specific frame detecting unit 16 to determine the specific target object by the frame determining device 12. Based on the result, it is detected whether or not each cut includes a frame in which a specific target object exists (specific frame).

When the specific frame image display unit 17 generates a cut display screen for each cut displaying the image of the specific frame, the specific frame group display unit 19
Thus, after the image of the specific frame on the cut display screen, with a depth corresponding to the time length of the specific frame group,
Color or highlight. As described above, in the color moving image cut search and display system according to the second embodiment, the section in which the frame group including the specific target in the cut exists is displayed. Thus, the user can easily confirm the amount of time of the frame including the specific target in the searched cut.

FIG. 13 is a block diagram showing a third embodiment of the structure according to the present invention of the color moving image cut search and display system of the present invention. In the third embodiment, a display unit 21 is provided instead of the specific frame image display unit 17 of the cut display control device 13 in FIG. The images 22 to 24 of the included frames are displayed in a reduced size, and cut display screens 25 to 29 are displayed in which the time lengths of the respective frame groups of the reduced images 22 to 24 of the frames are graphically displayed.

The cut display control device 13b, when displaying a continuous color moving image output from the moving image reproducing device 11 in units of cut, by the specific frame detecting section 16 and the specific object by the frame determining device 12 , It is detected whether or not each cut includes a frame (specific frame) in which a specific target object exists. Then, the specific frame detecting unit 1 is displayed on the display unit 21.
The images 22 to 24 of the plurality of specific frames detected in step 6 are displayed in a reduced size, and the cut display screen 2 of a rectangular parallelepiped is displayed.
5 to 29 are displayed. In the third embodiment, the display unit 21 highlights the time length of a specific frame group corresponding to each of the images 22 to 24 on the upper surface of the cut display screens 25 to 29 with the display bars 22a to 24a. .

As described above, in the color moving image cut search and display system according to the third embodiment, not only a single object but also a plurality of objects are displayed at the same time as to which position of which cut appears. . Then, draw a bar (straight line) in the depth (thickness) portion of each cut in a color and expression format unique to the object, and according to the position and length of the bar,
Expresses the section where each object appears. Thus, the user can grasp at a glance which object appears where and for how long. Further, the appearance relationship between the objects, for example, the frame portion where the objects shown in the images 22 to 24 simultaneously appear can be seen at a glance,
The user can easily search for a plurality of specific objects.

FIG. 14 is a block diagram showing a fourth embodiment of the configuration according to the present invention of the color moving image cut search and display system of the present invention. In the fourth embodiment, each of the reference screens 3 used for specifying a plurality of types of objects specified by the user is provided to the cut display control device 13 in FIGS.
1 to 33 and time chart screens 31a to 33a indicating time positions in continuous color moving images output from the moving image reproducing apparatus 11 of each specific frame extracted using the respective reference screens, The object screen display unit 34 displayed on the screen of the display 15 and the user's designation information using the mouse 30 or the like for the reference screens 31 to 33 and the time chart screens 31a to 33a displayed on the object screen display unit 34 are displayed. The specific frame detecting unit 16 and the designated frame detecting unit 36 that causes the specific frame image display unit 17 to highlight the corresponding cut display screen 35 are provided.

The operation of the fourth embodiment will be described with reference to FIGS. FIG. 15 is an explanatory diagram showing a first example of the screen content of the display in FIG. This figure shows a state in which no designation has been made by the user using the mouse 30 in FIG. On this screen, cut display screens 37 to 4 each including an image of the first frame of each cut of the color moving image and a depth (thickness) representing a time length.
1, and reference screens 31 to 33 and time chart screens 31a to 33a by the object screen display unit 34 in FIG. 14 are displayed. In this state, when the user specifies the reference screen 32 using the mouse 30 in FIG. 14, the screen changes to the next screen shown in FIG.

FIG. 16 is an explanatory diagram showing a second example of the screen content of the display in FIG. This figure shows an operation example when the user specifies the reference screen 32 using the mouse 30 in FIG. When detecting the click information from the mouse 30 of FIG. 14, that is, the search instruction information of the reference screen 32, the designated frame detection unit 36 of FIG. 14 activates the frame determination device 12 of FIG. The determination process of the frame to be performed is started. The specific frame detection unit 16 and the specific frame image display unit 17 in FIG. 14 switch the cut display screens 37 to 41 to a display based on the reference screen 32 based on the determination result from the frame determination device 12 in FIG. That is, each of the display screens 37, 40, and 41 in FIG. 15 is switched to the display screens 37a, 40a, and 41a, and the reference screen 3
A screen displaying the second specific object appears. Further, a screen of the reference screen 32 displaying a specific target is displayed at a position corresponding to each cut.

As described above with reference to FIGS. 1 to 16, according to the method and the apparatus for determining the presence / absence of a specific object in a color image and the color moving image search / display system using the apparatus according to the present invention, The user can reliably detect the frame including the specific target from the color moving image without omission. The method and apparatus for determining the presence / absence of a specific object in a color moving image and the color moving image search and display system using this apparatus according to the present invention are, for example, as described in FIGS. ~ FIG.
However, the present invention is not limited to the embodiment described with reference to FIG.

[0077]

According to the present invention, how to select a reference color pair,
Technology related to color moving image detection that can efficiently prevent omission of detection of frames containing objects from color moving images, which has been caused by changes in the color of the object due to changes in the color temperature of the light source, etc. And the performance of the color moving image cut search and display system can be supported in more detailed editing work on the color moving image.

[0078]

[Brief description of the drawings]

FIG. 1 is a flowchart showing a first embodiment of a method for determining the presence / absence of a specific target in a color image according to the present invention.

FIG. 2 is a flowchart showing a first embodiment of a method for determining the presence / absence of a specific target in a color image according to the present invention;

FIG. 3 is a block diagram showing a configuration of an apparatus used for carrying out the method for determining the presence / absence of a specific object in a color image according to the present invention in FIGS. 1 and 2;

FIG. 4 is a flowchart illustrating a second embodiment of the method for determining the presence / absence of a specific object in a color image according to the present invention.

FIG. 5 is a flowchart illustrating a third embodiment of the method for determining the presence or absence of a specific target in a color image according to the present invention.

FIG. 6 is a flowchart showing a third embodiment of the method for determining the presence / absence of a specific object in a color image according to the present invention.

FIG. 7 is an explanatory diagram showing a fourth embodiment of the method for determining the presence / absence of a specific target in a color image according to the present invention.

FIG. 8 is a flowchart showing a fifth embodiment of the method for determining the presence / absence of a specific object in a color image according to the present invention.

FIG. 9 is a block diagram illustrating an embodiment of a configuration according to the present invention of the apparatus for determining the presence / absence of a specific target in a color image according to the present invention.

10 is a flowchart illustrating a processing operation according to the present invention of the apparatus for determining the presence / absence of a specific object in a color image in FIG. 9;

FIG. 11 is a block diagram showing a first embodiment of the configuration according to the present invention of the color moving image cut search and display system of the present invention.

FIG. 12 is a block diagram showing a second embodiment of the configuration according to the present invention of the color moving image cut search and display system of the present invention.

FIG. 13 is a block diagram showing a third embodiment of the configuration according to the present invention of the color moving image cut search and display system of the present invention.

FIG. 14 is a block diagram showing a fourth embodiment of the configuration according to the present invention of the color moving image cut search and display system of the present invention.

FIG. 15 is an explanatory diagram showing a first example of screen contents of the display in FIG. 14;

FIG. 16 is an explanatory diagram showing a second example of the screen content of the display in FIG. 14;

FIG. 17 is a flowchart illustrating an example of a conventional method for selecting feature information of a color moving image object.

FIG. 18 is a flowchart showing a conventional color moving image object search method.

FIG. 19 is an explanatory diagram showing a search example using the algorithm shown in FIGS.

FIG. 20 is an explanatory diagram illustrating an example of a reference color pair extracted from a reference image.

FIG. 21 is a block diagram showing a configuration of a conventional color moving image cut search and display system.

[Explanation of symbols]

 Reference Signs List 1 computer 2 display 3 moving image reproducing device 4 A / D converter 5 control line 6 external storage device 7 input device 8a-8e interface 9 CPU 10 memory 11 moving image reproducing device 12 frame judging device 13, 13a, 13b cut display control Apparatus 14, 14a Cut display screen 15 Display 16 Specific frame detection unit 17 Specific frame image display unit 18 Target screen 19 Specific frame group display unit 20 Cut display screen 21 Display unit 22 to 24 frame images 22a to 24a Each display bar 25 To 29 cut display screen 30 mouse 31 to 33 reference screen 31a to 33a time chart screen 34 target screen display section 35 cut display screen 36 designated frame detection section 37 to 41 cut display screen 37a, 40a, 41a cut display screen 71 Color group pair 72 Distribution map of primary color group 73 Distribution map of secondary color group 74 Distribution map of color group pair 75, 76 Direction mask 77 Distribution map of primary color group with noise removed 78 Distribution of secondary color group with noise removed Map 90 Specific object presence / absence determination device in color image 91 Main control unit 92 Image input / output unit 93 Color correction unit 94 Object designation unit 95 Color group extraction unit 96 Unique color selection unit 97 Color group pair registration unit 98 Presence / absence determination unit 99 Judgment method storage unit 191 Frame including object 192 Frame including search object 193 Frame not including search object 194, 195 Distribution map 291 Reference image 292, 293 RGB color space coordinates 294 Color difference 2101 Video playback device 2102 Cut detection device 2103 Cut display control device 2104 Display 2105 Cut display screen

────────────────────────────────────────────────── ─── Continuation of the front page (56) References Nagasaka et al., Automatic indexing method and object search method for color video images, Transactions of the Information Processing Society of Japan, Vol. 33, No. 4 (April 1992), pp. 543-550 Outside Nagasaka, 2 Techniques for Linking Video Information, Technical Report of the Institute of Television Engineers of Japan, Vol. 16, No. 10 (January 1992), p.p. 25-30 Ueda et al., Proposal of an interactive video editing system using recognition technology, IEICE Transactions D-II, Vol. J-75-D-II, No. 2 (February 1992), p.p. 216-225 (58) Fields surveyed (Int.Cl. ⁷ , DB name) G06T ^7/ 00-7/60 H04N 9/64 H04N 9/79 G06F 17/30 JICST file (JOIS)

Claims (3)

(57) [Claims] 1. A means for creating a group (color group) in which respective image areas are grouped for each similar color for a plurality of areas included in an object on a reference screen, and corresponding different color groups. Means for creating a color group pair forming a pair by combining based on the adjacency of the partial areas; and means for detecting, from a moving image, a frame (specific frame) in which the color group pair exists at a predetermined value or more. A moving image search device, characterized in that: 2. The moving image search apparatus according to claim 1, wherein the cut of the moving image is displayed as a cube whose depth is the time length, and the time of the specific frame group which is temporally continuous is further reduced. A moving image search device characterized by displaying a colored position corresponding to a length. 3. The search system of the moving image according to claim 2, the temporally first of the specific frame included in the cut, that arranging and displaying the temporally corresponding depth position of the cube Search device of a moving image and said.