JP2001167110A - Picture retrieving method and its device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To distinguish and display the faces of persons appearing in video by detecting the faces in particular from the video and identifying the detected face in addition. SOLUTION: The device provided with a means for detecting a face from the video and a means for identifying the detected face detects a frame including the face from the video, extracts a face picture from the frame, and groups the faces of the same appearing person from all the extracted face pictures to extract the representative face picture of each appearing person to identify the face of the appearing person in the video. Thus, the face of the person appearing in the video can be distinguished and displayed.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and an apparatus for searching for an image in a moving image in which a specific object is shown in a video editing apparatus or an image search apparatus.

[0002]

2. Description of the Related Art As a conventional image retrieval apparatus, Japanese Patent Application Laid-Open No.
What is described in 3179 gazette is known. The above application is an image search device that detects a frame in which a specific object (a search target) appears from a color moving image. FIG. 1 is a block diagram showing the configuration. In FIG. 1, reference numeral 1 denotes a computer as a central processing unit for determining a specific target object;
Is a display such as a CRT for displaying an output screen of the computer 1, 3 is a moving image reproducing device such as an optical disk, 4 is an A / D converter for converting an analog signal into a digital signal,
Reference numeral 5 denotes a control line for connecting control signals between the moving picture reproducing device 3 and the computer 1, 6 denotes an external storage device such as a hard disk, 7 denotes an input device such as a mouse, and 8a to 8e
An interface for connecting the computer 1 to peripheral devices; and 9, a CP for performing arithmetic processing in the computer
U and 10 are memories directly accessed by the CPU 9.

FIG. 12 is a flowchart showing the operation of the above-described conventional image search apparatus. Hereinafter, the operation will be described with reference to the flowchart of FIG. First, when a searcher selects an image to be searched and inputs the image to the apparatus from the input device 7 or the like, one image including the target is designated (step 20).
01). The apparatus divides an input image into regions of similar colors (step 2002). A color histogram is generated for each of the divided partial regions (step 2004), and N colors having a higher frequency are selected in order, and the color histogram is determined as shown in Table 1.
Is created (step 2005).
Here, CG (r) indicates a list of the r-th partial area. Further, for each partial area, a list CG (r),
A list R shown in Table 2 shows a correspondence relationship between the partial region and a list CG (r) of adjacent partial regions in a positional relationship in the image.
A CGP is created (Step 2007).

Next, a search operation will be described. FIG.
Is a flowchart showing a search operation. First, the frame to be determined is divided into a plurality of cells C (x, y) (20
1) A color histogram is generated for each cell (step 203), and a color having a histogram frequency greater than a set threshold is registered in the list CC (x, y) (step 20).
5, 206). Next, for each cell C (x, y) (step 208), a color group CG forming a color pair in RCGP
One of the colors belonging to one of the color groups is included in the list CC.
(X, y) (step 209), and one of the colors belonging to the other color group is the cell C (x, y) itself,
Alternatively, if the cell is included in the list CC of any one of the eight adjacent cells, the cell C (x, y) is extracted as a valid cell (step 210), and the total number of valid cells in one frame is calculated. It is determined (step 211).
If the number of valid cells in one frame is greater than or equal to the threshold
(Step 212), the color group pair CG is counted as an effective color group pair (Step 213). If the total number of counted effective color group pairs is equal to or larger than the set threshold value, it is determined that the valid color group pair exists in the object (search target) frame (step 215). If the value is equal to or smaller than the threshold value, it is determined that the target object does not exist (step 216).

In this image search apparatus, there is a demand for a technique for reducing the burden on a video editor or an image searcher and efficiently searching for an image to be searched. However, in the above-described conventional technique, since the search is performed based on the information on the color grouping of the image to be searched and the positional relationship between the color groups, for example,
When the clothes are the same, there is a problem that the search target is determined to be a person regardless of whether there is actually a different person from the person to be searched, and this leads to an increase in erroneous detection.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned conventional problems, and solves the above-mentioned problems by performing a detection specialized for a face from an image and identifying the detected face. .

[0007]

SUMMARY OF THE INVENTION In order to solve this problem, the present invention provides means for detecting a face from an image and means for identifying the detected face so that false detection is reduced and the same clothes are provided. However, the gist of the present invention is to be able to identify even a different person.

[0008] As an invention having such an aspect, the invention according to claim 1 of the present invention, as an image search method, detects a frame including a face from a video, extracts a face image from the frame, Image retrieval characterized by grouping faces of the same character from all the extracted face images and extracting a representative face image for each character, and identifying the faces of the characters in the video This is an apparatus, and has an effect that a face of a person appearing in a video can be distinguished and displayed.

According to a second aspect of the present invention, in the image retrieving method according to the first aspect, the detection of the frame in which the face is captured from the video is performed by detecting a break in the scene from the video. The present embodiment detects a frame including a face as an image, and has an effect that an image in which a face appears as a representative image of each scene can be selected.

According to a third aspect of the present invention, in the image retrieval method according to the first or second aspect, the detection of the frame in which the face is captured from the video is performed by detecting the number of faces (number of people) and the number of faces. At least one of size, face orientation, gender, facial expression, (age estimation, face race determination), presence or absence of glasses, or presence or absence of a beard is detected as a predetermined condition. By extracting a face image that satisfies a predetermined condition, an image including a face image that matches a condition specified by a searcher can be extracted.

According to a fourth aspect of the present invention, as an image search method, a similarity between a face of a character in a video and a face image designated by a searcher is calculated, and a face having a predetermined similarity or more is calculated. Is detected so that it is possible to know where in the video the face designated by the searcher is recorded.

According to a fifth aspect of the present invention, in the image retrieval method according to the fourth aspect, a scene break is detected from a video, and a frame including a face is selected as a representative image of each scene. The image retrieval apparatus is characterized in that an image having a face can be selected as a representative image of each scene.

According to a sixth aspect of the present invention, as an image search method, a frame in which a person is captured from a video is detected, and a frame in which the posture or clothes of the person satisfying a predetermined condition is captured is extracted. This has the effect that an image containing the attributes of a specific person can be extracted.

According to a seventh aspect of the present invention, in the image search method according to the sixth aspect, the detection of a frame in which a person is present in a video is performed by detecting a break in a scene from the video and representing a representative of each scene. The present embodiment detects a frame including a face as an image, and has an effect that an image in which a face appears as a representative image of each scene can be selected.

According to an eighth aspect of the present invention, in the image search method according to the fourth aspect, the face image specified by the searcher is at least one or more face images from a face database registered in advance by a character list. An image is designated, and has an effect that a person can be easily specified.

According to a ninth aspect of the present invention, in the image retrieval method according to the eighth aspect, the character list is prepared in advance by a searcher or generated from a program table. By creating in advance, the face image can be easily specified by the searcher, and the character list can be easily and easily generated by generating from the program table.

According to a tenth aspect of the present invention, in the image retrieving method according to the fourth aspect, a face image of a character in a video is designated and registered in advance as a face image specified by a searcher. This has the effect that the face image can be easily designated by the searcher by designating and registering the face image in advance.

According to an eleventh aspect of the present invention, there is provided an image search device, comprising: a face detection unit for detecting a frame in which a face is present in a video and extracting at least one face image from the frame; A character identification unit that groups the faces of the same character from all the extracted face images and extracts a representative face image for each character, and identifies the faces of the characters appearing in the video. It has the effect that it can be displayed separately.

According to a twelfth aspect of the present invention, as the image search device, a scene change detecting unit for detecting a scene change by monitoring a frame image in a time series with a video signal as an input; A face detection unit that detects a frame including a face from a representative image of each scene from a change detection unit, and extracts at least one or more face images from the frame; and a face detection unit that extracts the same character from all the extracted face images. It is equipped with a character identification unit that groups faces and extracts a representative face image for each character, and can be displayed on a display or stored as a frame representative image showing the face for each scene It has the action of:

According to a thirteenth aspect of the present invention, as an image search device, a character designating section for designating at least one or more face images from a face database registered in advance by a searcher with a character list; A video input unit that receives a signal as an input and outputs the frame unit, a face detection unit that detects a face area from a frame image output from the video input unit, a face image output from the character designating unit, and the face A character identification unit that determines whether or not the image corresponds to a character from the face image detected in the face area from the detection unit, and an image output unit that displays and records the identification result determined by the appearance identification unit. This has the effect that characters in a certain video program can be displayed on a display or stored for each character.

According to a fourteenth aspect of the present invention, as an image search device, a face image reading unit that reads an image to be searched, and a video input unit that receives a video signal as an input and outputs the frame signal in frame units A face detection unit that detects a face area from a frame image output from the video input unit;
A matching unit that determines whether the search target image and the face image detected in the face area output from the face detection unit are a search target, and an image output unit that displays and stores the matching result This has the effect that a frame including a face image matching or similar to the face image specified by the searcher can be displayed on the display or stored.

[0022]

(Embodiment 1) Hereinafter, an embodiment of the present invention will be described with reference to FIGS. FIG. 1 to FIG. 3 are diagrams illustrating an image search device according to a first embodiment of the present invention. In the present embodiment, detection of a face has been described. Specifically, it is determined whether or not a face exists in the read image. If a face area exists, the position and number of faces in the image, the face size,
The direction of the face (front, upward, downward, left, right) is determined, and the result is displayed on a monitor or the like.

FIG. 1 is an overall configuration diagram of the face detecting apparatus. In FIG. 1, reference numeral 1 denotes a computer for detecting a target image, and 2 denotes a C for displaying a detection result of the computer 1.
RT, etc., 3 is a moving image reproducing apparatus such as an optical disk, and 4 is an A / A for converting an analog signal into a digital signal.
Reference numeral 5 denotes a D converter, 5 denotes a control line for connecting a control signal between the moving picture reproducing device 3 and the computer 1, 6 denotes an external storage device such as a hard disk, 7 denotes an input device such as a mouse and a keyboard, and 8a to 8a to 8d. Reference numeral 8e denotes an interface for connecting the computer 1 to a peripheral device, 9 denotes a CPU of the computer, and 10 denotes a memory directly accessed from the CPU 9.

The video signal output from the moving picture reproducing device 3 is sequentially converted into a digital image by the A / D converter 4 and sent to the computer 1. In the computer 1, the digital image enters the memory 10 via the interface 8c, and is processed by the CPU 9 according to a program stored in the memory 10.

FIG. 2 is a flowchart showing the flow of the detection method. First, when the frame image r is read into the computer 1 (step 101), a face area is detected according to a function f described in advance by a program (step 102). The output P of the function f is a matrix in which all face coordinates detected on the frame are recorded. For example, the vector in the i-th column of the matrix P stores the face detected i-th in the frame r, and the (i, 1) component of the matrix P is, for example, when the face region is cut out in a rectangular region. Then, the upper left x coordinate of the face area, the (i, 2) component is the upper left y coordinate of the face area, the (i, 3) component is the lower right x coordinate of the face area,
The (i, 4) component stores the y coordinate at the lower right of the face area.
If no face area is found, the function f
Returns, for example, -1. The output of matrix P is-
If it is not 1, the detection result is output to the display 2 (step 104).

On the display, only the detected face area may be displayed, or a specific mark may be added to the face area to display the entire frame. Note that since the column size of the matrix P indicates the number of detected persons, the detected number of persons can also be displayed on the display.

For example, in the case of a color image, the specific function f can be realized by skin color detection. The (R, G, B) value of each pixel of the input image is stored in a color space (for example, Y
uv space), and only pixels included in a predefined skin color space are selected. Further, of the image regions formed by only the corresponding pixels, only those regions having an area equal to or larger than a predetermined threshold and having the same or similar shape as a predetermined shape (for example, an ellipse) By selecting, face detection is realized. Also, as a method that can handle both monochrome images and color images,
A method using template matching is also conceivable. One or more face images are stored in advance as standard patterns. For the frame image, a partial area on the frame is sequentially cut out while moving the position of the cut-out window, and the similarity between the cut-out image and the above-described template is calculated. For example, if the correlation value between the cut-out image and the template image is equal to or greater than a set threshold, the cut-out area is determined to be a face image.

By preparing templates of different sizes, it is possible to detect face regions having different face sizes. Also, by preparing templates having different face directions, face areas having different face directions can be detected. For example, four types of face image templates, upward, downward, leftward, and rightward, are prepared, and the similarity between each of the cut-out images and the four types of templates is obtained. If it is equal to or greater than a predetermined threshold, the direction of the face can be specified from the corresponding template.

As another method of realizing the function f, a method may be considered in which a discriminant function for determining whether the cut-out area is a face or a non-face is prepared in advance. For example, a learning function using a neural network is used to create a face / non-face discrimination function. The neural network can be trained so as to identify a category formed by applying different labels to images having different face directions in advance, and the face direction can be detected.

FIG. 3 is a block diagram showing the face detection processing. In FIG. 3, reference numeral 11 denotes an image input unit, 12 denotes a face detection unit, and 13 denotes an image output unit.
The image input unit receives a video signal T output from the A / D converter 4 as an input and converts an image for one frame into a frame signal r.
Output as The face detection unit 12 receives the frame signal r output from the image input unit as an input, and outputs a face detection signal P that outputs information on the coordinates of the face position and the face direction existing in one frame, and a frame signal r. Output. Image output unit 1
3 is a detection result signal S based on the face detection signal P output from the face detection unit 12 and the frame signal r.
Output as The detection result signal S is displayed on the display 2 via the interface 8b in the computer. In the frame of FIG. 2, for example, a face image cut out for each frame is displayed. As shown in FIG. 2, various face sizes and orientations can be handled. For example, the direction of the face may be displayed on the display with an arrow or the like. Note that the detection signal S may be an image of the entire frame in which a specific mark is attached to the face area. In addition, beards and glasses can be detected by a technique similar to face detection.

(Embodiment 2) FIG. 4 is a flowchart for explaining the operation of an image retrieval apparatus according to a second embodiment of the present invention. The image search device according to this embodiment has the same configuration as the device configuration according to the first embodiment. In the second embodiment, when a representative image in a certain scene is selected by combining the video scene change function and the face detection function described in the first embodiment, an image in which a human face is shown is selected. Is a representative image. Figure 4 below
This will be described with reference to the flowchart shown in FIG.

In FIG. 4, first, a flag Flg as to whether or not the representative image of the current scene has already been stored is initialized (step 107). Next, the r-th frame r of the video is read, and it is determined whether or not the scene has changed (step 109). If it is determined that the scene has changed, the representative image RT of the previous scene stored in advance is displayed on the display (step 110), and the flag is set.
Flg is reset to 0 (step 111). However,
Immediately after the start of video input, there is no representative image RT of the previous scene, so nothing is displayed on the display.

Next, the flag Flg is evaluated (step 11).
2) If Flg is 0, the currently read frame r
Is recorded as the representative image RT (step 113), and the flag Flg is set to 1 (step 114). This processing takes into account a case where a human face is never included in the scene, and stores a frame immediately after a scene change is determined as a representative image in advance. Note that a frame after a predetermined frame has elapsed after the scene change is determined may be stored in advance as a representative image.

When the flag is set to 1, since the previous frame has already recorded the representative image, the face detection is performed as it is. Step 11
Even after the first representative image is captured in steps 3 and 114, face detection is performed by the method described in the first embodiment (step 115).

As a result of the detection, when the output P of the function f is -1 (step 116), that is, when no face is detected,
Read the next frame. If P is not -1, that is, if a face is detected in the currently read frame r, the representative image is updated (step 117).

According to the above-described processing procedure, when a face area exists in a certain scene, a frame in which the face is shown can be set as the representative image. The method described in the first embodiment can also determine the direction, size, and number of faces. For example, a frame of a frontal face is used as a representative image, or a frame with a larger face size is used as a representative image. Or, a frame having a larger number of faces can be selected as the representative image.

FIG. 5 is a block diagram showing the above processing procedure. In FIG. 5, reference numeral 11 denotes an image input unit, which is the same as that described in the first embodiment. 14
Is a scene change detection unit, and 12 is a face detection unit, which is the same as that described in the first embodiment. Reference numeral 13 denotes an image output unit, which is the same as that described in the first embodiment.

The image input unit 11 receives a video signal T output from the A / D converter 4 and outputs an image for one frame as a frame signal r. The scene change detection unit 14 determines a scene change from the discontinuity of the video, and outputs a scene switching signal C and a frame signal r. The face detection unit 12 receives the scene switching signal C and the frame signal r output from the scene change detection unit 14 as inputs, and receives the coordinates of the position of the face in one frame,
A face detection signal P for outputting information of the face direction and a frame signal r are output. The image output unit 13 outputs a frame including a face as a detection result signal S from the face detection signal P and the frame signal r output from the face detection unit 12.
The detection result signal S is transmitted to the interface 8 in the computer.
b is displayed on the display 2. The display 2 displays, for example, a representative image for each scene. When a face image is included in the scene, an image including a face can be used as the representative image.

The realization of the scene change detection unit is described in, for example, the document “Video index creation and editing technology”, (Mats
ushita Technical Journal Vol.44 No.5), Shin Yamada et al.

(Embodiment 3) FIG. 6 is a flowchart for explaining the operation of an image retrieval apparatus according to a third embodiment of the present invention. The image search device according to this embodiment has the same configuration as the device configuration according to the first embodiment. In the present embodiment, a face is detected from a read frame, and gender identification is performed from the detected face.

This will be described with reference to the flowchart of FIG.
First, when the frame image r is read into the computer 1 (step 119), a face area is detected in accordance with f according to a function described in advance by a program (step 120). The function f is the same as that described in the first embodiment. If the output P of the function f is not -1, that is, if a face area is detected, it is determined whether the face in the detection area is a man or a woman (step 122). The function g is a gender discrimination function. The function g uses the coordinates P of the face area on the frame and the frame r as arguments to discriminate between genders and stores the discrimination result in the output Q. The output Q is, for example, a matrix, and the vector in the i-th column of the matrix Q is recorded for the i-th detected face. The (i, 1) component of the matrix Q is i
The x coordinate at the upper left of the face area detected the third time, the (i, 2) component is the y coordinate, the (i, 3) component is the x coordinate at the lower right of the face, (i,
The 4) component stores the y coordinate at the lower right of the face, and the (i, 5) component stores +1 for males and -1 for females.

As a method of realizing the function g, it is possible to collect a large number of face images whose genders are known in advance, and to implement the function g by a statistical method such as discriminant analysis based on the collected face images. Further, for example, a discriminant function can be realized by learning a neural network. Finally, display 2
Is output (step 123).

Therefore, in this embodiment, after the face is detected, the detected face can be displayed as a man or a woman. Further, if the video editor and the video searcher have input in advance via the input device 7 that the user wants to search only a specific gender, it is possible to detect a frame including a face image of only the specific gender.

FIG. 7 is a block diagram showing the above processing. In FIG. 7, 11 is an image input unit, 12 is a face detection unit, 18 is a gender identification unit,
Reference numeral 13 denotes an image output unit. The image input unit 11 receives a video signal T output from the A / D converter 4 and outputs an image for one frame as a frame signal r. The face detection unit 12 receives, as an input, the frame signal r output from the image input unit, the coordinates of the position of the face existing in one frame,
A face detection signal P for outputting information such as face orientation and size,
It outputs a frame signal r. The gender discriminating unit receives the face detection signal P output from the face detecting unit 12 and the frame signal r, and performs gender discrimination based on the image of the face area recorded in P. The discrimination result and the face detection signal The result of P is output together with the gender identification signal Q. It also outputs a frame signal r. The image output unit 13 cuts out only the face region from the gender discrimination signal Q and the frame signal r output from the gender discrimination unit 15, and detects an image in which male and female identification marks are added to each face image. Output as a signal S. The detection result signal S is displayed on the display 2 via the interface 8b in the computer. For example, the face image cut out for each frame and the gender mark of each face image are displayed on the display of FIG.

The display method on the display is not limited to the above-described method, and the entire frame may be displayed on the display after a specific mark is attached to the face area. In addition, by inserting the scene change detection unit 14 described in the second embodiment immediately after the image input unit 11 in FIG. 7, the representative image of each scene can be a face in which the gender of the characteristic appears. is there.

Further, as shown in the block diagram of FIG. 8, by adding, for example, an age discriminating unit 16, a facial expression discriminating unit 17, and a race discriminating unit 18, more detailed search and editing can be performed. The age discrimination unit 16 estimates the age from the detected face from the face detection signal P and the frame signal r output from the face detection unit 12, and outputs an age discrimination signal y. The age is output, for example, the age of the twenties or the like. The age discrimination 16 can be realized by the same method as that for realizing the gender discriminator.

A large number of face images whose dates are known in advance can be collected, and based on the collected face images, a statistical method such as discriminant analysis can be used. Further, for example, a discriminant function can be realized by learning a neural network. The expression identifying section 17 receives the face detection signal P and the frame signal r output from the face detecting section 12 as inputs, estimates an expression from the detected face, and outputs an expression identifying signal H. For the identification of the facial expression, for example, a label attached to each facial expression such as “laughing”, “crying”, “angry” is output. As a method of realizing the facial expression identification unit 17, it can be realized by the same method as the method of implementing the gender identification unit.

A large number of face images whose expressions are known in advance can be collected, and a statistical method such as discriminant analysis can be implemented based on the collected face images. Further, for example, a discriminant function can be realized by learning a neural network.

The race identification unit 18 receives the face detection signal P output from the face detection unit 12 and the frame signal r as inputs,
The race is estimated from the detected face, and a race identification signal L is output. For the identification of the race, for example, a label attached to the race such as “yellow race”, “white”, “black”, or the like is output. As a method of realizing the facial expression identification unit 17, it can be realized by the same method as the method of implementing the gender identification unit.

A large number of face images whose races are known in advance can be collected, and the collected face images can be realized by a statistical method such as discriminant analysis. Further, for example, a discriminant function can be realized by learning a neural network.

The image output unit 13 includes a gender discrimination signal Q and a frame signal r output from the gender discrimination unit 15, an age discrimination signal y output from the age discrimination unit, and an expression discrimination signal H output from the expression discrimination unit 17. And the race identification signal L output from the race identification unit as input, attaches gender, age, facial expression, and race labels to each face image, and via the interface I / F 8b in the computer. It is displayed on the display 2.

By other similar methods, it is possible to identify a hairstyle or a hat. In addition, by cutting out the image around the cut-out face image, other information about the detected person can be obtained. For example, the color of the clothes worn by the person, the color of the tie, the type of clothes, and the like can be identified from the image located below the face image. Furthermore, the posture of the person can be detected from a technique such as background subtraction, and the movement can be estimated by capturing the change in the posture of the person from a plurality of continuous frames.

Also, by inserting the scene change detection unit 14 described in the second embodiment immediately after the image input unit 11 in FIG. 8, the representative images of each scene can be identified by specific gender, specific age, and It is also possible to use a face with a specific expression and a specific race.

The application of the embodiment described here is assumed to be a video editing and retrieval device. However, the present invention can be used not only for limited applications but also for purchaser analysis at retail stores. For example, a camera is installed at the cash register of a retail store such as a supermarket, and the face is detected from the images of the purchasers lined up at the cash register and gender identification and age identification are performed. Information can be obtained. It is a technology that can contribute to sales improvement by changing purchased goods based on such information.

(Embodiment 4) FIG. 9 is a flowchart for explaining the operation of an image retrieval apparatus according to a fourth embodiment of the present invention. Also in this embodiment, the image search device has the same configuration as the device configuration according to the first embodiment. In the present embodiment, identification of characters in a certain program has been described. Here, the character identification means a function of distinguishing and displaying characters appearing in a certain program.

First, the following description is made on the assumption that a list of persons appearing in a certain video program is known and a face image of the persons can be extracted from an image database or the like. The operation will be described with reference to the flowchart of the process shown in FIG. First, the program character list is read (step 126). In this list, an ID number is given to each character, and the ID number and the storage address of the face image database are recorded. Next, a face image is read from a face database registered for each ID number. It is assumed that, for example, images having different orientations, sizes, and expressions are registered as the face images.

As described above, feature values are extracted from a plurality of face images for each character (step 128). For example, there is KL expansion as an implementation method of the feature amount extraction method.
That is, KL expansion is performed for each face image for each character.

The above is the preparation for character identification.

Next, the target program video is read (step 129), and a face image is detected for each frame by the method described in the first embodiment (step 13).
0). The detected face image is subjected to face identification and associated with the characters (step 131). As the face identification method, for example, a subspace method or the like can be used. Finally, the character identification result is displayed on the display (step 132).

FIG. 10 is a block diagram showing the above processing. In FIG. 10, reference numeral 19 denotes a character list, reference numeral 20 denotes a face database storing face images of the characters, reference numeral 21 denotes a face image capturing unit, and reference numeral 22 denotes a face image capturing unit.
Is a feature extraction unit that extracts features from a face image, 23 is a face identification unit that identifies the face of a character, and 11 is an image input that receives a video signal as input and outputs a frame signal r for each frame. 12 is a face detection unit, and 1 is a face detection unit.
Reference numeral 3 denotes an image output unit for outputting the identification result, 8b denotes an interface in the computer, and 2 denotes a display for displaying the identification result. Hereinafter, the operation will be described for each block.

The image capturing section 21 captures a face image of a character stored in the face database from the input character list, and outputs the corresponding face image as a face database signal ft. The feature extracting unit 22 calculates the face database signal ft output from the face image capturing unit,
Extract feature values. For example, an eigenvector obtained by performing KL expansion for each character is output as a feature amount signal K.

On the other hand, the image input section 11 takes in the video signal T input via the A / D converter 4 and outputs it as a frame signal r frame by frame. The operation of the face detection unit 12 is the same as that described in the first embodiment. A face area in a frame is detected from the frame signal r output from the image input unit 11 by the method described in the first embodiment, and a face detection signal P and a frame signal r are output. The character identification unit 23 receives the above-described feature amount signal K, face detection signal P, and frame signal r as inputs, and first extracts a face area from the face detection signal P and the frame signal r, and converts it into a vector. For this vector, for example, a subspace method is performed,
Find out which character the cut-out face image is most similar to, and find the ID number of the character with the highest similarity,
The position of the corresponding face image on the frame is output as the identification result signal Res. Also, the frame signal r is output at the same time.

The image output unit 13 outputs an identification result signal S from the identification result signal Res and the frame signal r output from the character identification unit 23 so as to display a face image for each character on the display 2. I do. The manner of displaying on the display is not limited to this, and the entire frame may be displayed on the display, and a different mark may be displayed for each character.

Further, by inserting the scene change detecting section 14 described in the second embodiment immediately after the image input section 11, it is possible to output the detection result of the characters for each scene.

In this embodiment, the case where the list of the characters appearing in the video program is known in advance is described. If the characters are not known, first the face is detected in the entire target video and then the unsupervised This can be realized by performing clustering. If the character is not registered in the face database and you want to detect where the character's face in the initial scene appears in the subsequent video, the searcher or This can be realized by cutting out characters from the scene and inputting them to the feature extraction unit 22.

(Embodiment 5) FIG. 11 shows a fifth embodiment of the present invention.
It is a block diagram which shows the structure of the image search device which concerns on Embodiment. In the present embodiment, detection of where a specific person's face exists in a target video is described.

In FIG. 11, reference numeral 25 denotes a face image to be searched, reference numeral 21 denotes a face image capturing unit for capturing the face image to be searched, reference numeral 22 denotes a feature extracting unit for extracting features from the face image, and reference numeral 11 denotes a feature extracting unit. , An image input unit, 12 is a face detection unit, 26 is a video signal feature extraction unit that extracts a feature amount from the face image detected by the face detection unit 12, and 2
4 is a collating unit, 13 is an image output unit that outputs the identification result, 8b is an interface in the computer, and 2 is a display that displays the identification result.

Hereinafter, the operation will be described for each block.
The image capturing unit 21 captures the face image 25 to be searched and executes preprocessing. For example, histogram smoothing or the like is performed and output as a preprocessed search image signal TTa.
The feature extracting unit 22 extracts a predetermined feature amount from the preprocessed search image signal TTa output from the face image capturing unit 21.

On the other hand, the image input section 11 takes in the video signal T input via the A / D converter 4 and outputs the frame signal r frame by frame. The operation of the face detection unit 12 is to detect a face area in a frame from the frame signal r output from the image input unit 11 and output a face detection signal P and a frame signal r. After the face image is cut out and pre-processed by the quantity extraction unit 26, the feature quantity signal k and the frame signal r are output. The matching unit 24 checks the feature amount signals output from the feature extracting unit 22 and the video signal feature amount extracting unit. If the feature amount signal is equal to or greater than a predetermined similarity, the matching unit 24 determines that the face is a search target face, and Output to the image output unit 13. The image output unit 13 outputs the input frame on the display 2 together with the frame number.

According to the above embodiment, for example, a search for famous actors, politicians, etc. can be performed quickly, and the burden on video editors and searchers can be reduced.

[0071]

As described above, according to the present invention, a face image is detected from a video database or the like stored in a large amount, and the detected face is identified. However, there is an advantageous effect that an image including a face image to be searched with high accuracy can be extracted without erroneous detection. Furthermore, as an effect of the present invention, there is an effect that it is possible to respond to a more detailed search request of a searcher such as gender identification.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating an overall configuration of an image search device according to a first embodiment of the present invention.

FIG. 2 is a flowchart for explaining the operation of the first embodiment;

FIG. 3 is a block diagram for explaining the first embodiment;

FIG. 4 is a flowchart for explaining the operation of the image search device according to the first embodiment of the present invention;

FIG. 5 is a block diagram for explaining the second embodiment;

FIG. 6 is a flowchart for explaining the operation of the image search device according to the third embodiment of the present invention.

FIG. 7 is a block diagram for explaining Embodiment 3 of the present invention;

FIG. 8 is a block diagram for explaining the third embodiment.

FIG. 9 is a flowchart for explaining the operation of the image search device according to the fourth embodiment of the present invention.

FIG. 10 is a block diagram for explaining the fourth embodiment;

FIG. 11 is a block diagram for explaining the operation of an image search device according to a fifth embodiment of the present invention.

FIG. 12 is a flowchart for explaining a conventional technique.

FIG. 13 is a flowchart for explaining a conventional technique.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 computer 2 display 3 moving image reproducing device 4 A / D converter 5 control line 6 external storage device 7 input device 8 a to 8 e interface 9 CPU 10 memory 11 image input unit 12 face detection unit 13 image output unit 14 scene change detection unit 15 Gender identification unit 16 Age identification unit 17 Expression identification unit 18 Race identification unit 19 Character list 20 Face database 21 Face image capture unit 22 Feature extraction unit 23 Character identification unit 24 Collation unit 25 Search target face image 26 For video signal Feature extraction unit

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Masanori Soma 3-10-1, Higashi-Mita, Tama-ku, Kawasaki-shi, Kanagawa Prefecture Inside Matsushita Giken Co., Ltd. (72) Inventor Kenji Nagao 3-chome, Higashi-Mita, Tama-ku, Kawasaki-shi, Kanagawa No. 10 No. 1 Matsushita Giken Co., Ltd. F term (reference) 5B075 ND12 NK07 PQ02 PR06 QM08 5C052 AA01 AC08 DD04 DD10 EE02 EE03

Claims (14)

[Claims] 1. A frame in which a face is shown in a video is detected, a face image is extracted from the frame, faces of the same character are grouped from all the extracted face images, and a representative face image is classified for each character. An image search method characterized by extracting a character string. 2. The method according to claim 1, wherein detecting a frame in which a face is included in the video includes detecting a scene break from the video and detecting a frame including the face as a representative image of each scene. Image search method described. 3. A method for detecting a frame in which a face is captured from a video includes detecting the number (number of faces) of the captured face, face size, face orientation, gender, facial expression, (age estimation, face race determination. 3. The method according to claim 1, wherein at least one of the presence or absence of glasses and the presence or absence of a beard is detected as a predetermined condition. 4. An image search method comprising calculating a similarity between a face of a character in a video and a face image designated by a searcher, and detecting a frame in which a face having a predetermined similarity or more is captured. . 5. The image according to claim 4, wherein, for the face of the character in the video, a break in the scene is detected from the video, and a frame including the face is selected as a representative image of each scene. retrieval method. 6. An image search method comprising: detecting a frame in which a person is captured from a video; and extracting a frame in which a posture or clothing of the person that satisfies predetermined conditions is extracted. 7. The method according to claim 6, wherein detecting a frame in which a person is included in the video includes detecting a scene break from the video and detecting a frame including a face as a representative image of each scene. Image search method described. 8. A face image designated by a searcher, wherein at least one face image is designated from a face database registered in advance by a character list.
Image search method described. 9. The image search method according to claim 8, wherein the character list is created in advance by a searcher or generated from a program table. 10. The image search method according to claim 4, wherein a face image of a character in the video is specified and registered in advance as the face image specified by the searcher. 11. A face detection unit that detects a frame in which a face is present in a video and extracts at least one face image from the frame, and detects a face of the same character from all the extracted face images. An image search device comprising: a grouping unit; and a character identification unit that extracts a representative face image for each character. 12. A scene change detecting section for detecting a scene change by monitoring a frame image in time series with a video signal as an input, and a face is taken from a representative image of each scene from the scene change detecting section. A face detection unit that detects a frame and extracts at least one or more face images from the frame; and groups faces of the same character from all the extracted face images, and extracts a representative face image for each character An image search device comprising a character identification unit. 13. A character designating unit for designating at least one or more face images from a face database previously registered by a searcher in a character list, and a video input unit for receiving a video signal as input and outputting it in frame units. A face detection unit that detects a face area from a frame image output from a video input unit; and a character based on a face image output from the character specification unit and a face image detected in the face area from the face detection unit. An image search device comprising: a character identification unit that determines whether or not the above condition is satisfied; and an image output unit that displays and records the identification result determined by the appearance identification unit. 14. A face image reading unit for reading an image to be searched, a video input unit for receiving a video signal as an input of a video signal and outputting the frame unit, and a face area from a frame image output from the video input unit. A face detection unit for detecting,
A matching unit that determines whether or not the search target image and a face image detected in the face area output from the face detection unit are a search target; and an image output unit that displays and stores the matching result. Characteristic image retrieval device.