JP5538781B2 - Image search apparatus and image search method - Google Patents

Description

  The present invention relates to an image search device and an image search method.

  2. Description of the Related Art Conventionally, there has been known an image search apparatus that searches a moving image for a subject image desired by a user. This image retrieval device can retrieve images in units of frames, and the images retrieved in units of frames are displayed as they are. Therefore, according to this image search device, it is possible to grasp a desired person appearing in the moving image.

  2. Description of the Related Art In recent years, a technique for recording video signals from a plurality of frames for a certain period of time before and after the time point from which a subject can be detected has been disclosed (for example, see Patent Document 1).

JP 2001-285787 A

  However, the above-described technique does not accurately specify the video section in which the subject is shown. That is, even if a video section is searched by the above-described technique, there is a problem that many video sections that are not closely related to the subject are searched.

  The present invention has been made in view of such problems, and an object thereof is to search for a video section that is closely related to a subject.

Therefore, the image search apparatus according to the present invention includes a detecting unit that detects a plurality of subject patterns from a moving image having a plurality of frames, and a subject image in the moving image corresponding to each of the plurality of subject patterns. Generating means for generating a sequence collected for each series of frames, and based on the state of each of the subject images of a plurality of subject images constituting the sequence generated by the generating means, each subject image is subject similarity A determination unit that determines whether the determination is effective; a feature amount calculation unit that calculates a feature amount of the subject image determined to be effective by the determination unit among the plurality of subject images; and A storage unit that stores the feature amount calculated by the feature amount calculation unit in the storage unit, and a sequence given as a search condition. Calculated by the similarity calculation unit from among the sequences stored in the storage unit and the similarity calculation unit that calculates the similarity between the feature amount and the feature amount for each sequence stored in the storage unit An extraction means for extracting a sequence based on the similarity and an output means for outputting video section information corresponding to the sequence extracted by the extraction means as a search result .

  According to the present invention, it is possible to search for a video section that is closely related to the subject.

It is a figure which shows the structure of an image search device. It is a figure which shows an example of the flowchart which shows the process in an image search device. It is a figure which shows the example in which face detection is performed for every frame of a predetermined interval. It is a figure which shows the example which detects a face pattern from the whole image area. It is a figure which shows an example of the flowchart which concerns on the process which produces | generates a face sequence. It is a figure which shows an example of the flowchart which concerns on the process which extracts a feature. It is a figure which shows the progress situation at the time of performing feature extraction. It is a figure showing the image regarding determination of a representative face feature-value. It is a figure which shows the example of a display of the user interface for inputting a query. It is the figure which showed the outline | summary of the process in a similarity determination part. It is a figure which shows the example of the display which concerns on the searched face sequence. It is a figure which shows the example of the display which concerns on the searched face sequence.

  Hereinafter, embodiments according to the present invention will be described with reference to the drawings. For example, the image search apparatus according to the present embodiment employs a person, particularly a face, as a subject to be searched by a user, and a face image as an image representing a video section in units of video sections in which a person is shown from a moving image. The video section is extracted and output. According to this configuration, the user can easily grasp the video section in which the subject to be searched is shown.

FIG. 1A is a diagram illustrating a hardware configuration of the image search apparatus. The image search device includes a CPU (Central Processing Unit) 1, a storage device 2, an input device 3, a display device 4, and an imaging device 5. Each device is configured to be able to communicate with each other, and is connected by a bus or the like.
The CPU 1 controls the operation of the image search device and executes a program stored in the storage device 2.
The storage device 2 is a storage device such as a magnetic storage device or a semiconductor memory, and stores a program read based on the operation of the CPU 1, data that must be stored for a long time, and the like.
In the present embodiment, the CPU 1 performs processing according to the procedure of the program stored in the storage device 2, thereby realizing functions in the image search device and processing according to a flowchart described later.
The input device 3 is a mouse, a keyboard, a touch panel device, a button, or the like, and inputs various instructions.
The display device 4 is a liquid crystal panel, an external monitor, or the like, and displays various types of information.
The imaging device 5 is a camcorder or the like, and includes an imaging device such as a CCD (Charge Coupled Devices) or a CMOS (Complementary Metal Oxide Semiconductor). Note that moving image data captured by the imaging device 5 is stored in the storage device 2 or the like. In addition, a moving image basically includes a series of a plurality of frames, and has a still image corresponding to each frame.
Note that the hardware configuration of the image search apparatus is not limited to the above-described configuration. For example, the image search device may include an I / O device for performing communication between various devices. For example, the I / O device is an input / output unit such as a memory card or a USB cable, or a transmission / reception unit such as wired or wireless.

  FIG. 1B is a diagram illustrating a functional configuration of the image search apparatus according to the present embodiment. The processing and functions of the image search apparatus are realized by the video input unit 100, the face sequence generation unit 200, the face sequence feature extraction unit 300, the face sequence storage unit 400, and the face sequence search unit 500.

  The video input unit 100 inputs moving image data related to the video imaged by the imaging device 5 to the image memory unit 210. The video input unit 100 may be configured to read moving image data from a storage medium that stores moving image data. The video input unit 100 may be configured to read moving image data stored in a server or the like via the Internet or the like.

The face sequence generation unit 200 includes an image memory unit 210, a face detection unit 220, a face tracking unit 230, and a representative pattern extraction unit 240.
The face sequence generation unit 200 analyzes the moving image input to the image memory unit 210, extracts a face image (subject image in a broad sense) from each frame in a video section in which a face is shown, and generates a face sequence ( In a broader sense, a sequence) is generated.
Here, the face sequence is face information extracted from consecutive video sections and incidental information related to the extracted face image. The supplementary information includes the start time and end time of the face sequence, the number of the frame from which the face image is extracted, information on the area where the face image is cut out from the frame from which the face image is extracted, and the like.
The image memory unit 210 is a storage area provided in the storage device 2 and temporarily stores the moving image data output from the video input unit 100 for each frame.
The face detection unit 220 detects a face pattern from a predetermined frame of moving image data, and outputs the detection result (detection result) to the face tracking unit 230.
The face tracking unit 230 tracks the face pattern detected by the face detection unit 220 from the subsequent frames, generates a face sequence based on the tracked result (tracking result), information on the face area, The section or the like is output to the representative pattern extraction unit 240 or the like.
The representative pattern extraction unit 240 extracts a face image representing a face sequence (representative face image) based on the output from the face tracking unit 230.

The face sequence feature extraction unit 300 includes a face state analysis unit 310 and a face feature amount calculation unit 320.
The face state analysis unit 310 analyzes a face state of each face image in the face sequence and extracts a face feature amount in order to extract a face image effective for determination based on the similarity of the face from the face images in the face sequence. The analysis result (analysis result) is output to the calculation unit 320.
The face feature amount calculation unit 320 calculates a face feature amount for a face image effective for determination based on the similarity of the face from the analysis result of the face state analysis unit 310. Further, the face feature amount calculation unit 320 extracts, as the representative face feature amount, the face feature amount that best represents the facial feature in the face sequence for each face sequence based on the calculated face feature amount.

  The face sequence storage unit 400 is a storage area provided in the storage device 2. The face sequence storage unit 400 stores incidental information, a frame in which a face feature amount is calculated, a face feature amount, a face image representing the face sequence, and the like.

The face sequence search unit 500 includes a query input unit 510, a similarity determination unit 520, and a display unit 530.
The query input unit 510 receives an input related to a search condition (query) for searching for a face sequence.
The similarity determination unit 520 calculates the similarity between the face sequence specified by the query and each face sequence stored by the face sequence storage unit 400. Then, the similarity determination unit 520 determines whether or not the calculated similarity is higher than a predetermined threshold, and outputs the face sequence having the similarity determined to be high to the display unit 530.
The display unit 530 arranges the face sequence output by the similarity determination unit 520 and displays the arranged face sequence on the display device 4 together with the representative face image stored in the face sequence storage unit 400.

  Next, the operation of the image search apparatus will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a flowchart illustrating processing in the image search apparatus.

  First, the video input unit 100 reads the moving image data input to the image memory unit 210 from the image memory unit 210 for each frame (step S100). The data read here is, for example, two-dimensional array data composed of 8-bit pixels, and is composed of R, G, B, and three surfaces. At this time, the image data may be compressed by a method such as MPEG (Moving Picture Experts Group) or JPEG (Joint Photographic Experts Group). In this case, the image data is decompressed according to a predetermined decompression method to obtain image data composed of RGB pixels.

Subsequently, the face detection unit 220, which is an example of a detection unit, detects a plurality of face patterns (subject patterns in a broad sense) from the frame of the moving image data, and outputs detection results to the face tracking unit 230 (step). S200).
For example, the face detection unit 220 performs face detection for each frame at a predetermined interval from the moving image data. Therefore, an example in which face detection is performed for each frame at a predetermined interval (in other words, an example of progress when video data is analyzed) will be described with reference to FIG.
As shown in FIG. 3, the moving image data A related to a video is configured to have a plurality of frames. In this example, face patterns are detected every seven frames. A rectangular image pattern B cut out from the frame is detected as a face image.

In the present embodiment, a configuration in which a method of detecting a face pattern in an image using a neural network (for example, see Reference 1) is employed.
Therefore, a method for detecting a face pattern in an image using a neural network will be described.
First, the face detection unit 220 reads image data to be detected into a memory and cuts out a predetermined area to be matched with the face from the read image. Then, the face detection unit 220 receives a distribution of pixel values of the cut out region as an input, and obtains one output by calculation using a neural network. At this time, the weights and threshold values of the neural network are learned in advance using a huge number of face patterns and non-face patterns. For example, the face detection unit 220 determines that a face pattern is output if the output of the neural network is 0 or more, and a non-face pattern otherwise.
Then, the face detection unit 220 scans the cutout position of the image pattern to be collated with the face which is an input of the neural network, for example, by scanning the entire image vertically and horizontally as shown in FIG. To detect the face pattern. Further, in the present embodiment, in order to cope with detection of face patterns of various sizes, the face detection unit 220 sequentially reduces the read images at a predetermined rate as shown in FIG. A configuration for performing face detection scanning is employed.
Reference 1: Rowley et al, "Neural network-based face detection", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.20, NO.1, JANUARY 1998

Note that the method of detecting a face pattern from an image is not limited to the method using a neural network, and various methods are applicable, and other methods may be employed (for example, see Reference 2). Of that.)
Reference 2: Yang et al, "Detecting Faces in Images: A Survey", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.24, NO.1, JANUARY 2002

Subsequently, the face tracking unit 230, which is an example of a generation unit, tracks the face pattern from the subsequent frame for each face pattern detected by the face detection unit 220, and outputs the tracked result to the representative pattern extraction unit 240 or the like. (Step S300).
That is, the face tracking unit 230 performs tracking in the subsequent frames for each of the face patterns (for example, the image pattern B shown in FIG. 3) detected in frames at a predetermined interval. As a result of the tracking, the face tracking unit 230 acquires incidental information such as the number of the frame from which the face image is extracted and information on the area in which the face image is cut out. Furthermore, the face tracking unit 230 generates a collection of face images continuous in time as a face sequence (for example, the sequence C shown in FIG. 3). In other words, the face tracking unit 230 generates a sequence in which subject images are collected for a series of frames for each subject pattern.

  Here, a configuration in which a face sequence is generated will be described with reference to FIG. FIG. 5 is a diagram illustrating an example of a flowchart relating to processing for generating a face sequence.

  First, the face tracking unit 230 sets a search area for searching for a face pattern in a subsequent frame based on information on an area where a face image corresponding to the face pattern is cut out (step S301). When searching from the frame next to the frame in which the face pattern is detected by the face detection unit 220, a neighboring rectangle in which the center position is shifted by a predetermined amount for the horizontal position and the vertical position with respect to the area where the face pattern is detected. The area is set as a face pattern search area. In the case of searching for the next frame, the face tracking unit 230 similarly uses the area where the face image of the face pattern tracked in the processing described later is cut out based on the tracking result. Set a new search area.

Subsequently, the face tracking unit 230 performs tracking by the face pattern based on the correlation between the image in the region cut out in the search region and the face image corresponding to the face pattern to be searched (step S302). That is, the face tracking unit 230 sequentially cuts out a rectangular area having the same size as the face pattern to be searched around the center position set as the search area, and the face image corresponding to the face pattern to be searched and the image in the cut-out area The correlation value using the brightness distribution of the template as a template is calculated. The correlation value is a value that quantitatively expresses the degree of relationship between two patterns. Then, the face tracking unit 230 outputs an area having the highest correlation value in the search area as a face pattern tracking result together with the correlation value.
In this embodiment, the face tracking unit 230 is described as using the correlation value of the luminance distribution for tracking the face pattern. However, the face pattern tracking is performed using the correlation of the pixel value distribution for each RGB. May be performed. Further, the face tracking unit 230 may track the face pattern using the correlation of the image feature amount such as the luminance distribution in the region and the histogram of the RGB values.

  Subsequently, the face tracking unit 230 determines whether the correlation value output in the tracking process is greater than or equal to a predetermined value (step S303). In the case of a predetermined value or more, the face tracking unit 230 can accurately track the face image corresponding to the face pattern detected by the face detection unit 220 and the face image related to the tracking result with a high probability of similarity. Determination is made, and the process proceeds to step S304. On the other hand, if it is equal to or smaller than the predetermined value, the face tracking unit 230 determines that there is no face image corresponding to the face pattern to be tracked because the similarity probability is low, and ends the tracking process.

  Then, the face tracking unit 230 moves the processing target related to tracking by the face pattern to the subsequent frame, and moves the process to step S301 (step S304).

Therefore, iterative processing is basically performed up to the frame immediately before the frame next to the frame where the face pattern is detected by the face detection unit 220, and a face sequence can be obtained for each detected face pattern.
As described above, the face tracking unit 230 according to the present embodiment employs a configuration in which a face pattern is searched and tracked from the subsequent frames for each face pattern detected by the face detection unit 220. However, for example, in addition to this configuration, the face tracking unit 230 may employ a configuration in which a face pattern is searched for and tracked even in the previous frame in which the face pattern was detected. In addition, for example, the face tracking unit 230 may employ a configuration that calculates a motion vector from a moving image and tracks a face pattern using the motion vector as a clue in addition to or instead of the configuration described above.

  Further, the face tracking unit 230 may track the face pattern using frames after a predetermined interval. According to this configuration, it is possible to prevent the face sequence from being divided too much due to something crossing in front of the face or the influence of the flash.

In addition, the face tracking unit 230 calculates a correlation between facial features of two neighboring face sequences in time (or frame position), and if the correlation is high, the sequences are integrated (combined into one). May be. That is, the face tracking unit 230 sets a sequence obtained by integrating from the start of the sequence on the front side to the end of the sequence on the back side as a new sequence (video section), and also integrates accompanying information. The face tracking unit 230 can use one of the face sequences as the representative face image.
The integration of the face sequences described above is sequentially repeated for all the front and back face sequences, and the face sequences are integrated.
However, the face tracking unit 230 does not use the video sequence represented by the face sequence as a candidate for the integration of the face sequence for a set in which the video sections represented by the face sequence are separated by a predetermined time or more. In addition, when a plurality of people appear in a video, there may be a case where video sections overlap in a plurality of face sequences. In such a case, since the person corresponding to each face sequence can be regarded as another person, the face tracking unit 230 is not used as a candidate for face sequence integration.

Subsequently, the representative pattern extraction unit 240 extracts a face image representing the face sequence based on the result of the face tracking unit 230 (step S400). That is, the representative pattern extraction unit 240 cuts out a rectangular region including the face region from one frame in the face sequence and stores it in the face sequence storage unit 400. The representative pattern extraction unit 240 sets the frame to be extracted as a frame at a predetermined interval from the beginning of the sequence (in other words, a frame located at a predetermined time) based on the sequence length, a frame having the largest face size, or the like. Also good. In addition, the representative pattern extraction unit 240 may set a frame to be cut out as a moving image obtained by cutting out a rectangular area including a face area from a part of each face sequence or a plurality of frames at a predetermined interval.
In the present embodiment, an index is created for the generated face sequence and the representative face image, and a configuration that can be easily accessed during search is adopted.

  Subsequently, the face sequence feature extraction unit 300 performs face sequence feature extraction based on the face sequence generated by the face tracking unit 230 (step S500).

  Here, the feature extraction of the face sequence will be described with reference to FIGS. FIG. 6 is a diagram illustrating a flowchart according to a process of analyzing a face sequence and extracting features. FIG. 7 is a diagram illustrating a progress situation when performing feature extraction of the face sequence along each step of FIG.

First, the face state analysis unit 310, which is an example of an analysis unit, analyzes the state of the face (subject) of each face image in the face sequence, and outputs the analysis result to the face feature amount calculation unit 320 (step S501). . The face state analysis unit 310 analyzes the state of the face of the face image (more broadly, the subject of the subject image), thereby enabling a face effective for determination based on the similarity of the face from the face image in each face sequence. Feature quantities can be extracted.
In order to make the determination based on the similarity of the face more accurate, it is important that each part of the face such as eyes, mouth, nose, etc. exists in the face image. In other words, the face facing forward rather than the face facing sideways or diagonally can accurately identify the eyes, mouth, nose, etc. This is effective for determination based on similarity.
Therefore, in this embodiment, the face state analysis unit 310 has a configuration for detecting the face orientation of each face image in the face sequence. FIG. 7 shows the face orientation B for each face image A in the face sequence.
On the other hand, for example, the face state analysis unit 310 may include a plurality of face discriminators having the same configuration as a face discriminator using a neural network. However, parameters for discrimination of each face discriminator are tuned and set by sample learning for each face direction. In this configuration, the face orientation corresponding to the face classifier having the highest output among the plurality of face classifiers, that is, the likelihood classifier, is set as the analysis result by the face state analysis unit 310.
In the present embodiment, the face state analysis unit 310 employs a configuration in which the face orientation is used for the analysis of the face state. For example, parts such as eyes, mouth, and nose are individually searched from the face image. It is also possible to adopt a configuration in which means for analyzing the presence / absence of each existence is provided.
Further, when the eyes are open, the similarity of the face can be determined more accurately than when the eyes are closed. Therefore, in addition to or instead of the above-described configuration, the face state analysis unit 310 employs a configuration that provides a means for determining whether the eyes are open or whether the eyes are closed, and analyzes the open / closed state of the eyes. May be.
In addition, when the face is well illuminated and the skin part is brightly photographed as a whole, the similarity of the face can be determined more accurately than when there is a partial shadow. Therefore, in addition to or instead of the above-described configuration, the face state analysis unit 310 is provided with means for determining the lighting state of the face from the brightness distribution of the skin portion of the face, and has a configuration for analyzing the lighting state. It may be adopted.

Subsequently, the face feature amount calculation unit 320, which is an example of a feature amount calculation unit, calculates a face feature amount for a face image effective for determination based on the similarity of the face based on the analysis result of the face state analysis unit 310 ( Step S502).
That is, the face feature amount calculation unit 320 calculates a face feature amount only for a face image effective for determination based on the similarity of the face from the analysis result. For example, the face feature amount calculation unit 320 calculates the face feature amount by narrowing down only to the front face image in a predetermined state.
More specifically, the face feature amount calculation unit 320 searches for a face feature point effective for face determination from the narrowed face image. For example, the face feature amount calculation unit 320 extracts the corners of the eyes, both ends of the mouth, the apex of the nose, and the like as the feature points of the face based on the pattern matching. Then, the face feature amount calculation unit 320 extracts a local luminance distribution at each feature point as a face feature amount by Gabor wavelet transform and vectorizes it. That is, as shown in FIG. 7, a face feature vector C (face feature amount) is calculated for each front face image.

Here, as a method for calculating the facial feature amount, a known method is adopted (for example, see Reference 3). Note that the method for calculating the face feature amount is not limited to the method described in Reference 3, and a method for calculating a local descriptor at each feature point (for example, see Reference 4). You may employ | adopt and may employ | adopt the simple method using the histogram etc. of the luminance distribution of a face image.
Reference 3: Wiskott et al, "Face Recognition by Elastic Bunch Graph Matching", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.19, NO.7, JULY 1997
Reference 4: Schmid and Mohr, "Local Grayvalue Invariants for Image Retrieval", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.19, NO.5, MAY 1997

Subsequently, the face feature quantity calculation unit 320 creates a plurality of clusters by clustering each calculated face feature quantity in order to extract a representative face feature quantity representing a face sequence (step S503).
For example, the face feature amount calculation unit 320 applies a K-means method (K-means method) to a predetermined number of clusters by inputting a plurality of face feature vectors (face feature amounts) calculated from the face sequence for clustering.

Subsequently, the face feature amount calculation unit 320, which is an example of the representative feature amount extraction unit, extracts the face feature amount that best represents the face feature of the face sequence as the representative face feature amount based on the clustering result (step S504). ).
That is, the face feature amount calculation unit 320 sets a cluster including a relatively large number of face feature vectors (for example, a cluster having the largest number of sample face feature vectors) as a main cluster, and an average vector of face feature vectors in the main cluster. Is calculated as the representative face feature amount.
For example, as shown in FIG. 8, a main cluster is determined based on a plurality of face feature vectors B, and an average vector (representative face feature amount C) of face feature vectors in the main cluster is obtained. Here, if the average vector of the main cluster is extracted as the representative face feature amount, the face feature is more resistant to the influence of noise contained in each face image than when one face feature amount is arbitrarily sampled. There is an effect that the amount can be extracted.

  Subsequently, the face feature amount calculation unit 320 stores the calculated face feature amount and the extracted representative face feature amount in the face sequence storage unit 400 (step S600). At this time, for the extracted representative face feature quantity, a configuration is adopted in which an index that matches the face image extracted by the representative pattern extraction unit 240 is created. According to this configuration, the face image and the representative face feature amount extracted by the representative pattern extraction unit 240 can be easily accessed during the search.

  Subsequently, the query input unit 510 receives an input of search conditions for searching for a face sequence (step S700). More specifically, the query input unit 510 receives designation by the user of a face sequence (that is, a search condition) associated with at least one face image.

Here, referring to FIG. 9, input of search conditions will be described. FIG. 9 is a diagram illustrating a display example of a user interface for inputting a search condition (query).
FIG. 9 shows an example in which the face sequences stored in the face sequence storage unit 400 are grouped for each video title and a list of representative face images of each face sequence is displayed in the input dialog window 800. . At this time, a video section or the like occupied by the face sequence in the video may be indicated. The user selects one face sequence as a query from this list display.

In FIG. 9, a title 801 is information indicating a video title. If there is no title, the shooting time, recording time, etc. are displayed. The representative face image 802 is a representative face image of the face sequence. A selection area 803 shows a state in which a representative face image is selected by the mouse cursor 804. The user can select the representative face image 802 by clicking the mouse cursor 804 on the representative face image 802. However, in this embodiment, there is only one representative face image that can be selected, and when a new selection is made, the representative face image that has been selected until then becomes a non-selected state.
The slider 805 is a vertical elevator. The user can display a representative face image that cannot be displayed on one screen by dragging the mouse cursor 804 on the slider 805 and scrolling the screen in the vertical direction.
The search execution button 806 is a button for executing a face sequence search.
Note that the search result by the query is not limited to the face sequence, and a face image may be the search result. In this case, a face area is detected by the face detection unit 220, and a feature amount is calculated by the face feature amount calculation unit 320 in the detected face area, and is vectorized, thereby being similar to the representative face image to be searched. The degree can be calculated.
In addition, a query may be set in combination with other auxiliary information such as a face sequence, a title of a video to which the face sequence belongs, a video keyword, a recording date, and the like, and the similarity determination unit 520 may narrow down the face sequences for which similarity determination is performed. .

  Subsequently, the similarity determination unit 520 as an example of a calculation unit calculates the similarity between the face sequence to be queried and each face sequence stored in the face sequence storage unit 400. That is, the similarity determination unit 520 calculates the similarity between the face image associated with the face sequence designated by the user and the face images that make up the generated face sequence. On the other hand, the similarity determination unit 520, which is an example of an extraction unit, extracts information related to a face sequence including a face image whose similarity satisfies a standard (in other words, the similarity is higher than a predetermined threshold). The extracted information is output to the display unit 530 (step S800). Note that the process of extracting a sequence is not limited to the configuration performed by the similarity determination unit 520, and may be performed by providing a new extraction unit. Here, FIG. 10 shows an outline of processing in the similarity determination unit 520.

In the present embodiment, as the similarity between face sequences, the one having the maximum similarity between a plurality of face feature amounts of each face sequence is used. However, the configuration is not limited to the configuration using the maximum similarity, and an average value of similarities may be used. That is, the similarity determination unit 520 calculates the similarity based on the correlation between the face feature amount calculated in step S502 and the face feature amount of the face image associated with each face sequence (search condition). Note that the reciprocal of the Euclidean distance between face feature vectors is used as the similarity between face feature amounts.
In this embodiment, the similarity between the face sequences is calculated after the query is set. However, the similarity between the face sequences is calculated in advance to store a highly correlated face sequence. This may be used when set or when a search is performed.

  For calculating the similarity between face sequences, a configuration using the representative face feature values of the face sequence may be employed in addition to or instead of the configuration using the plurality of face feature values of each face sequence. For example, the similarity determination unit 520 has the largest similarity between each face feature amount of the face sequence to be queried and the representative face feature amount of each face sequence stored in the face sequence storage unit 400. Used as similarity. That is, the similarity determination unit 520 calculates the similarity based on the correlation between the representative face feature value extracted in step S504 and the face feature value of the face image associated with each face sequence (search condition).

  Based on these, the similarity determination unit 520 is an example of a search unit that searches for a face sequence including a face image associated with a face sequence specified by the user.

  Subsequently, the display unit 530, which is an example of an output unit, organizes the results of the similarity determination unit 520 and displays them on the display device 4 (step S900). For example, the display unit 530 displays a video section (video section information) or the like corresponding to the face sequence based on the accompanying information (frame information) of the extracted face sequence.

The content displayed on the display device 4 is a representative face image (one face image of the face images constituting the face sequence) stored in the face sequence storage unit 400 associated with the video section. is there. Here, examples of display related to the face sequence are shown in FIGS.
In FIG. 11, the display unit 530 displays the video section C corresponding to each face sequence in the video together with the representative face image D stored in the face sequence storage unit 400 on the display device 4. In the moving image data A, thumbnails B of frames are displayed at predetermined time intervals. Video section C is an example of display of a video section based on the analysis result of the face sequence. When the display unit 530 displays the video section C, the user can grasp the appearance section of the person in the video. The representative face image D is a representative face image. When the display unit 530 displays the representative face image, the user can check the face representing the video section C.

  On the other hand, in FIG. 12, the display unit 530 displays the search result window 900 on the display device 4. The title 901 is information indicating a video title. When there is no title, the display unit 530 displays the shooting time, the recording time, and the like. The bar 902 is a bar that represents the entire time of the video. A video section 903 indicates a video section occupied by the face sequence in the video. The display unit 530 displays the bar 902 and the video section 903, so that the user can grasp the appearance section of the person in the video. The representative face image 904 is a representative face image. When the display unit 530 displays the representative face image, the user can check the face representing the video section. It should be noted that the representative face image 904 is within a predetermined range (for example, from the center of the video section 903 rather than the distance from the center of another video section different from the video section 903 so that the relation with the video section 903 can be recognized. It is preferable to be provided at a position closer to the distance.

  In the above embodiment, the image search apparatus employs a configuration that detects a human face as a subject pattern and generates a face sequence. However, the image search device has a configuration that generates a pattern of another subject that can grasp the content of the video. It may be adopted.

<Other embodiments>
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

According to the configuration of the above-described embodiment, a search can be performed in units of a video section in which a predetermined subject is accurately reflected in the video, so that the efficiency of video search and video editing is improved. In addition, accuracy is improved because a plurality of pieces of image information of the same person are used for collation.
Further, according to each of the above-described embodiments, a face can be detected from a video, a face image detected over a plurality of frames can be set as one face sequence, and search can be performed in units of face sequences. , Video editing efficiency is improved. Further, since the determination based on the similarity is performed using a plurality of pieces of image information of the same person, the accuracy is improved.

  Further, according to the above-described embodiment, it is possible to perform a search in units of video sections in which a predetermined subject is accurately reflected from many video sections. For this reason, the efficiency of video search is improved. In addition, since the best image information of the subject in the unit video section is extracted and used as the target of the video section search, collation is performed, so that the accuracy of person determination in the video section in which the subject is shown is further improved. Can do.

  Further, according to the above-described embodiment, when the similarity of the sequence is compared with the feature amount representing each sequence among the image information of the subject in the unit video section, the video section in which the subject is captured is used. The person determination can be performed more stably.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications can be made within the scope of the gist of the present invention described in the claims.・ Change is possible.

  100 video input unit, 200 face sequence generation unit, 300 face sequence feature extraction unit, 400 face sequence storage unit, 500 face sequence search unit

Claims (6)

Detecting means for detecting a plurality of subject patterns from a moving image having a plurality of frames;
Generating means for generating a sequence in which subject images in the moving image corresponding to each of the plurality of subject patterns are collected for a series of frames for each subject pattern;
Determination means for determining whether or not each of the subject images is effective for determining the similarity of the subject based on the state of each of the subject images of the plurality of subject images constituting the sequence generated by the generation means;
A feature amount calculating means for calculating a feature amount of the subject image determined to be valid by the determining means among the plurality of subject images;
Storage means for storing the feature quantity calculated by the feature quantity calculation means for the sequence in a storage unit;
Similarity calculation means for calculating the similarity between the feature amount for the sequence given as a search condition and the feature amount for each sequence stored in the storage unit;
Extraction means for extracting a sequence based on the similarity calculated by the similarity calculation means from the sequences stored in the storage unit;
Output means for outputting video section information corresponding to the sequence extracted by the extraction means as a search result;
An image search device comprising: The image search apparatus according to claim 1, wherein the output unit outputs one subject image of the subject images constituting a sequence corresponding to the video segment information to be output together with the video segment information . Based on the feature quantity distribution calculated by the feature quantity calculation means, further comprising representative feature quantity generation means for generating a representative feature quantity representative of the sequence;
The storage means further stores the representative feature amount generated by the representative feature amount generation means in the storage unit with respect to the sequence,
The similarity calculation means calculates a similarity based on the correlation between the feature quantity and the representative feature quantity for each sequence stored in the storage unit, for a sequence given as the search condition, according to claim 1 or 2 The image search apparatus described. The representative feature quantity generating means creates a plurality of clusters by clustering the feature quantities of the subject image calculated by the feature quantity calculation means, and the sequence based on a cluster containing relatively many feature quantities. The image search device according to claim 3 , wherein a representative feature amount is generated . An image search method executed by an image search device,
A detection step of detecting a plurality of subject patterns from a moving image having a plurality of frames;
Generating a sequence of collecting subject images in the moving image corresponding to each of the plurality of subject patterns for a series of frames for each subject pattern;
A determination step of determining whether or not each subject image is effective in determining the similarity of the subject based on the state of each subject image of the plurality of subject images constituting the sequence generated in the generation step;
A feature amount calculating step of calculating a feature amount of the subject image determined to be valid by the determination step among the plurality of subject images;
A storage step of storing in the storage unit the feature amount calculated by the feature amount calculation step for the sequence;
A similarity calculation step of calculating a similarity between a feature amount for a sequence given as a search condition and a feature amount for each sequence stored in the storage unit;
An extraction step for extracting a sequence based on the similarity calculated in the similarity calculation step from the sequences stored in the storage unit;
An output step for outputting video section information corresponding to the sequence extracted in the extraction step as a search result;
Image search method including A detection step of detecting a plurality of subject patterns from a moving image having a plurality of frames;
Generating a sequence of collecting subject images in the moving image corresponding to each of the plurality of subject patterns for a series of frames for each subject pattern;
A determination step of determining whether or not each subject image is effective in determining the similarity of the subject based on the state of each subject image of the plurality of subject images constituting the sequence generated in the generation step;
A feature amount calculating step of calculating a feature amount of the subject image determined to be valid by the determination step among the plurality of subject images;
A storage step of storing in the storage unit the feature amount calculated by the feature amount calculation step for the sequence;
A similarity calculation step of calculating a similarity between a feature amount for a sequence given as a search condition and a feature amount for each sequence stored in the storage unit;
An extraction step for extracting a sequence based on the similarity calculated in the similarity calculation step from the sequences stored in the storage unit;
An output step for outputting video section information corresponding to the sequence extracted in the extraction step as a search result;
A program that causes a computer to execute .

Images