JPWO2006025272A1 - Video classification device, video classification program, video search device, and video search program - Google Patents

Abstract

Disclosed is a video classification device that classifies a video scene that is a still image or a moving image. The video input unit inputs a video to be classified. The face posture detection unit detects face posture data from the input video. Here, the face posture data is at least a part of the data of the number of faces of the characters, the position, the size, and the direction of the face of each character. The reference face posture storage unit stores face posture data in advance as reference face posture data in association with each scene for each of various scenes that are classification categories. The face posture evaluation unit calculates the similarity between the face posture data detected from the input video and the reference face posture data, and classifies the scenes of the input video based on the similarity.

Description

  The present invention relates to a video classification device and a video classification program that classify video scenes that are still images or moving images, a video search device and a video search program that search for a specific scene from video scenes.

  In recent years, with the digitization of video data and the spread of data compression technology and large-capacity storage devices, it has become possible to store a large amount of video data. There is an increasing demand for technologies that will

  Conventionally, image features are converted into data and shown by image feature amount, the similarity of each image is determined using the image feature amount, and similar images are searched (for example, from accumulated images). Image search is being performed, and various methods, devices, and computer programs have been developed for that purpose.

  An example of a conventional video search device is described in Japanese Patent Application Laid-Open No. 2001-167110 (hereinafter referred to as Patent Document 1). FIG. 1 is a block diagram showing the configuration of the video search device described in Patent Document 1. The face image capturing unit 121 captures a face image from the face database 120 that stores face images of the characters based on the input character list (list of characters appearing in the video), and the face image is used as a face. Output as a database signal. The feature extraction unit 122 extracts the feature amount from the face database signal and outputs the feature amount signal. Further, the image input unit 111 reads a video to be searched and outputs the video as a frame signal for each frame. The face detection unit 112 detects the face area in the frame and outputs it as a face detection signal. The character identification unit 123 cuts out a face area using the face detection signal and the frame signal. Further, the feature signal is used to find out which character the cut-out face image most resembles. The image output unit 113 displays each frame on the display device 119 for each character through the interface 118. With such a video search device, an image can be searched for each person who appears in the video. For example, a scene in which a famous actor, politician, or the like appears can be quickly searched, and the burden on the video editor and the searcher can be reduced.

  However, although the video search device described in Patent Document 1 can classify or search for a scene in which a specific character appears, it cannot classify or search for a scene depending on the content of the scene of the video. For example, the video search device described in Patent Document 1 can perform a search by specifying the character “scene in which the person A is captured”. However, in the video search device described in Patent Document 1, who is the character. Regardless of the situation, you cannot search for scenes with similar contents. Specifically, for example, it is not possible to search only the scenes in which the caster and the commentator have a conversation from a plurality of news program videos having different performers broadcasted in one day. In addition, for example, it is not possible to narrow down and search only a scene having a specific content (for example, an important scene in a drama in which the person A appears) from the video in which the person A is shown. The reason for this is that the video search device described in Patent Document 1 searches the video only based on information about who the characters appearing in the video are.

  An object of the present invention is to provide a video classification device and a video classification program that can classify scenes of a video regardless of who the characters are.

  Another object of the present invention is to provide a video search device and a video search program capable of searching for a scene similar to a scene in which a user is interested.

  The image classification device according to the present invention is a face posture detection unit that detects at least a part of the face number, face position, size, and orientation data of each character from the image as face posture data, And a face posture evaluation means for classifying the scenes of the image using the face posture data.

  Since the scenes of the video are classified by the face posture data that are closely related to the scenes, the scenes of the video can be classified regardless of who the characters are.

  The image classification device further includes a reference face orientation storage unit that stores reference face orientation data, which is face orientation data to be compared with the face orientation data detected by the face orientation detection unit, in advance in association with each scene. The evaluation unit classifies the scenes of the video in which the face posture data is detected by comparing the face posture data detected by the face posture detection unit with the reference face posture data stored in the reference face posture storage unit. May be configured.

  The face posture detecting means may be configured to detect the reference face posture data from the learning video showing the scene and store the reference face posture data in the reference face posture storing means in association with the scene.

  The image classification device further includes a reference face orientation detection unit that detects reference face orientation data that is face orientation data that is compared with the face orientation data detected by the face orientation detection unit, from the reference image showing a specific scene. The face posture evaluation means compares the face posture data detected by the face posture detection means with the reference face posture data detected by the reference face posture detection means, so that the scene of the video in which the face posture data is detected is It may be configured to determine whether or not the particular scene is classified as a scene of the same type. According to this configuration, it is not necessary to determine and store the reference face posture data for each scene in advance.

  The image classification device further includes a reference condition storage unit that stores in advance the condition that the face posture data in each scene to be classified corresponds to each scene, and the face posture evaluation unit detects the face posture detection unit. The face posture data is configured to classify the scenes of the video in which the face posture data is detected by determining which of the conditions stored in the reference condition storage unit corresponds to which scene. May be.

  The face posture detection means is configured to calculate at least a part of the data of the position, size and orientation of the face of at least one character as a value relative to the faces of other characters. May be. According to this configuration, if the scenes in which the positional relationships of the characters to be photographed are similar to each other, the scenes can be classified as the same type of scenes regardless of the position of the camera that photographs the scene.

  In the case where the image is a moving image, the face posture detecting means determines the amount of change over time of at least part of the number of faces of the characters, the position of each character, the size, and the orientation data. It may be configured to be detected as face posture data.

  Further, the video search device according to the present invention includes a video storage unit that stores a plurality of still images or moving images as search target videos, the number of faces of the characters in the video stored by the video storage unit, and the number of each character. Face orientation evaluation means for searching for a specific scene from video scenes using at least a part of the face position, size, and orientation data.

  The video search device detects, from each video read from the video storage means, at least a part of the number of faces of the characters, the position, the size, and the orientation data of the faces of the characters as face posture data. A face posture detecting means, a reference face posture storing means for storing reference face posture data, which is face posture data to be compared with the face posture data detected by the face posture detecting means, in advance in association with each scene; The apparatus further includes a scene designating unit that designates a scene according to the operation, and the face posture evaluating unit compares the face posture data detected by the face posture detecting unit with the reference face posture data corresponding to the designated scene. Thus, the designated scene may be searched from among the scenes of each video. With this configuration, a scene similar to the scene in which the user is interested can be searched.

  The face posture detecting means may be configured to detect the reference face posture data from the learning video showing the scene and store the reference face posture data in the reference face posture storing means in association with the scene.

  In the video search device, the video storage means associates at least a part of the number of faces of the characters in each video, the position, the size and the orientation of the face of each character in association with the scene of each video. A reference face posture storage unit that stores reference face posture data that is face posture data that is stored as face posture data and that is compared with each face posture data read from the video storage unit in advance in association with each scene, and The face posture evaluation means compares each face posture data read from the video storage means with the reference face posture data corresponding to the designated scene. May be configured to search for a specified scene among the scenes of each video. With this configuration, a scene similar to the scene in which the user is interested can be searched.

  The video search device detects, from each video read from the video storage means, at least a part of the number of faces of the characters, the position, the size, and the orientation data of the faces of the characters as face posture data. Face orientation detecting means, and reference face orientation detecting means for detecting reference face orientation data, which is face orientation data to be compared with the face orientation data detected by the face orientation detecting means, from a reference image showing a specific scene are further provided. The face posture evaluating means compares the face posture data detected by the face posture detecting means with the reference face posture data detected by the reference face posture detecting means to identify a specific scene from each video scene. It may be configured to search for a scene. According to this configuration, it is not necessary to determine and store the reference face posture data for each scene in advance.

  In the video search device, the video storage means associates at least a part of the number of faces of the characters in each video, the position, the size and the orientation of the face of each character in association with the scene of each video. Reference face posture detecting means for detecting reference face posture data, which is face posture data to be compared with each face posture data read from the image storage means, is further provided from a reference video stored as face posture data and showing a specific scene. The face posture evaluation means compares each face posture data read from the video storage means with the reference face posture data detected by the reference face posture detection means to determine a specific scene from the scenes of each video. It may be configured to search. According to this configuration, it is not necessary to determine and store the reference face posture data for each scene in advance.

  The video search device detects, from each video read from the video storage means, at least a part of the number of faces of the characters, the position, the size, and the orientation data of the faces of the characters as face posture data. Face orientation detecting means and reference condition storing means for storing the condition satisfied by the face orientation data in the specific scene to be searched are provided, and the face orientation evaluating means selects from the face orientation data detected by the face orientation detecting means. The specific condition may be searched from the scenes of each video by searching the face posture data satisfying the condition stored in the reference condition storage means.

  In the video search device, the reference condition storage unit further includes a scene designation unit that stores a condition, which is satisfied by the face posture data in each scene, for each scene, and that designates the scene according to a user's operation. By searching the face posture data detected by the face posture detecting means for face posture data satisfying the condition corresponding to the designated scene, thereby searching the designated scene from the scenes of each video. It may be configured as follows. With this configuration, a scene similar to the scene in which the user is interested can be searched.

  In the video search device, the video storage means stores at least a part of the number of faces of the characters in each video, the position, the size, and the orientation of the face of each character in association with the scene of each video. The face posture evaluation unit further stores a reference condition storage unit that stores a condition that is stored as face posture data and that is satisfied by the face posture data in the specific scene to be searched. It may be configured to search for a specific scene from scenes of each video by searching for face posture data that satisfies the condition stored in the reference condition storage means.

  In the video search device, the reference condition storage unit further includes a scene designation unit that stores a condition, which is satisfied by the face posture data in each scene, for each scene, and that designates the scene according to a user's operation. By searching for face pose data satisfying the condition corresponding to the specified scene from each face pose data read from the video storage means, thereby searching the specified scene from the scenes of each video. May be configured. With this configuration, a scene similar to the scene in which the user is interested can be searched.

  The face posture detecting means is configured to calculate at least a part of the data of the position, size and orientation of the face of at least one character as a value relative to the faces of other characters. May be. According to this configuration, if the scenes in which the characters to be photographed have the same positional relationship, the scenes of the same type can be searched regardless of the position of the camera that photographs the scene.

  In the case where the image is a moving image, the face posture detecting means determines the amount of change over time of at least a part of the number of faces of the characters, the position of each character, the size, and the orientation data. It may be configured to be detected as face posture data.

  Since the video scene is searched by the face posture data that is closely related to the scene, the video scene can be searched regardless of who the characters are.

  INDUSTRIAL APPLICABILITY The present invention is a video classification device that classifies and recognizes videos according to the content of the scene, a video search device that searches a database for a video containing specified scene content, and records and stores a large amount of video and uses it. The present invention can be applied to a video reproducing/displaying device for automatically reproducing and displaying a video required by a person, a video summarizing device for summarizing a video by searching and extracting only an important portion from the entire video. Further, the video in which the scenes are classified and searched according to the present invention may be, for example, a video of a television broadcast or a movie, or a still image such as a photograph.

It is a block diagram of the conventional example of a video search device. 1 is a block diagram showing an image classification device according to a first exemplary embodiment of the present invention. It is a block diagram showing an example of composition of a face posture primary detecting element. It is explanatory drawing which shows the process by a face posture detection part. 3 is a flowchart showing the operation of the video classification device of the first exemplary embodiment. It is a block diagram which shows the 1st modification of 1st Embodiment. It is a block diagram which shows the 2nd modification of 1st Embodiment. FIG. 6 is a block diagram showing a video search device according to a second embodiment of the present invention. It is a block diagram which shows the 1st modification of 2nd Embodiment. It is a block diagram which shows the 2nd modification of 2nd Embodiment. It is a block diagram which shows the 3rd modification of 2nd Embodiment. It is a block diagram which shows the video search apparatus by the 3rd Embodiment of this invention. It is a block diagram which shows the modification of 3rd Embodiment. It is a block diagram of the video classification device by the 4th Embodiment of this invention. It is explanatory drawing which shows the example of various scenes. It is a block diagram of a video search device according to a fifth embodiment of the present invention. It is explanatory drawing which shows the flow for learning images and reference face posture data in a preparation process. It is a flowchart which shows the example of a process progress of a preparation process. FIG. 13 is a block diagram of a video search device according to a sixth exemplary embodiment of the present invention.

First Embodiment Referring to FIG. 2, a video classification device according to a first embodiment of the present invention includes a video input unit 11, a face posture detection unit 12, a face posture evaluation unit 13, and a reference face posture storage unit 14. Equipped with.

  The video input unit 11 inputs a video into which scenes are classified. The video input by the video input unit 11 may be a still image or a moving image. The video input unit 11 may divide the input video for each scene when the video includes a plurality of different scenes (for example, when the scenes are switched in the moving image). The face posture detection unit 12 and the face posture evaluation unit 13 may perform the processing described below on the video divided for each scene. Note that the face posture detection unit 12 may perform the process of dividing the image for each scene. The video input unit 11 may also input a video that has been divided into scenes in advance. Although various known techniques can be used for scene division (detection of division points), for example, the technique described in Japanese Patent No. 2839132 can be used.

  The video input unit 11 is realized by a receiving device that receives a broadcast video signal, a device that reads a video from a video storage device that stores the video, and the like.

  The face posture detection unit 12 detects face posture data from a video into which scenes are classified. The face posture data is the number of faces of the characters, the position, size, and orientation of the face of each character, or a part of these data. Also, when the video is a moving image, the number of faces of the characters, the position, size and orientation data of the faces of each character, or the amount of change of some of these data over time. Also corresponds to the face posture data.

  As shown in FIG. 3, the face posture detection unit 12 includes a face area detection unit 21 and a detection unit 22.

  The face area detection unit 21 extracts a partial area (hereinafter, referred to as a face area) in which the face of a person is captured from the input video. Although various known techniques can be used for this, as an example, the technique described in Japanese Patent Laid-Open No. 2003-178304 can be used. In paragraph 0032 of the publication, pixels having a color close to the color specified as the skin color are extracted to extract the skin color area, and pixels having a color close to the color specified to the hair are extracted to extract the hair area. Describes a technique for detecting a face area. In the technique described in the publication, the face area is extracted from the image using the color information, and then face part extraction and person recognition are performed. However, in the present invention, it is not necessary to perform person recognition. All you have to do is extract. The face area may be extracted by a technique other than the technique described in Japanese Patent Laid-Open No. 2003-178304.

  The face area detection unit 21 extracts a plurality of face areas when there are a plurality of characters included in the video. FIG. 4A shows an example of the input video. FIG. 4B shows an example of the face area extracted by the face area detecting unit 21.

  The detection unit 22 detects, in each face area detected by the face area detection unit 21, the posture (face position and orientation) of the face of the person shown in the image, regardless of who the person is. .. A known method for detecting the face posture may be applied to the process for detecting the face posture. For example, as an example, “Rishi Ishiyama, two others, “Fast and high-precision face pose estimation using 3D appearance model”, Proceedings of 2004 IEICE General Conference, D-12-99 (hereinafter referred to as Reference 1 The face posture estimation technology described in "." can be used. The face pose estimation technique described in Reference 1 is a technique for obtaining a three-dimensional position/posture of a face from a face image by using a three-dimensional appearance model capable of generating a face image under arbitrary poses and illumination conditions. Regarding the "3D appearance model that can generate a face image in an arbitrary posture and lighting condition" used in the technique described in Reference 1, "Rishi Ishiyama, 2 people," by the appearance model construction on the 3D object surface Robust face matching against changes in posture and lighting," IPSJ Research Report, 2003-CVIM-139, July 4, 2003, pp. 61-68 (hereinafter referred to as Reference 2)". There is. If the person who appears in the video is known and the three-dimensional face shape data of the person is available, the techniques described in Literature 1 and Literature 2 can be used as they are in the present invention. .. However, in the present invention, the following technique is used to enable the estimation of the position and orientation of the face even when the characters are unknown and the three-dimensional shape data of the characters cannot be obtained. That is, the average three-dimensional shape data of a human face and the brightness image of each point on the face surface are prepared, and by using the technique described in Literature 2, an arbitrary posture/illumination condition of an average face is obtained. A model (average face model) capable of generating an image in is generated. By using this average face model in the technique described in Document 1, it is possible to estimate the position and orientation of the face even if the characters in the video are unknown.

  The technique that can be used for the detecting unit 22 described here is merely an example, and the posture of the face may be detected by another technique. For example, the posture of the face may be detected by the technique described in paragraphs 0038 and 0039 of Japanese Patent Laid-Open No. 2003-281549.

  When the input video is a moving image, the detection unit 22 performs this posture estimation process for each image of each frame to obtain time-series face posture data. A frame is a still image included in a moving image. For example, a video signal complying with the standard of NTSC (National Television System Committee) transmits 30 still images per second. In this example, each still image transmitted every 1/30 second becomes a frame.

  In addition, when there are a plurality of characters, the detecting unit 22 detects the face postures of the respective characters, and collects the number of persons (that is, the number of faces) and the face postures of the respective characters to obtain face posture data. To do. The detection unit 22 may detect the size of the face of each character and include the data of the size of the face in the face posture data.

  FIG. 4C shows an example of the face posture data detected by the detection unit 22. In FIG. 4C, the coordinates (x1, y1), (x2, y2) indicate the position of the face. The vectors R1 and R2 indicate the orientation of the face. Further, z1 and z2 are values indicating the size of the face.

  Further, when the input video is a moving image and is not divided into scenes, the detection unit 22 included in the face posture detection unit 12 can perform the scene division processing. For example, with respect to the time-series posture data detected by the detection unit 22, the detection unit 22 calculates a difference for each frame, and a frame having a large difference in posture data (for example, the position of the face where the number of characters changes, If there is a frame whose direction has changed significantly), split the scene there. The face posture evaluation unit 13 may perform the process described below (the process of calculating the similarity described below) on the frames up to immediately before the frame where the difference greatly changes.

  The reference face posture storage unit 14 is a storage device that stores face posture data in advance in association with each scene for each of various scenes that are classified. The face posture data stored in the reference face posture storage unit 14 is compared with the face posture data detected by the face posture detection unit 12, and the face posture data detected by the face posture detection unit 12 is classified into which scene. It is used to determine whether to do. Hereinafter, the face pose data compared with the face pose data detected from the scene classification target video will be referred to as reference face pose data, in distinction from the face pose data detected from the scene classification target video. In the present embodiment, the reference face pose data is used to determine which category of scene the scene of the video in which the face pose data is detected by the face pose detecting unit 12 is to be classified.

  As an aspect of storing the reference face orientation data in the reference face orientation storage unit 14, for example, there are the following aspects. An image (hereinafter referred to as a learning image) prepared in advance for each of various scenes that are to be classified is input. Then, the face orientation detecting unit 12 detects the reference face orientation data from the learning image and stores it in the reference face orientation storage unit 14 in association with the scene of the learning image. At this time, for example, the video input unit 11 may input the learning video. In this case, the video input unit 11 also serves as a learning video input unit. Further, a learning video input unit for inputting a learning video may be provided separately from the video input unit 11.

  Further, instead of inputting the learning image and detecting the reference face posture data from the learning image, the reference face posture data corresponding to each scene is manually determined in advance, and the reference face posture data corresponding to the scene is determined. The data may be stored in the reference face posture storage unit 14.

  The face posture evaluation unit 13 calculates the degree of similarity between the face posture data detected from the images into which the scenes are classified and each reference face posture data stored in the reference face posture storage unit 14. The face posture evaluation unit 13 classifies the scene of the video (that is, the input video) in which the face posture data is detected, as the scene corresponding to the reference face posture data having the highest similarity to the face posture data. Then, which scene the input video scene is classified into is output as a scene classification result 51. As an output mode of the scene classification result 51, for example, each classified scene may be displayed on the display device. Further, for example, the classified scenes may be stored in the storage device. In this case, each stored scene may be reproduced later and displayed on the display device.

Various methods can be used for setting the parameters of the face posture data and calculating the degree of similarity. As an example, a three-dimensional vector T having the position (x, y) of the face on the image and the size z of the face as elements, and a three-dimensional direction vector R indicating the direction of the face (in this example, the face direction is The rotation about the axis is ignored) and is used as the face posture data. Of the face posture data of the t-th frame of the video (the parameter t is unnecessary in the case of a still image), a three-dimensional vector having the position and size of the face as elements is represented by T _q (t). Also, of the face posture data of the t-th frame of the video, a three-dimensional vector representing the orientation of the face is represented as R _q (t). However, the norm of T _q (t) and R _q (t) is 1. Also, one of the various scenes that are the categories of classification is referred to as scene i. Of the reference face pose data corresponding to the scene i, a three-dimensional vector having the face position and size as elements is represented as T _i (t). Of the reference face posture data corresponding to the scene i, a three-dimensional vector representing the face orientation is represented as R _i (t). In this case, the similarity between the scene of the input video and the scene i can be expressed as, for example, the reciprocal of the weighted sum obtained by Expression 1 below.

式１において、ａ，ｂは、それぞれ顔の位置および大きさと、顔の向きのどちらをどの程度重視して類似度を決定するのかを定める重み定数であり、ともに任意に定めることができる。なお、式１におけるＲ_ｑ（ｔ）・Ｒ_ｉ（ｔ）は、ノルム１のベクトルの内積なので−１から＋１の値をとる。

In Expression 1, a and b are weighting constants that determine which of the position and size of the face and the orientation of the face should be emphasized to determine the similarity, and both can be arbitrarily set. It should be noted that R _q (t)·R _i (t) in the equation 1 takes a value from −1 to +1 because it is the inner product of the vector of norm 1.

  Although the case where the reciprocal of Expression 1 is used as the similarity is shown here, Expression 1 is an example of an expression for calculating the similarity. The parameters of the position and orientation of the face and the calculation formulas of the degree of similarity can be changed in various ways. That is, the calculation formula of the similarity is not limited to a particular formula. Further, instead of calculating the degree of similarity as a numerical value as in this example, it is also possible to apply various methods of pattern recognition that classify each scene classification by using face posture data as a feature amount. ..

  The processes of the face posture detection unit 12 and the face posture evaluation unit 13 including the face area detection unit 21 and the detection unit 22 are executed by, for example, a CPU of a computer that operates according to a program. The processes of the face posture detection unit 12 and the face posture evaluation unit 13 may be executed by the same CPU.

  FIG. 5 is a flowchart showing the operation of the video classification device according to the first embodiment. Here, it is assumed that an image that has been divided in advance for each scene is input and it is determined to which scene the scene of the input image is classified. In step 101, the video input unit 11 takes in the video as a scene classification target into the video classification device as data. In step 102, the face area detecting section 21 in the face posture detecting section 12 detects the area of the face of the character appearing in the image input in step 101. At this time, when a plurality of characters appear in the face area detection unit 21, the face area detection unit 21 detects a plurality of face areas corresponding to the respective characters. In step 103, the detection unit 22 of the face posture detection unit 12 detects the position and posture of the face in each detected face area. At this time, face size data may be detected. The detection unit 22 passes the number of characters detected as described above, the position, orientation, and size of each face, or a part of these data to the face posture evaluation unit 13 as face posture data. In step 104, the face posture evaluation unit 13 reads out the reference face posture data corresponding to each scene that is a classification category from the reference face posture storage unit 14. Subsequently, in step 105, the face posture evaluation unit 13 compares the face posture data with the respective reference face posture data to calculate the degree of similarity, and in each scene corresponding to the reference face posture data, the input image is input. The scene with the highest similarity to the scene is identified. Then, in step 106, the face posture evaluation unit 13 classifies the input video scene as a scene of the same type as the scene identified as the scene having the highest degree of similarity, and determines which of the input video scenes. Whether or not the scene is classified is output as a scene classification result 51.

  In the present embodiment, rather than recognizing who the characters are, the face posture data of the characters that are closely related to the content of the video scene is detected and used to detect the scene of the input video. Classify. Therefore, the video can be classified based on the content of the video scene regardless of who the characters in the video are.

  Next, a modified example of the first embodiment will be described. FIG. 6 is a block diagram showing a first modification. In this modification, a reference condition storage unit 15 is provided instead of the reference face posture storage unit 14 shown in FIG. The reference condition storage unit 15 is a storage device that stores, for each of various types of scenes that are classified into classifications, a condition that the face posture data in the scene satisfies, in association with the scene. That is, while the reference face posture storage unit 14 stores the face posture data, the reference condition storage unit 15 stores the condition satisfied by the face posture data in each scene. In the case of such a configuration, the face posture evaluation unit 13 reads the conditions, which are satisfied by the face posture data in each scene, from the reference condition storage unit 15 in step 104. Then, in step 105, the face posture evaluation unit 13 determines whether the face posture data passed from the detection unit 22 (that is, the face posture data detected from the input video) satisfies the condition corresponding to which scene. judge. That is, the face posture evaluation unit 13 specifies the condition that the face posture data satisfies among the conditions read in step 104, and the scene of the input video is classified into the scene corresponding to the condition. judge. Then, which scene the input video scene is classified into is output as the scene classification result 51.

  FIG. 7 shows a second modification. This modified example includes a reference image input unit 16 that inputs a reference image, and a reference face orientation detection unit 17 that detects reference face orientation data from the reference image, instead of the reference face orientation storage unit 14 illustrated in FIG. 2. The reference face posture detection unit 17 may detect the reference face posture data from the reference video, in the same manner as the face posture detection unit 12 detects the face posture data from the video to be classified. The face posture evaluation unit 13 also determines the degree of similarity between the face posture data detected from the scene classification target video and the reference face posture data detected from the reference video. Then, the face posture evaluation unit 13 determines whether or not the scene of the video to be classified is the same type of scene as the specific scene projected by the reference video, based on the similarity. If the similarity is equal to or higher than a predetermined threshold value, the scene of the video to be classified is classified as a scene of the same kind as the specific scene projected by the reference video. If the similarity is less than the threshold value, the scene of the video to be classified is classified as a scene different from the specific scene projected by the reference video.

  In this modified example, it is not necessary to store the reference face posture data for each scene in the storage device in advance. The user can classify the scenes of the video input to the video input unit 11 into a scene of the same type as the reference video and a scene different from the reference video simply by inputting the reference video.

  The reference video input unit 16 is realized by, for example, a device that reads the reference video from the reference video storage device that stores the reference video. The processing of the reference face posture detection means 17 may be executed by, for example, the CPU of a computer that operates according to a program.

  Further, in the first embodiment, when there are a plurality of characters, the detection unit 22 in the face posture detection unit 12 calculates the face posture data of a certain one character as a value relative to the other characters. You may. For example, when two faces are facing each other, a vector from one face to the other face may be calculated as the face direction.

  The number of characters who calculate face posture data as a value relative to other characters may be one. In addition, face posture data may be calculated as a relative value with respect to other characters for all the characters or a part of the characters.

  As a mode of calculating the face posture data of a character as a value relative to other characters, for example, the following modes are available. For example, a reference character (character A) is set in advance, and face posture data of all other characters (characters B, C, D,...) Are used as a reference. It may be set as a relative value with respect to the character A. Further, the reference person may be changed for each character for which the face posture data is calculated. For example, it is assumed that four characters A to D appear in the video as characters. At this time, when calculating the face posture data of the character A, it is calculated as a relative value to the character B, and when calculating the face posture data of the character C, it is calculated as a relative value to the character D. May be.

  When calculating the position of the face in the face posture data as a value relative to other characters (reference characters), for example, when the face position of the reference character is the origin of the face in the coordinate system, The position table can be calculated.

  When the face size of the face posture data is calculated as a value relative to other characters (reference characters), for example, if the face size of the reference character is 1 (not 1) The size of the face can be calculated.

  When face orientation data is calculated as a relative value with respect to other characters (reference characters), for example, the vector of the face orientation of the reference character and the face attitude data are calculated. It is sufficient to calculate the angle with respect to the vector of the face direction of the character who is playing, and use that angle as a relative value of the face direction.

  Generally, when detecting the position and orientation of a face, the position and orientation of the face are calculated as absolute values in a specific coordinate system that is independent of the characters. This coordinate system is often based on, for example, the position or orientation of the camera that is shooting. In such a case, for example, even in a scene in which the faces of the characters to be photographed are the same, if the position and orientation of the camera that captures the characters are different, different face positions and orientations are detected. It will be. As a result, even scenes in which the faces of the photographed characters are the same cannot be classified as the same type of scene.

  However, when calculating the position, size, and orientation of a face or some of these data as values relative to the faces of other characters, the calculated value depends on the position and orientation of the camera. Absent. Therefore, if the scenes in which the positional relationships of the photographed characters are similar, the scenes can be classified as the same kind of scenes regardless of the position of the camera that captures the scenes. In this case as well, the calculated value differs depending on who is the reference person. For example, if the reference person is arbitrarily changed and recalculated, it is possible to easily determine whether or not the scenes are of the same type. be able to.

Second Embodiment A video search device according to a second embodiment searches a video database for a video showing a scene of the same type as a designated scene, and as shown in FIG. 8, the first video shown in FIG. In addition to the configuration of the embodiment, a video database 18 and a scene designation unit 19 are provided.

  The video database 18 is a storage device that stores various videos to be searched. The video stored in the video database 18 is a plurality of still images and moving images.

  The video input unit 11 reads each search target video (video group) stored in the video database 18 and passes it to the face posture detection unit 12.

  The face posture detection unit 12 reads each search target video stored in the video database 18 via the video input unit 11. Then, the face-posture detecting unit 12 detects the face-posture data of the character from each of the read search target videos. The process in which the face posture detection unit 12 detects face posture data from each search target video is similar to the process in which the face posture detection unit 12 described in the first embodiment detects face posture data. The face posture detection unit 12 passes each face posture data (face posture data group) detected from each video to the face posture evaluation unit 13.

  The scene designating unit 19 designates a scene from various scenes according to a user operation. The scene designation unit 19 may be realized by, for example, a user interface device such as a keyboard or a mouse and a CPU that operates according to a program.

  Similar to the first embodiment, the reference face posture storage unit 14 stores reference face posture data in advance for each scene in association with each scene. The mode of storing the reference face posture data in the reference face posture storage unit 14 is also the same as in the first embodiment. That is, the learning image input means (not shown) inputs the learning image, and the face posture detection unit 12 detects the reference face posture data from the learning image and associates the reference face posture data with the scene of the learning image. It may be stored in the posture storage unit 14. Alternatively, the reference face posture data corresponding to each scene may be manually determined in advance, and the reference face posture data may be stored in the reference face posture storage unit 14 in association with the scene. The reference face posture storage unit 14 passes the reference face posture data corresponding to the scene designated by the scene designation unit 19 to the face posture evaluation unit 13. As a result, the face posture evaluation unit 13 reads the reference face posture data corresponding to the scene designated by the scene designation unit 19.

  The face posture evaluation unit 13 calculates the degree of similarity with the reference face posture data from the reference face posture storage unit 14 for each face posture data detected from each search target video. The calculation of the degree of similarity by the face posture evaluation unit 13 may be performed, for example, in the same manner as the calculation of the degree of similarity described in the first embodiment. However, as described above, the formula used for calculating the similarity is not limited to a specific formula. When the calculated similarity is equal to or higher than a predetermined threshold, the face posture evaluation unit 13 determines that the scene of the video in which the face posture data is detected is the same kind of scene as the designated scene. When the calculated similarity is less than the threshold value, it is determined that the scene of the video in which the face posture data is detected is a scene different from the designated scene. The face posture evaluation unit 13 determines the degree of similarity with the reference face posture data for each face posture data detected from each search target video, and retrieves the scene determined to be the same kind of scene as the designated scene. Output as 52. As an output mode of the search result 52, for example, the searched scene may be displayed on the display device. Further, for example, the searched scene may be stored in the storage device. In this case, the stored scene may be reproduced later and displayed on the display device.

  In the present embodiment, from a wide variety of videos stored in the video database 18, videos in which the reference face pose data preset for the video scene specified by the user and the face position and pose of the character are similar are extracted. Is configured to. Therefore, it is possible to search for videos having similar video scene contents.

  Note that, as in the first embodiment, when the video is a moving image, the face posture detection unit 12 determines the number of faces of the characters, the position, size, and orientation data of the faces of the characters, Alternatively, a change amount of some of these data over time may be passed to the face posture evaluation unit 13 as face posture data.

  In addition, when there are a plurality of characters, the face posture detection unit 12 may calculate the face posture data of one character as a relative value with respect to another character. For example, when two faces are facing each other, a vector from one face to the other face may be calculated as the face direction. When the face posture data of a character is calculated as a value relative to another character, it may be calculated in the same manner as in the first embodiment.

  As already explained, when calculating the position, size, and orientation of a face or some of these data as values relative to the faces of other characters, the calculated value is the position of the camera. It doesn't depend on the direction. Therefore, if the scenes in which the positional relationships of the photographed characters are similar, the scenes of the same type can be searched regardless of the position of the camera capturing the scene. If face pose data is not calculated as a relative value with respect to other characters, even if the positional relationships of the characters being photographed are similar, it is determined that the scene is different depending on the position and orientation of the camera. , It may be excluded from the search target.

  Next, a modified example of the second embodiment will be described. FIG. 9 is a block diagram showing a first modification of the second embodiment. In the configuration shown in FIG. 8, the video database 18 stores each search target video, and the face posture detection unit 12 detects the face posture data from each search target video. On the other hand, in the modified example shown in FIG. 9, the video database 18 stores face posture data in each search target video in advance in association with each search target video. The data input unit 30 reads the face posture data (face posture data group) corresponding to each search target video from the video database 18, and passes it to the face posture evaluation unit 13. The face posture evaluation unit 13 reads the face posture data group stored in the video database 18 via the data input unit 30. Further, the face posture evaluation unit 13 reads the reference face posture data corresponding to the scene designated by the scene designation unit 19 from the reference face posture storage unit 14, as in the case already described. The face posture evaluation unit 13 calculates the degree of similarity with the reference face posture data from the reference face posture storage unit 14 for each face posture data. Subsequent operations are similar to those of the embodiment shown in FIG. The operations of the scene designation unit 19 and the reference face posture storage unit 14 are also the same as in the case of the configuration shown in FIG.

  FIG. 10 is a block diagram showing a second modification of the second embodiment. The modified example illustrated in FIG. 10 includes a reference condition storage unit 31 instead of the scene designation unit 19 and the reference face posture storage unit 14 illustrated in FIG. The reference condition storage unit 31 in the present modification is a storage device that stores a condition that the face posture data in the specific scene satisfies in association with the specific scene. In the case of this configuration, the face posture evaluation unit 13 receives the face posture data group from the face posture detection unit 12 as in the case of the configuration shown in FIG. Further, the face posture evaluation unit 13 reads from the reference condition storage unit 31 the condition that the face posture data satisfies in a specific scene. Then, the face posture evaluation unit 13 extracts face posture data that satisfies the condition read from the reference condition storage unit 31 from the face posture data passed from the face posture detection unit 12. The face posture evaluation unit 13 outputs the scene of the video in which the face posture data is detected as the search result 52.

  FIG. 11 is a block diagram showing a third modification of the second embodiment. This modification has a configuration in which a scene designation unit 19 is added to the modification shown in FIG. The reference condition storage unit 31 in the present modification is a storage device that stores, for each of various scenes, a condition that the face posture data in the scene satisfies in association with the scene. The scene designating unit 19 designates a scene from various scenes according to a user operation. The reference condition storage unit 31 passes the conditions corresponding to the scene designated by the scene designation unit 19 to the face posture evaluation unit 13. As a result, the face posture evaluation unit 13 reads the conditions corresponding to the scene designated by the scene designation unit 19. The face posture evaluation unit 13 extracts, from the face posture data passed from the face posture detection unit 12, face posture data satisfying the condition passed from the reference condition storage unit 31. The face posture evaluation unit 13 outputs the scene of the video in which the face posture data is detected as the search result 52.

  In the modifications shown in FIGS. 10 and 11, the operations of the video database 18, the video input unit 11, and the face posture detection unit 12 are the same as those in the configuration shown in FIG. Further, in the modified examples shown in FIGS. 10 and 11, the image database 18 may store the face posture data of each search target image in advance in association with each search target image. Then, instead of the video input unit 11 and the face posture detection unit 12, a data input unit 30 having the same configuration as that shown in FIG. 9 may be provided. In this case, the face posture evaluation unit 13 may read the face posture data group stored in the video database 18 via the data input unit 30.

Third Embodiment The third embodiment is a video search device that searches a video database for a video that shows a scene of the same type as a scene of an input reference video, and as shown in FIG. Instead of the scene designation unit 19 and the reference face posture storage unit 14 in the embodiment, a reference video input unit 41 and a reference face posture detection unit 42 are provided. The operations of the video database 18, the video input unit 11, the face posture detection unit 12, and the face posture evaluation unit 13 are the same as those in the second embodiment.

  The reference video input unit 41 inputs a reference video showing a specific scene. The reference image input to the reference image input unit 41 is selected by the user. That is, an image showing a specific scene that the user wants to search is selected by the user, and the reference image is input to the reference image input unit 41 by the user.

  The reference face posture detection unit 42 uses the reference image input by the reference image input unit 41 to obtain the number of faces of the characters in the reference image, the position, the size and the direction of the face of each character, or these data. A part of the data is detected as reference face posture data. The process in which the reference face posture detection unit 42 detects the reference face posture data is the same as the process in which the face posture detection unit 12 detects the face posture data. The reference face posture detection unit 42 passes the detected reference face posture data to the face posture evaluation unit 13.

  The video input unit 11 reads each search target video (video group) stored in the video database 18 and passes it to the face posture detection unit 12. The face posture detection unit 12 reads the video group stored in the video database 18 via the video input unit 11. Then, the face posture detection unit 12 detects the face posture data of the characters from each of the read search target videos, and passes each face posture data (face posture data group) to the face posture evaluation unit 13.

  The face posture evaluation unit 13 calculates the degree of similarity with the reference face posture data from the reference face posture detection unit 21 for each face posture data detected from each search target video. The calculation of the degree of similarity is the same as in the second embodiment. When the calculated similarity is equal to or higher than a predetermined threshold value, the face posture evaluation unit 13 determines that the scene of the video in which the face posture data is detected is the same kind of scene as the specific scene projected by the reference video. .. When the calculated similarity is less than the threshold value, it is determined that the scene of the video in which the face posture data is detected is a scene different from the specific scene projected by the reference video. The face posture evaluation unit 13 determines the degree of similarity with the reference face posture data for each face posture data detected from each search target video, and the scene determined to be the same kind of scene as the specific scene projected by the reference video. Is output as the search result 52. The output mode of the search result 52 is similar to that of the second embodiment.

  In this embodiment, when a user inputs a certain reference video into the apparatus, the database is searched for a video corresponding to the same scene. Therefore, it is not necessary to define the scene classifications to be searched in advance and set the reference face pose data for each of them, and the user can input the video corresponding to the video scene to be searched, and the similar scene content can be obtained. You can search for videos.

  Note that, as in the first embodiment, when the video is a moving image, the face posture detection unit 12 determines the number of faces of the characters, the position, size, and orientation data of the faces of the characters, Alternatively, a change amount of some of these data over time may be passed to the face posture evaluation unit 13 as face posture data.

  In addition, when there are a plurality of characters, the face posture detection unit 12 may calculate the face posture data of one character as a relative value with respect to another character. For example, when two faces are facing each other, a vector from one face to the other face may be calculated as the face direction. When the face posture data of a character is calculated as a value relative to another character, it may be calculated in the same manner as in the first embodiment. Similar to the second embodiment, when the face posture data is calculated as a relative value with respect to other characters, if the positions of the characters to be photographed are similar to each other, a camera for photographing that scene is used. It is possible to search for the same type of scene regardless of the position, etc.

  Next, a modified example of the third embodiment will be described. FIG. 13 is a block diagram showing a modification of the third embodiment. In the configuration shown in FIG. 12, the video database 18 stores each search target video, and the face posture detection unit 12 detects face posture data from each search target video. On the other hand, in the modified example shown in FIG. 13, the video database 18 stores face posture data of each search target video in advance in association with each search target video. The data input unit 30 reads the face posture data (face posture data group) corresponding to each search target video from the video database 18, and passes it to the face posture evaluation unit 13. The face posture evaluation unit 13 reads the face posture data group stored in the video database 18 via the data input unit 30. The face posture evaluation unit 13 calculates the degree of similarity with the reference face posture data from the reference face posture detection unit 42 for each face posture data. Subsequent operations are the same as in the case of the configuration shown in FIG. The operations of the reference video input unit 41 and the reference face posture detection unit 42 are also the same as in the case of the configuration shown in FIG.

Fourth Embodiment As a fourth embodiment, a video classification device that classifies program videos received by television broadcasting according to scenes will be described. This embodiment corresponds to a modification of the first embodiment shown in FIG. Referring to FIG. 14, the video classification device according to the fourth exemplary embodiment includes a reception device 61 that receives a television broadcast, a computer 71 that performs processing, and a reference condition storage device 81.

  The receiving device 61 corresponds to the video input unit 11 in FIG. The reference condition storage device 81 corresponds to the reference condition storage unit 15 in FIG.

  The computer 71 includes a CPU 72, a storage device 73, an interface 74 with the receiving device 61, an interface 75 with the reference condition storage device 81, and a display device 76. Note that a keyboard, a mouse, or the like that serves as a user interface operated by the user may be provided. The CPU 72 operates according to the video classification program stored in the storage device 73. The interface 74 receives the video signal from the receiving device 61. The interface 75 sends and receives data to and from the reference condition storage device 81. Operates according to the video classification program.

  The configuration illustrated in FIG. 14 is an example of the configuration of the video classification device, and the video classification device may have another configuration. For example, instead of processing the received video, the classification processing for each scene may be performed on the video stored (for example, recorded) in the video storage device (not shown). In this case, the video storage device functions as the video input unit 11, reads the stored video, and sends the video to the computer 71. In the case of such a configuration, the receiving device 61 is unnecessary. Further, the video storage device and the reference condition storage device 81 may be realized as the same device.

  The receiving device 61 receives a television broadcast signal, converts the signal into digital data of video that can be processed by a computer, and transmits the digital data to the computer 71. The CPU 72 receives the digital data of the video and detects the face posture data from the digital data of the input video.

In this example, the face posture data is described by the following parameters. However, the method of describing the face posture data is not limited to the case shown below. Let n be the number of faces of the characters. The position of the face of the character j is expressed by coordinates as (x _j , y _j ). Similarly, the size of the face of the character _j is represented by z _j . Further, the direction vector of the face of the character _j is represented as R _j . Parameters of other characters (for example, j) are similarly expressed using the subscript j.

For example, the CPU 72 detects, as a cut point, a frame in which the inter-frame change amount of the face posture data exceeds a predetermined threshold in the video, and divides the video for each scene. The change amount of the direction vector is evaluated by the angle formed by the two vectors as an example. As an example of the threshold value of each change amount, the change amount of n is 1, the change amount of x and y indicating the position and the change amount of z indicating the size of the face are 10% of the image size and the change of the face orientation vector, respectively. The amount is set as 20° or the like. In this case, if the number n of faces of the characters changes by 1, the image is divided at that time. If the x-coordinate x _j changes by 10% or more of the width of the display area of the display device 76, the image is divided at that time. Similarly, if the y-coordinate y _j changes by 10% or more of the vertical length of the display area of the display device 76, the image is divided at that time. If z _j representing the size of the face changes by 10% or more of the area of the display area of the display device 76, the image is divided at that point. The width of the face may be used as the size of the face. In this case, if the face width z _j changes by 10% or more of the horizontal width of the display area of the display device 76, the image is divided at that time. If the face orientation vector changes by 20° or more, the image is divided at that point. The dividing method mentioned here is an example, and the image may be divided according to other criteria depending on the content of the image.

  The reference condition storage device 81 stores a condition satisfied by the face posture data in each scene. In this example, "the scene where the caster reads the news (hereinafter referred to as scene A)", "the reporter's scene from the site (hereinafter referred to as scene B)", "the caster and the news commentator" It is assumed that the conditions that are satisfied by the face posture data in scene A, scene B, and scene C are stored in association with each of the "interchange scenes (hereinafter, referred to as scene C)".

  15(a), 15(b), and 15(c) show examples of scene A, scene B, and scene C, respectively. The images of these scenes are displayed on the screen with typical face positions (they may have almost the same size) and orientations, regardless of who the performers are. Therefore, by extracting the position and orientation of the faces of the characters (the size may be extracted) from the video and classifying the video using this, classification according to the content of each scene becomes possible. An example of conditions that the face posture data satisfies is shown below. Scene A has a feature that the size of the face is relatively large because there is only one character and the face is near the center. In scene B, the face is facing forward, but the size is about half of that in scene A, and it is not in the center of the screen. Scene C is characterized by having a plurality of characters and their faces facing each other. The reference condition storage device 81 stores the conditions of the face posture data representing these characteristics. As an example of a conditional expression representing such a condition, the condition of the following expression is used. However, the direction vector facing the front is F. The size of the face is represented by the width of the face, and the horizontal width of the display area of the display device 76 is represented by s.

The condition corresponding to the scene A is expressed as follows, for example. Since there is only one character, the face orientation vector R and the face size z are represented without subscripts. Also, cos ⁻¹ is an inverse cosine function (arc cosine).

n=1 and cos ⁻¹ (F·R)<20° and z>0.2·s
The condition corresponding to the scene B is expressed as follows, for example.

n=1 and cos ⁻¹ (F·R)<20° and z<0.2·s
The condition corresponding to the scene C is expressed as follows, for example. Note that one of the characters is a character i, and the other one is a character j.

n≧2 and cos ⁻¹ (R _i ·R _j )<20° for a set of certain characters i and j
An example of another condition stored in the reference condition storage device 81 will be described. In the above examples of scenes A to C, when the video is a moving image, it is determined whether or not the face posture data at each time of the scene of the video satisfies the conditions of the scenes A to C. Become. When the video is a moving image, it is possible to classify the scenes based on the amount of change in the face posture data between frames, instead of making a determination on the face posture data at each time. Hereinafter, this case will be described. In this example, in a fighting game video such as boxing, "a scene in a stalemate (hereinafter, referred to as scene D)", "a KO scene or a scene in which a decisive punch is hit (hereinafter, referred to as scene E)". , "Condition scenes with a close distance (hereinafter referred to as scene F)", the conditions in scene D, scene E, and scene F are stored. This condition is a condition that is satisfied by the amount of change in the face pose data of the character over time.

  Scene D is a scene in which two opponents face each other and maintain a constant distance. The relative face directions of the two opponents are opposite to each other, and the distance between the two faces relative to the size of the face is substantially constant. There is a feature called. Scene E is characterized in that the directions of the faces change drastically and the faces do not face each other. The scene F is characterized in that the distance between both faces with respect to the size of the face is extremely small. As an example of the conditional expression representing the above condition, the condition of the following expression is used, for example. However, the distance between the faces of the two people at time t when two characters are extracted in the order of larger faces in the scene is d(t). Further, the average value of the face size is f(t), and the unit vectors of the face orientations of the two characters are R1(t) and R2(t), respectively. Further, the time difference between the frames to be processed is p. For example, in the case of complying with the NTSC standard, p=1/30 seconds is set when it is determined whether or not the condition is satisfied for each frame. Further, when thinning out some frames and determining whether or not the condition is satisfied every few frames, p becomes larger than 1/30 second. Here, the case of complying with the standard of NTSC is illustrated, but the moving image may not comply with the standard of NTSC.

  The condition corresponding to the scene D is expressed as follows, for example.

At an arbitrary time t>p, |d(t)−d(t−p)|<f(t) and d(t)>3.0·f(t) and R1(t)·R2(t)< -Cos 20°
The condition corresponding to the scene E is expressed as follows, for example. However, max(m,n) represents the larger value of m and n.

For a certain time t, max(|cos ⁻¹ (R1(t)·R1(t−p))|, |cos ⁻¹ (R2(t)·R2(t−p))|)>45° and | cos ⁻¹ (R1(t), R2(t))|>30°
The conditions corresponding to the scene F are expressed as follows, for example.

At an arbitrary time t>p, |d(t)-d(tp)|<f(t) and d(t)<3.0·f(t)
The conditions in the scenes A to C and the conditions in the scenes D to F shown above are examples, and the conditions are not limited to the above. Conditions other than the above may be used for more accurate classification.

  Next, the CPU 72 determines which of the above-exemplified conditions the face posture data of each frame of the input video satisfies, and determines which scene each frame is classified into. To do. In the present embodiment, since the moving image is targeted, it is determined whether or not any of the exemplified conditions is satisfied with respect to the face posture data detected from each frame image of the image, and all the frames included in the image are determined. When 90% or more of the face posture data of the frame images satisfy the same condition, it may be determined that the entire video corresponds to the scene according to the condition. The conditional expression described here and the ratio (90%) to all frames serving as the determination standard are examples, and can be variously changed.

  In the present embodiment, for example, the above-described scene classification may be automatically performed on all news program videos broadcasted in one day, and each scene may be stored in the storage device for each scene classification. Then, when it is desired to view the comments of various commentators regarding news that occurred in one day, the scene classified as the scene C may be reproduced and displayed on the display device. As a result, the viewer does not need to watch all of the many news programs broadcast on that day, and there is an effect that only the scene in which the caster and the news commentator are talking can be extracted and efficiently viewed. At this time, unlike the prior art, it is not necessary to know who is the announcer or commentator who appears in various programs of the day, and the comment video of any commentator can be viewed.

Fifth Embodiment Next, as a fifth embodiment, a video search device for searching a recorded video for a specific scene will be described. The configuration of this embodiment corresponds to the second embodiment of FIG. Referring to FIG. 16, the video search device according to the present embodiment stores a video storage device 91 that records various videos to be searched, a computer 71 that performs processing, and reference face posture data for each scene. The reference face posture storage device 82.

  The video storage device 91 includes a video storage unit 92, a video reading unit 93, and an interface 94 with the computer 71. The video storage unit 92 stores various videos to be searched. The image reading unit 93 reads various images stored in the image storage unit 92 and transmits a signal of the read image to the computer 71 via the interface 94.

  The computer 71 includes a CPU 72, a storage device 73, an interface 74 with the video storage device 91, an interface 75 with the reference face posture storage device 82, a display device 76, and an operation unit (a user interface operated by a user). (Eg, keyboard, mouse, etc.) 77. The CPU 72 operates according to the video search program stored in the storage device 73. The interface 74 receives a video signal from the video storage device 91. The interface 75 sends and receives data to and from the reference face posture storage device 82.

  In the present embodiment, the video storage unit 92 corresponds to the video database 18 in FIG. The video reading unit 93 corresponds to the video input unit 11 in FIG. The reference face posture storage device 82 corresponds to the reference face posture storage unit 14 in FIG. The CPU 72 that operates according to the video search program executes the processing of the face posture detection unit 12 and the face posture evaluation unit 13 in FIG. The operation unit 77 and the CPU 72 correspond to the scene designation unit 19.

  The configuration shown in FIG. 16 is an example of the configuration of the video search device, and the video search device may have another configuration. For example, the computer 71 may be built in the video storage device 91.

  First, as a preparation process, a process of storing reference face posture data, which serves as a reference for each scene to be classified, in the reference face posture storage device 82 is performed in advance. In the present embodiment, an example of a method of preliminarily collecting a video group belonging to each scene to be classified as a learning video group and storing face posture data detected from the learning video group as reference face posture data will be described. .. Here, it is assumed that the learning video input unit (not shown in FIG. 16) included in the video storage device 91 inputs the learning video from the outside. FIG. 17 is an explanatory diagram showing the flow of the learning video and reference face posture data in the preparation process. FIG. 18 is a flowchart showing an example of the progress of the preparation process.

The learning image input unit 97 of the image storage device 91 inputs a learning image showing a certain scene (scene S _k ) and transmits the signal of the learning image to the computer 71 through the interface 94. To do. The CPU 72 receives the learning video signal via the interface 74. In this manner, the CPU 72 reads the learning video by receiving the signal from the learning video input unit 97 (step 107).

The CPU 72 that executes the process of the face posture detecting unit 12 detects the reference face posture data from the learning video (step 108). Here, it is assumed that the number of faces of the characters in the learning video is _nk . Further, the position of the face of a certain character i is _defined as coordinates (x _ki , y _ki ). In addition, the size of the face of the character i is z _ki . Further, the face direction of this character is R _ki . The CPU 72, for example, _represents the reference face pose data of the scene S _k as D _k ={n _k , x _ki , y _ki , z _ki , R _ki }, and stores the reference face pose data in the reference face pose storage device 82 (step 109). ). When the target video is a moving image, the CPU 72 operating as the face posture detection unit 12 detects the face posture data in each frame or every few frames, and obtains the time-series reference face posture data D _k (t). Then, the reference face posture data is stored in the reference face posture storage unit 14.

  Here, a case has been described in which the reference face posture data is detected and stored from the learning video that shows the scene Sk, but the reference face posture data is similarly detected from the learning video that shows another scene and the reference face posture data is also stored. The posture data is stored in the reference face posture storage device 82.

By performing the above-described preparation process in advance, it becomes possible to search for images belonging to various scenes such as the scene S _k from various images stored in the image storage unit 92.

The CPU 72 that executes the process of the scene designation unit 19 designates a scene according to the operation of the operation unit 77 by the user. For example, the CPU 72 causes the display device 76 to display a list of searchable scene classifications, and in response to an operation of the operation 77, specifies a classification corresponding to a scene to be searched by the viewer from among the displayed classifications. You may. Here, it is assumed that the scene S _i is designated.

The CPU 72 that executes the process of the scene designation unit 19 designates the scene S _i and requests the reference face posture storage device 82 for the reference face posture data corresponding to the scene S _i . In response to this request, the reference face posture storage device 82 transmits the reference face posture data corresponding to the scene S _i to the computer 71. The CPU 72 that executes the processing of the face posture evaluation unit 13 receives the reference face posture data corresponding to the scene S _i from the reference face posture storage device 82.

  The video reading unit 93, which operates as the video input unit 11, reads each video group to be searched, which is stored in the video storage unit 92, and transmits the video group to the computer 71. At this time, the CPU 72 of the computer 71 detects each cut point in the image group received from the image storage device 91 and divides the image group into each scene, as in the fourth embodiment. Further, the image storage device 91 may perform a process of dividing the image group for each scene, and the image storage device 91 may transmit each image divided for each scene to the computer 71.

  The CPU 72, which executes the processing of the face posture detection unit 12, extracts the face posture data of the characters (the number of the characters (the number of faces) and the position and size of each face) from each search target video read from the video storage device 91. , Direction) is detected. This face posture data is represented by Q. When the search target video is a moving image, the face posture data is detected every frame or every few frames, and the time-series face posture data Q(t) is obtained.

The CPU 72 that executes the processing of the face posture evaluation unit 13 calculates the degree of similarity between the face posture data Q(t) detected from the search target video and the reference face posture data D _i (t) of the designated scene S _i. To do. When the calculated similarity is equal to or higher than a predetermined threshold value, the CPU 72 determines that the scene of the video in which the face posture data Q(t) is detected has the same content as the scene S _i . If the similarity is less than the threshold value, the CPU 72 determines that the scene of the video in which the face posture data Q(t) is detected has a content different from that of the scene S _i, and _excludes it from the search candidates.

Various pattern recognition methods can be used to calculate the similarity. For example, the reciprocal of the weighted sum obtained by Expression 1 may be obtained as the similarity. T _q (t) in Expression 1 is a three-dimensional vector representing the position and size of the face in the face posture data Q(t). T _i (t) in Expression 1 is a three-dimensional vector representing the position and size of the face in the reference face posture data D _i (t). R _q (t) is a three-dimensional vector representing the orientation of the face in the face posture data Q(t). R _i (t) is a three-dimensional vector representing the orientation of the face in the reference face posture data D _i (t). “A” and “b” are weighting constants that determine how much importance is placed on the position and size of the face and the orientation of the face to determine the similarity, and both can be arbitrarily set. In addition, when there are a plurality of characters, the value of Expression 1 may be calculated for each face orientation data of each character, and the result may be added. However, Formula 1 is an example of a formula for calculating the similarity, and the formula used for calculating the similarity is not limited to a specific formula.

  An example of another similarity calculation method will be described below. When there are a plurality of characters, the similarity may be calculated by adding the value of Expression 1 to the face posture data of each character, but the relative positions and orientations of the faces of a plurality of characters, Higher performance search may be performed by using the face size relationship. An example of such a case is a search for a scene in which a caster and a commentator discuss a comment in a news program. In such a scene, the position and size of the face change depending on the sitting position of the speaker, but since the faces face each other and speak, the relative face directions are opposite (face to face). You can search as a scene with the opposite direction. Also, in a drama or movie kissing scene, regardless of the absolute position and orientation of the faces of the characters, it is possible to search for a scene in which the faces of the two persons are in opposite directions and the relative distance is extremely small.

  An example of calculation of the similarity used in such a case will be shown. First, two characters are selected from the characters in the search target video. An example of the selection method is a method of selecting two persons having the largest face size. The distance between the faces of the two persons in the face posture data Q(t) is dQ(t). The average value of the sizes of the faces of the two persons in the face posture data Q(t) is set to fQ(t). Further, the face direction of the character 2 with respect to the character 1 is defined as a vector RQ(t). Further, the distance between the faces of two persons (two persons having the largest face size) in the reference face posture data is dD(t). Let fD(t) be the average value of the sizes of the faces of the two persons in the reference face posture data. Further, the direction of the face of the character 2 with respect to the character 1 in the reference face posture data is a vector RD(t). In this case, the reciprocal of the value of Expression 2 shown below can be used as the similarity.

以上に示した類似度の計算式はあくまで一例で、顔姿勢データＱ（ｔ）や参照顔姿勢データＤ（ｔ）の特徴量の選び方や類似度の計算式は様々に変更可能であり、様々な距離関数やパターン認識手法を利用可能である。また、複数の類似度の計算式を組み合わせて計算を行うことももちろん可能である。

The calculation formula of the similarity shown above is merely an example, and the method of selecting the feature amount of the face posture data Q(t) and the reference face posture data D(t) and the calculation formula of the similarity can be changed variously. Various distance functions and pattern recognition methods can be used. Further, it is of course possible to perform the calculation by combining a plurality of calculation formulas of the similarities.

  The CPU 72, which executes the processing of the face posture evaluation unit 13, outputs the face posture data and the reference face posture data for all the videos (all the videos received from the video storage device 91) stored in the video storage device 91. Processing for determining the degree of similarity is performed. Then, among the images received from the image storage device 91, the image of the scene determined to have a high degree of similarity is set as the search result.

  In the present embodiment, reference face posture data and face positions of the characters preset for the video scene designated by the user are selected from a wide variety of videos stored in the video storage unit 92 (corresponding to the video database 18). It is configured so as to extract images with similar postures. Therefore, it is possible to search for videos having similar video scene contents.

  In the present embodiment, the case where the reference face posture storage device 82 is provided and the scene is searched based on the reference face posture data stored in the reference face posture storage device 82 has been described. Instead of the reference face posture storage device 82, a reference condition storage device corresponding to the reference condition storage unit 31 shown in FIG. 11 may be provided. That is, the CPU 72 includes a reference condition storage unit that stores conditions that face orientation data satisfies in various scenes, and the CPU 72 searches for face orientation data that satisfies the conditions corresponding to the designated scene, and the same as in the designated scene. You may search for the scene.

Sixth Embodiment Next, as a sixth embodiment, instead of designating a predetermined scene, a video is input, and a video search is performed to search for a video having a scene content similar to that of the video. An example of the device will be described. The configuration of this embodiment corresponds to the configuration of the third embodiment shown in FIG. Referring to FIG. 19, the video search device according to the third embodiment of the present invention specifies a video storage device 91 that records various videos to be searched, a computer 71 that performs processing, and a scene to be searched. A reference video input device 84 to which a video storage medium 87 for recording a reference video for recording is mounted.

  The configuration and operation of the video storage device 91 are the same as those of the video storage device 91 in the fifth embodiment of FIG. 16, and thus description thereof will be omitted.

  The reference video input device 84 includes a storage medium mounting unit 86 in which a video storage medium 87 is mounted, a video reading unit 88, and an interface 85 with the computer 71. The video storage medium 87 is a video storage medium that can be attached to and detached from the storage medium mounting portion 86, and for example, a video cassette or a DVD (Digital Versatile Disk) can be used as the video storage medium 87. A video storage medium 87 storing a video (reference video) showing a scene that the viewer wants to search is mounted on the storage medium mounting unit 86. The image reading unit 88 reads the reference image stored in the image storage medium 87 attached to the storage medium attaching unit 86, and transmits the reference image signal to the computer 71 via the interface 85.

  The computer 71 includes a CPU 72, a storage device 73, an interface 74 with the video storage device 91, an interface 75 with the reference video input device 84, and a display device 76. Note that a keyboard, a mouse, or the like that serves as a user interface operated by the user may be provided. The CPU 72 operates according to the video search program stored in the storage device 73. The interface 74 receives a video signal from the video storage device 91. The interface 75 receives the video signal of the reference video from the reference video input device 84.

  In this example, the video storage unit 92 corresponds to the video database 18 in FIG. The video reading unit 93 corresponds to the video input unit 11 in FIG. The reference video input device 84 corresponds to the reference video input unit 16 in FIG. The CPU 72, which operates according to the video search program, performs the processes of the face posture detection unit 12, the reference face posture detection unit 42, and the face posture evaluation unit 13 of FIG.

  The configuration illustrated in FIG. 19 is an example of the configuration of the video search device, and the video search device may have another configuration. For example, the computer 71 may be built in the video storage device 91. Further, for example, the reference video input device 84 may be configured as a part of the video storage device 91.

  A video storage medium 87 storing a reference video of a video scene that the user wants to search is loaded in the storage medium loading unit 86 by the user. The video reading unit 88 reads the reference video stored in the video storage medium 87 and transmits it to the computer 71 via the interface 85.

  The CPU 72 that executes the process of the reference face posture detection unit 42 determines the number of characters (the number of faces) in the reference video, the position and the size of the face of each character, from the reference video read from the reference video input device 84. , The orientation is detected in time series, and is detected as reference face posture data D(t).

  Similar to the CPU 72 in the fifth embodiment, the CPU 72 that executes the process of the face posture detection unit 12 determines the number of characters (the number of faces) and each character from each search target video read from the video storage device 91. The face posture data Q(t) representing the position, size, and orientation of the face of the is read.

  The CPU 72 that executes the process of the face posture evaluation unit 13 calculates the similarity between the reference face posture data D(t) and the face posture data Q(t) detected from the search target video. As the similarity calculation method, for example, the calculation method already described may be applied. Further, as described above, the method of calculating the degree of similarity is not limited to a specific calculation method. The CPU 72 determines that the search target video is a video having the same scene content as the reference video input by the user when the similarity is equal to or higher than a predetermined threshold, and when the similarity is low, the search target video is a search candidate. Remove from.

  The CPU 72, which executes the processing of the face posture evaluation unit 13, outputs the face posture data and the reference face posture data for all the videos (all the videos received from the video storage device 91) stored in the video storage device 91. Processing for determining the degree of similarity is performed. Then, among the images received from the image storage device 91, the image of the scene determined to have a high degree of similarity is set as the search result.

In this embodiment, when a video storage medium 87 storing a certain reference video is attached to the reference video input device 84 by a user, a video corresponding to the same scene as the reference video is searched from the video storage device 91. It is configured. Therefore, it is not necessary to define the scene classifications to be searched in advance and set the reference face posture data for each of them, and the user only needs to input the video corresponding to the video scene to be searched, and the similar scene You can search the video of the content.

Claims (38)

A video classification device for classifying video scenes that are still images or moving images,
Face posture detecting means for detecting at least a part of the number of faces of characters from the video, the position of each face, size and orientation data as face posture data,
And a face posture evaluation unit that classifies the scenes of the video using the face posture data. A reference face posture storage unit that stores reference face posture data, which is face posture data to be compared with the face posture data detected by the face posture detection unit, in advance in association with each scene;
The face posture evaluation means compares the face posture data detected by the face posture detection means with the reference face posture data stored in the reference face posture storage means, thereby detecting the face posture data. The image classification device according to claim 1, wherein the scenes are classified.   3. The face posture detecting means detects the reference face posture data from a learning video showing a scene, and stores the reference face posture data in the reference face posture storing means in association with the scene. The described image classification device. From a reference image showing a specific scene, a reference face posture detecting means for detecting reference face posture data, which is face posture data to be compared with the face posture data detected by the face posture detecting means, is further provided.
The face posture evaluation unit detects the face posture data by comparing the face posture data detected by the face posture detection unit with the reference face posture data detected by the reference face posture detection unit. The video classification device according to claim 1, wherein it is determined whether or not the scene of the video is classified into the same type of scene as the specific scene. Further provided is a reference condition storage means for storing a condition satisfied by the face posture data in each classified scene in advance in association with each scene,
The face posture evaluation unit determines whether the face posture data detected by the face posture detection unit satisfies a condition corresponding to which scene among the conditions stored in the reference condition storage unit, The video classification device according to claim 1, wherein the scenes of the video in which the face posture data is detected are classified.   2. The face posture detecting means calculates at least a part of data of the position, size and orientation of the face of at least one character as a relative value with respect to the face of another character. The image classification device according to claim 5.   When the video is a moving image, the face posture detection means changes at least a part of the number of faces of the characters, the position, the size, and the orientation data of each character with the passage of time. The image classification device according to claim 1, wherein the amount is detected as face posture data. A video search device for searching a specific scene from video scenes that are still images or moving images,
Video storage means for storing a plurality of still images or moving images as video to be searched,
A specific scene is selected from the scenes of the video by using at least a part of the number of the faces of the characters in the video stored in the video storage means, the position, the size and the orientation of the face of each character. And a face posture evaluation means for searching for a video search device. Face posture detecting means for detecting at least a part of data of the number of faces of the characters, position, size and orientation of the faces of the characters as face posture data from each image read from the image storage means. When,
Reference face posture storage means for storing reference face posture data, which is face posture data to be compared with the face posture data detected by the face posture detection means, in advance in association with each scene,
Further comprising a scene designation means for designating a scene according to a user's operation,
The face posture evaluating means compares the face posture data detected by the face posture detecting means with reference face posture data corresponding to the scene designated by the scene designating means, and The video search device according to claim 8, wherein a designated scene is searched from the inside.   10. The face posture detecting means detects reference face posture data from a learning video showing a scene, and stores the reference face posture data in the reference face posture storing means in association with the scene. Video search device. The image storage means associates at least a part of the number of faces of characters in each image, the position, size and orientation of the face of each character in each image with the face posture data. Remember as
Reference face posture storage means for storing reference face posture data, which is face posture data to be compared with each face posture data read from the image storage means, in advance in association with each scene,
Further comprising a scene designation means for designating a scene according to a user's operation,
The face pose evaluation means compares the face pose data read from the video storage means with the reference face pose data corresponding to the scene designated by the scene designating means, so that the face pose evaluation means The video search device according to claim 8, wherein a scene specified by is searched. Face posture detection that detects at least a part of the number of faces of the characters, the position, size, and orientation data of the faces of the characters from each image read from the image storage means as the face posture data. Means and
A reference face posture detecting unit for detecting reference face posture data, which is face posture data compared with the face posture data detected by the face posture detecting unit, from a reference image showing a specific scene,
The face posture evaluation means compares the face posture data detected by the face posture detection means with the reference face posture data detected by the reference face posture detection means to select from among the scenes of the respective videos. The video search device according to claim 8, wherein the specific scene is searched. The image storage means associates at least a part of the number of faces of characters in each image, the position, size and orientation of the face of each character in each image with the face posture data. Remember as
A reference face posture detecting means for detecting reference face posture data, which is face posture data compared with each face posture data read from the image storing means, from a reference image showing a specific scene,
The face posture evaluation means compares the face posture data read from the video storage means with the reference face posture data detected by the reference face posture detection means to select the face posture data from the scenes of the respective videos. The video search device according to claim 8, which searches for a specific scene. Face posture detection that detects at least a part of the number of faces of the characters, the position, size, and orientation data of the faces of the characters from each image read from the image storage means as the face posture data. Means and
And a reference condition storage unit that stores a condition satisfied by the face posture data in a specific scene to be searched,
The face posture evaluation unit searches the face posture data detected by the face posture detection unit for face posture data satisfying the condition stored in the reference condition storage unit, thereby performing the processing in the scene of each video. The video search device according to claim 8, wherein the specific scene is searched from. The reference condition storage means stores a condition, which is satisfied by the face posture data in each scene, for each scene,
Further provided is a scene designation means for designating a scene according to a user operation,
The face posture evaluating means searches the face posture data detected by the face posture detecting means for face posture data satisfying a condition corresponding to a designated scene, thereby selecting the face posture data from the scenes of each video. The video search device according to claim 14, wherein the scene specified by the scene specifying means is searched. The image storage means associates at least part of the number of faces of the characters in each image, the position, the size and the direction of the face of each character as face posture data in association with the scene of each image. Remember
Further comprising a reference condition storage means for storing a condition satisfied by the face posture data in the specific scene to be searched,
The face posture evaluation means retrieves the face posture data satisfying the conditions stored in the reference condition storage means from the face posture data read from the video storage means, thereby selecting from the scenes of the respective videos. The video search device according to claim 8, wherein the specific scene is searched. The reference condition storage means stores a condition, which is satisfied by the face posture data in each scene, for each scene,
Further provided is a scene designation means for designating a scene according to a user operation,
The face posture evaluation means retrieves the face posture data satisfying the condition corresponding to the scene designated by the scene designation means from the face posture data read from the video storage means, thereby obtaining the scene of each video. The video search device according to claim 16, wherein a specified scene is searched from among the.   10. The face posture detecting means calculates at least a part of data of the position, size and orientation of the face of at least one character as a relative value with respect to the faces of other characters. The video search device according to any one of 1, 10, 12, 14, and 15.   The face posture detection means, when the video is a moving image, the amount of change over time of at least a part of the number of faces of the characters, the position, the size, and the orientation data of each character. The video search device according to any one of claims 9, 10, 12, 14, 15, and 18, which detects as face orientation data. A video classification program for causing a computer to classify video scenes that are still images or moving images,
A first instruction set for detecting at least a part of the number of faces of characters, the position, size, and orientation data of each character from the video as face posture data;
A second instruction set for classifying the scenes of the video using the face posture data.   In the second command step, the face pose data detected by the first command set and reference face pose data, which is face pose data to be compared with the face pose data, are stored in advance in association with each scene. 21. The image classification program according to claim 20, comprising an instruction set for classifying a scene of an image in which the face pose data is detected by comparing with reference face pose data stored in a reference face pose storage device.   The first instruction set includes an instruction set that detects reference face posture data from a learning video showing a scene and stores the reference face posture data in a reference face posture storage device in association with the scene. The image classification program according to Item 21. A third instruction set for detecting reference face posture data, which is face posture data to be compared with the face posture data detected by the first instruction set, from a reference image showing a specific scene,
The second instruction set compares the face posture data detected by the first instruction set with the reference face posture data detected by the third instruction set to obtain the face posture data. 21. The video classification program according to claim 20, further comprising an instruction set for determining whether a detected video scene is classified into a scene of the same type as the specific scene.   The second condition set is a reference condition storage device that stores the condition that the face pose data detected by the first command set is satisfied by the face pose data in each classified scene in advance in association with each scene. 21. The instruction set of claim 20, including an instruction set for classifying a scene of a video in which the face pose data is detected by determining which of the conditions stored in the corresponding to which scene is satisfied. Video classification program.   The first instruction set is an instruction set for calculating at least a part of data of the position, size, and orientation of the face of at least one character as a value relative to the faces of other characters. An image classification program according to any one of claims 20 to 24, including:   The first instruction set is a change over time of at least a part of the number of faces of the characters, the position, the size, and the orientation data of each character when the video is a moving image. The image classification program according to any one of claims 20 to 25, comprising an instruction set for detecting an amount as face posture data. A video search program for causing a computer to search for a specific scene from video scenes that are still images or moving images,
Of the data of the number of faces of the characters in the video, the position of each character's face, the size, and the orientation, which is stored in the video storage device that stores multiple still images or moving images as the video to be searched A video search program comprising a first instruction set for searching for a specific scene from scenes of the video using at least a part of the above. A second command for detecting at least a part of the data of the number of faces of the characters, the position, the size, and the direction of the faces of the characters from each image read from the image storage device as face posture data. A set,
A third instruction set that specifies the scene,
Further equipped with,
The reference face posture storage device stores the reference face posture data, which is face posture data to be compared with the face posture data detected by the second instruction set, in advance in association with each scene. Of the stored reference face pose data, the reference face pose data corresponding to the designated scene and the face pose data detected by the second command set are compared to obtain the scene of each of the images. The video search program according to claim 27, comprising a command set for searching a designated scene from the inside.   The second instruction set includes an instruction set for detecting reference face posture data from a learning video showing a scene and storing the reference face posture data in the reference face posture storage device in association with the scene. The video search program according to claim 28. Further comprising a third instruction set specifying a scene,
The first instruction set is such that at least a part of the face number, face position, size, and orientation data of each character in each image is associated with a scene of each image, and a face posture Reference face posture memory that stores in advance each face posture data read from a video storage device stored as data and reference face posture data that is face posture data to be compared with each face posture data in association with each scene in advance. A reference set of face pose data stored in the device, and a set of instructions for searching the specified scene from the scenes of the respective images by comparing the reference face pose data corresponding to the specified scene. The video search program according to claim 27. A second command for detecting at least a part of the data of the number of faces of the characters, the position, the size, and the direction of the faces of the characters from each image read from the image storage device as face posture data. A set,
And a fourth instruction set for detecting reference face posture data, which is face posture data to be compared with the face posture data detected by the second instruction set, from a reference image showing a specific scene.
The first command set compares the face pose data detected by the second command set with the reference face pose data detected by the fourth command set to obtain a scene of each of the images. The video search program according to claim 27, comprising an instruction set for searching for the specific scene from inside. An image storage device that stores at least part of the number of faces of characters in each image, the position, size, and orientation data of each character in each image as face posture data in association with the scene of each image. A fifth instruction set for detecting reference face pose data, which is face pose data to be compared with the stored face pose data, from a reference image showing a specific scene,
The first command set compares the face pose data read from the video storage device with the reference face pose data detected by the fifth command set, thereby selecting the face pose data from the scenes of the videos. The video search program according to claim 27, comprising an instruction set for searching for a specific scene. A second command for detecting at least a part of the data of the number of faces of the characters, the position, the size, and the direction of the faces of the characters from each image read from the image storage device as face posture data. Further equipped with a set,
The first instruction set is stored in a reference condition storage device that stores a condition that is satisfied by the face posture data in a specific scene to be searched from the face posture data detected by the second instruction set. The video search program according to claim 27, further comprising an instruction set for searching for the specific scene from among the scenes of each video by searching for face pose data satisfying a condition. Further comprising a third instruction set specifying a scene,
The first command set searches the face pose data detected by the second command set for face pose data satisfying the condition corresponding to the designated scene among the conditions stored in the reference condition storage device. The video search program according to claim 33, further comprising an instruction set for searching for a designated scene from among the scenes of each video.   The first command set is such that at least a part of the number of faces of characters in each image, the position, size, and orientation of the face of each character in each image is associated with face pose data. From the face posture data stored in the video storage device stored as, the face posture data satisfying the condition stored in the reference condition storage device that stores the condition satisfied by the face posture data in the specific scene to be searched is stored. The video search program according to claim 27, comprising an instruction set for searching for the specific scene from among the scenes of each video by searching. Further comprising a third instruction set specifying a scene,
The first instruction set is to retrieve face pose data that satisfies a condition corresponding to a designated scene among the conditions stored in the reference condition storage device from the face pose data stored in the video storage device. The video search program according to claim 35, further comprising an instruction set for searching for a designated scene from among the scenes of each video according to.   The second instruction set is an instruction set for calculating at least a part of the face position, size, and orientation data of at least one character as a value relative to the faces of other characters. The video search program according to any one of claims 28, 29, 30, 33, and 34, including:. The second instruction set, when the video is a moving image, changes with time with respect to at least a part of the number of faces of the characters, the position, the size, and the orientation data of each character. The video search program according to any one of claims 28, 29, 31, 33, 34, and 37, which includes an instruction set for detecting an amount as face posture data.

Images