JP5360979B2 - Important information extraction method and apparatus - Google Patents

Description

  The present invention relates to an important information extraction method and apparatus for extracting important information by analyzing a moving image such as a program video, and more particularly to an important information extraction method and apparatus for extracting an important frame in which an important object appears from a video. .

  Conventionally, as a method for recognizing an object such as an object or a person included in a video, a method using a detailed feature amount of a target object is generally used. In Non-Patent Document 1, for each object such as a national flag, a mountain, and a police officer, feature quantities of each video / audio are learned in advance and accumulated in a database, and extracted from the analysis target video. And a feature amount of each object stored in a database, and a technique for recognizing an object based on the similarity between the two is disclosed.

  Also, in Patent Document 1, scenes in which a specific person is shown are detected from program videos produced and broadcasted by a broadcasting station or a video production company based on image feature amounts, and these are displayed as a list of thumbnail images. Techniques to do this are disclosed.

JP 2009-110460 A

"High-Level Feature Extraction Experiments for TRECVID 2007", Proc of TRECVID 2007

  However, in the above-described prior art, attention is paid only to whether or not an object exists, and whether or not it is an object that a program producer wants to impress viewers (hereinafter sometimes referred to as “important object”). Was not considered. For this reason, in the prior art, for example, even if it is understood that a national flag is reflected, it has not been determined whether or not it is important information to be particularly conveyed to the viewer.

  An object of the present invention is to solve the above-described problems of the prior art and provide an important information extraction method and apparatus for extracting an important frame in which an important object appears from a program video or the like.

  In order to achieve the above object, the present invention is characterized in that the following measures are taken in an important information extracting apparatus for extracting important information by analyzing a video.

  (1) means for extracting feature values from video, means for dividing the video into a plurality of shots based on the feature values, a database for storing representative feature values of important frames in which important objects appear, and Means for extracting a representative frame from a shot; means for extracting an important frame candidate from the representative frame based on the feature quantity; means for obtaining meta information of a video; and representing a feature quantity of the important frame candidate as a representative of the important frame. Means for evaluating each important frame candidate in comparison with a characteristic amount, and means for extracting an important frame from the important frame candidates based on the meta information and the evaluation value of each important frame candidate. It is characterized by.

  (2) The database stores unique feature values for each object type, and the evaluation means compares the feature values of each important frame candidate with each unique feature value stored for each object type. An evaluation value is calculated for each type, and the meta information describes the number of appearances of important objects, the order of appearance, and the object type. The means for extracting the important frame is the important frame from a set of important frame candidates. Means for extracting important frame candidates in a combination according to the appearance order and the number of appearances of objects, and an evaluation value for each object type of each extracted important frame candidate based on the type of each important object, And a means for extracting the important frame candidate of the combination that maximizes the sum as an important frame. .

  (3) The means for extracting a representative frame is characterized in that the representative frame is extracted from a shot in a still image section.

  (4) The means for extracting the representative frame is characterized in that the representative frame is extracted from a shot of the moving object follow section.

  (5) A means for displaying a list of important frames as thumbnails is further provided.

  Each thumbnail image is associated with information for reproducing the video from the position of each important frame.

  According to the present invention, the following effects are achieved.

  (1) Important objects included in the video are extracted based not only on the evaluation based on the feature value but also on the collation result with the meta information describing the context of the video. Can be extracted with high accuracy.

  (2) Each important object candidate extracted from the video is evaluated for each object type, and the evaluation value for each object type is compared with the appearance count, appearance order, and object type of the important object described in the meta information. Thus, it is determined whether or not each important object candidate is an important object, so that the important object can be extracted with high accuracy.

  (3) Since the still image section is positioned as an important shot and the representative frame is extracted from the shot of the still image section, it is possible to efficiently extract a representative frame that is likely to cause an important object.

  (4) Since the animal body follow section is positioned as an important shot and the representative frame is extracted from the shot of the animal body follow section, it is possible to efficiently extract a representative frame where an important object is likely to appear.

  (5) Since the important frames extracted from the video are displayed as thumbnails and displayed in a list, information on the playback position of each important frame is linked to each thumbnail image. Playback from a desired important position becomes possible.

It is the block diagram which showed the structure of the system containing the important information extraction apparatus which concerns on this invention. It is the figure which showed an example of meta information. It is the flowchart which showed operation | movement of one Embodiment of this invention. It is the flowchart which showed the calculation procedure of the correlation between frames. It is the flowchart which showed the calculation procedure of information density distribution. It is the figure which showed typically the extraction method of the representative frame. It is the figure which showed the identification method of the important frame candidate typically. It is the figure which showed typically the evaluation method of an important frame candidate. FIG. 10 is a diagram illustrating a display example of a thumbnail image of an important frame.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing the overall configuration of a system including an important information extracting device according to the present invention, and a video distribution device 2 that distributes a program video to be analyzed and a separate program video. The meta information providing apparatus 3 that provides meta information, and the likelihood that each frame extracted from the program video is an important frame (likeness of an important frame) is compared with the meta information of the video to extract an important frame in the video The important information extracting apparatus 1 is a main component.

  The important information extraction device 1 is, for example, an STB (Set-top Box), and outputs a thumbnail image of an important frame extracted from a program video to the television / monitor device 4. As the meta information, context information that is manually created by a metadata provider or a viewer and broadcast on the Web or the like before or after the program video is broadcast can be used.

  In the important information extracting apparatus 1, the video receiving unit 101 includes a video accumulating unit 101a, and receives and stores the program video and the audio distributed from the video distributing apparatus 2. The feature amount extraction unit 102 includes a camera motion extraction unit 102a, a shot boundary extraction unit 102b, a color histogram distribution extraction unit 102b, an edge information extraction unit 102d, a feature point information (corner) extraction unit 102e, a telop information extraction unit 102f, and face recognition. An extraction unit 102g, an audio extraction unit 102h, and a volume extraction unit 102i are provided, and various image feature amounts and audio feature amounts are extracted from the temporarily stored video and its audio.

  The shot division unit 103 divides the program video into a plurality of shots based on the image feature amount and the audio feature amount. The appearance time length detection unit 104 targets an important shot (a “camera still section” and “animal body follow section”, which will be described later) that is likely to include an important frame, and the same object appears across the shot boundary. Detect the total length of time.

  In the appearance time length detection unit 104, the important shot detection unit 104a detects a camera still section and a moving object follow section based on the camera motion feature amount. The camera still section is a shot in which camera work does not exist for a certain time or more, and the moving object follow section is a shot in which the camera tracks the moving object for a certain time or more.

  In a shot focused on an important object, the subject is photographed on the camera for a certain period of time. Therefore, if the subject is a stationary object, the camera motion is stationary for a certain time or more, and if the subject is a moving object, the camera tracks the object for a certain time or more. Therefore, in the present embodiment, still shots and moving object follow shots of a certain time or longer are detected based on the camera motion feature amount, and these are extracted as positions of important shot candidates that are likely to include important frames. For example, “Torii et al.,“ Animal up-shot / follow-shot detection using video motion ”, Image Recognition / Understanding Symposium 2005 (July 2005)” is used to extract the follow-up shot of the moving object. Can be used.

  The representative frame extracting unit 104b extracts a representative frame from the shot in the camera still section and the shot in the moving object follow section based on various feature amounts extracted from the video. As will be described later in detail, the inter-frame correlation calculation unit 104c calculates an inter-frame correlation related to the image feature amount between the representative frames N and M of two important shot candidates that are continuous in time series. The appearance time length measurement unit 104d measures the total time length of shots including the representative frames N and M whose inter-frame correlation exceeds a predetermined threshold, and when this total time length exceeds a predetermined reference value, the total time length Is output as one of the indicators of likelihood (likeness) that the representative frame is an important frame.

  The information density distribution calculation unit 105 detects a Harris feature amount for each of the central image trimmed at the central portion and the remaining peripheral images for the representative frame of each important shot candidate whose total time length exceeds the threshold. A detection unit 105a that detects a luminance average, a detection unit 105c that detects a standard deviation of hue, and a DCT coefficient detection unit 105d that detects a DCT coefficient.

  The database (DB) 108 stores typical feature amounts of important frames for each object type. In this embodiment, as will be described in detail later, various objects including a Shop feature amount 108a representing a store image, a Product feature amount 108b representing a product image, and a Person feature amount 108c representing a human image. Features are stored.

  The important frame candidate evaluation unit 106 compares the time length of each important shot candidate and the Harris feature amount, luminance average, hue standard deviation, and DCT coefficient with the important frame teacher data stored in the database DB 108. Thus, an evaluation value of the likelihood of an important frame is calculated for each representative frame. The meta information acquisition unit 110 acquires meta information related to the analysis target video from the meta information providing apparatus 3.

  FIG. 2 is a diagram showing an example of the meta information, in which an important object is indexed with a broadcast channel (<channel>), a broadcast start time (<Time>), and a program name (<Title>) of program video. A plurality of subsections (<Subsection>) representing the appearance period of are described.

  In this example, there are three subsections in the program. The first subsection is “13:02”, “ABC restaurant (Shop)”, “Special curry rice (Product)” and “Special hamburger steak”. “(Product)” appears as an important object in this order ”. The second sub-section indicates that “From 13:15”, “General store Iroha (Shop)” will appear as an important object ”. Furthermore, the third sub-section starts from “13:30” and “○ × Coffee shop (Shop)”, “Homemade cheesecake (Product)” and “Store manager Mr. Sato (Person)” are important objects in this order. Appears at

  As described above, in this embodiment, important objects are classified into any one of shop, product, and person object types, and the meta information includes the number of appearances of important objects in the video, The appearance order and the object type are described as the video context.

  Returning to FIG. 1, as will be described in detail later, the important frame determination unit 107 determines the appearance order and the number of appearances of each important object described in the meta information from a set of important frame candidates extracted from the program video. The combination extraction unit 107a that extracts the important frame candidates in accordance with the combination, and the evaluation value for each object type of each extracted important frame candidate is added based on the type of each important object, and the sum is maximized. And an evaluation unit 107b that determines an important frame candidate of the combination as an important frame. The information providing unit 109 generates a thumbnail image of the determined important frame and provides it to the television / monitor apparatus 4.

  Next, the operation of the embodiment of the present invention will be described in detail with reference to the flowcharts of FIGS.

  In step S1, the video temporarily stored in the video storage unit 101a of the video receiving unit 101 is taken into the feature quantity extraction unit 102, and includes a shot boundary feature quantity, a camera motion feature quantity, a telop feature quantity, and a face recognition feature quantity. Various feature quantities are extracted.

  A shot boundary is a point at which the camera viewpoint is switched by video editing (a point at which a video is cut). For example, as disclosed in Japanese Patent Application Laid-Open No. 2007-134986, a shot boundary is a difference amount between successive frames. Can be based on. Camera motion is the up / down motion (tilt), left / right motion (pan), and zoom operation of the camera that shoots the video, and its feature value is calculated, for example, as a motion vector or optical flow that is MPEG encoded information. Can be obtained.

  For example, as disclosed in Japanese Patent Application Laid-Open No. 12-23062, the telop feature amount is displayed in (1) a telop in a predetermined area at the top or bottom of the screen. (2) Luminance when the telop appears and when the telop ends It can be detected using the feature that changes occur. The face recognition feature amount can be extracted by a conventional method disclosed in, for example, Japanese Patent Laid-Open No. 2006-508461.

  In step S2, the shot division unit 103 divides the program video into a plurality of shots based on each feature amount. In step S3, the telop shot centered on the telop as in the title screen is removed from the program video. In step S4, the important shot detection unit 104a of the appearance time length detection unit 104 acquires a camera still section and a moving object follow section as important shot candidates from each shot based on the camera motion feature amount.

  In step S5, as shown in an example in FIG. 6, a representative frame is acquired by the representative frame extraction unit 104b from the extracted camera still section and moving object follow section. As shown on the right side of the figure, the representative frame is acquired in the middle of each section, as shown on the left side of the figure, and the telop is associated with the image displayed on the screen. In the case where a new telop appears in a section using the feature that there are very many cases, a method of acquiring the moment when the telop completely appears as a representative frame may be used. In step S6, the inter-frame correlation calculation unit 104c calculates the inter-frame correlation related to the image feature amount between the representative frames N and M of the two consecutive important shot candidates n and m.

  FIG. 4 is a flowchart showing the calculation procedure of the inter-frame correlation. In step S21, representative frames of two consecutive shots n and m are selected from the representative frames extracted from the respective important shot candidates. Extracted as frames of interest N and M. In step S22, the preceding attention frame N is trimmed randomly or according to a predetermined rule to generate a plurality of types of partial images Nk (N1, N2, N3...). In step S 23, histogram correlation values r (N, M), r (N 1, M), r (N 2) relating to each feature amount between the target frame N and its post-partial image Nk and the subsequent target frame M. M), r (N3, M)... Are calculated.

  In step S24, the maximum values Max {r (N, M), r (N1, M)...} Of all histogram correlation values obtained in step S23 are compared with the reference correlation value Rref, and the maximum value Max { ...} is greater than or equal to the reference correlation value Rref, the process proceeds to step S25. In step S25, the sum [tn + tm] of the time length tn of the shot n having the target frame N as the representative frame and the time length tm of the shot m having the target frame M as the representative frame is compared with the reference time length tref. . If the time length sum [tn + tm] is greater than or equal to the reference time length tref, the process proceeds to step S26. In step S26, the two current frames of interest M and N are both determined to be important frame candidates, and are associated with the time length sum [tn + tm].

  On the other hand, if it is determined in step S24 that the maximum value Max {...} is less than the reference correlation value Rref, the process proceeds to step S27, and the time length tn of the shot with the target frame N as the representative frame is the reference time length. Compared with tref. If the time length tn is equal to or greater than the reference time length tref, the process proceeds to step S28, where the current frame of interest N is determined as an important frame candidate and is associated with the time length tn. In step S29, it is determined whether or not the above determination has been completed for all representative frames. If not completed, the process returns to step S21, and the above processes are repeated for the remaining representative frames.

  FIG. 7 shows the relationship between the transition of the camera motion, the position of the shot boundary, and the representative frame in a series of video sequences.

  Here, since the objects appearing in the representative frame 1 and the representative frame 2 are different, the histogram correlation value between them is low. Therefore, the representative frame 1 is not classified as an important frame candidate. On the other hand, since the representative frame 2 and the representative frame 3 are a still image and a zoom image of the same object, the histogram correlation value between them is high. If the sum [t2 + t3] of the time lengths t2 and t3 of two shots including the representative frames 2 and 3 exceeds a predetermined threshold value tref, both the representative frames 2 and 3 are determined as important frame candidates. The

  Returning to FIG. 3, in step S <b> 7, the information density distribution calculation unit 105 calculates how much the amount of information included in the image is unevenly distributed for the extracted important frame candidate. Thus, it is finally identified whether the frame is an important frame including an important object.

  FIG. 5 is a flowchart showing the calculation procedure of the information density distribution. In step S41, a central image obtained by trimming a certain area (for example, about 60%) in the central portion of the image from each representative frame is displayed. Extracted. In step S42, Harris feature amounts of the central image and the peripheral image are calculated for each important frame candidate. In step S43, the average brightness of the central image and the peripheral image is calculated for each important frame candidate. In step S44, the standard deviation (or variance) of the hues of the central image and the peripheral image is calculated for each important frame candidate. In step S45, the DCT coefficients of the central image and the peripheral image are calculated for each important frame candidate.

  Returning to FIG. 3, in step S <b> 8, for each important frame candidate, an evaluation value is calculated for each object type (here, “Person”, “Product”, and “Shop”). In this embodiment, in order to focus on the three elements (characters, objects, and background) constituting the video, the object types are classified into Person, Product, and Shop, but the present invention is limited to this. For example, the object type for artificial processing (telop or animation) may be added or replaced.

  The calculation of the evaluation value uses a representative color of the screen obtained from the feature amount extracted by the feature amount extraction unit 102, a color histogram distribution for each region when the screen is divided into 3x3, an edge distribution, and the like. The following parameters may be added as unique features for each type.

  (1) Person evaluation value calculation

  The probability is evaluated based on the area of the face area acquired by the face recognition and the barycentric position of the face area, and a higher evaluation value is given as the face appears larger in the center.

  For important objects other than a person, two different assumptions are used based on the size of the target. In other words, if an important object is a shooting target, it can be assumed that the object is presented to the viewer in an emphasized form in the screen frame, so the following evaluation is performed according to the size of the object, and the important object The probability Pp that is a Product is required, and the probability Ps that is a Shop is required.

  (2) Product reliability evaluation

  Here, it is assumed that the object is small relative to the shooting range of the camera. That is, the object is displayed near the center of the screen, and a background area exists at any or all of the top, bottom, left and right edges of the frame. If the object being photographed is an important object, the videographer configures the screen to emphasize the subject to be photographed.

  (2.1) Therefore, in the present embodiment, the frame screen is classified into the central region C and the peripheral amount O, the feature point number of the central region C is FC, and the feature point number of the peripheral region O is FO. The evaluation value Pp is increased.

  FC> FO × K

  (2.2) Furthermore, assuming that Dev (Hc) is the standard deviation of hue in the center area C of the frame screen and Dev (Ho) is the standard deviation of hue in the peripheral area O, the evaluation value Pp is increased if the following equation holds: Let

  Dev (Hc)> Dev (Ho) x L

  (2.3) Furthermore, when the average value of the luminance in the central area C of the frame screen is Ave (Vc) and the average value of the luminance in the peripheral area O is Ave (Vo), the evaluation value Pp is increased if the following equation holds: Let

  Ave (Vc)> Ave (Vo) × M

  However, K, L, and M are variables determined by the area ratio between the central region C and the peripheral region O of the image, respectively.

  The central region C is a region in which the upper, lower, left, and right edges of the screen are trimmed at a certain rate from the center of the frame screen. The region may be the central region C.

  (3) Evaluation of the certainty of Shop

  Here, based on the premise that the object is larger than the shooting range of the camera, the fact that the building is shot in a certain area or more in the image is used. However, unlike the case of Product, the building may be displayed on the entire image. In this case, there is not necessarily a difference between the central area and the peripheral area of the frame screen due to the number of feature points and hue dispersion. On the other hand, the area where the building is displayed is continuous, and the building includes many straight lines due to structural features. Therefore, in this embodiment, the following two feature amounts are used to evaluate shop.

  (3.1) Domain continuity

  The unit area of the frame screen is clustered according to the color distribution and frequency characteristics. For the clustering, for example, a method of grouping adjacent regions having similar features by repeating the process of replacing the feature amount of the target pixel with the average of the color feature amounts of each pixel by an average value shift method is used. The probability Ps that the image includes Shop as an important object is increased from the size and the center of gravity of the region thus collected.

  For the mean value shift method, see D. Comaniciu et al., “Mean Shift: A Robust Approach Toward Feature Space Analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol.24, no.5, May 2002. It is discussed in detail.

  (3.2) Edge distribution

  Edge extraction processing is performed on the binarized important frame candidates, and the probability Ps that the image contains Shop as an important object is calculated from the number of extracted vertical edges that have a certain length or more. increase.

  In the present embodiment, as described above, the values of Pp and Ps are calculated and compared as weighted sums of evaluation values, respectively, and the calculated feature values are, for example, classified by classifying processing by SVM (Support Vector Machine) or the like. By applying to the statistical processing, an evaluation value of Person-likeness, an evaluation value of Product-likeness, and an evaluation value of Shop-likeness are calculated for each important frame.

  In the present embodiment, for each of the “Person”, “Product”, and “Shop” object types, the data of each evaluation item is obtained in advance for samples of positive examples and negative examples of representative important frames. By constructing SVM as teacher data and applying the time length of shots extracted from each important frame candidate and various amounts of information to this, each important frame candidate has an important frame likelihood (likelihood). Calculated for each type.

  In step S9, the combination extraction unit 107a of the important frame determination unit 107 extracts important frame candidates from the set of important frame candidates in combinations according to the appearance order and the number of appearances of the important objects described in the meta information. The In step S10, the evaluation unit 107b adds the extracted evaluation values for each object type of each important frame candidate based on the types of each important object described in the meta information, and the sum is maximized. A combination of important frame candidates is determined as an important frame.

  FIG. 8 is a diagram showing a method of evaluating each important frame candidate in the important frame determining unit 107. Here, the method corresponds to this subsection based on the first subsection of the meta information shown in FIG. A case where four important frame candidates A, B, C, and D extracted from the selected video period are evaluated will be described as an example

  In this embodiment, the evaluation value of the “Shop” likelihood of the important frame candidate A is “0.95”, and the evaluation value of the “Product” likelihood is “0.21”. Similarly, the evaluation values of the important frame candidate B are “0.51” and “0.41”, the evaluation values of the important frame candidate C are “0.01” and “0.91”, respectively, and the evaluation values of the important frame candidate D are “0.34”, respectively. It is “0.85”.

  Here, according to the corresponding subsection, three important objects appear, and the appearance order is “Shop” → “Product” → “Product”. Therefore, all combinations ([A, B, C], [A, B, D], [A, C, D]... For selecting three candidates in this order from the four candidates A, B, C, D). ) Are extracted in step S9, and the total score is calculated in step S10 assuming that the first candidate is “Shop” and the second and third candidates are “Product”.

  For example, in the case of a combination of candidates [A, B, C], the total score is 2.27 (= 0.95 + 0.41 + 0.91). Similarly, for a combination of candidates [A, B, D], the total score is 2.21 (= 0.95 + 0.41 + 0.85). In this embodiment, the total score becomes the highest score 2.71 (= 0.95 + 0.91 + 0.85) when the candidate [A, C, D] is combined.

  In step S11, the candidate [A, C, D] whose total score is the highest score 2.71 is taken as a combination of important frames. Therefore, here, candidate A is an important frame corresponding to “ABC restaurant”, candidate C is an important frame corresponding to “special curry rice”, and candidate D is an important frame corresponding to “special hamburger” . The remaining candidate B is discarded as a misidentification candidate.

  The above processing is executed for all subsections. When extraction of all important frames from the program video is completed, in step S12, the information presenting unit 109 creates thumbnails of the important frames for each object type. The

  FIG. 9 is a diagram showing an example of a display method when thumbnail images of each important frame are displayed as a list. In this embodiment, tabs of “store”, “product”, and “performer” are prepared. When the “Store” tab is clicked by remote control operation or the like, thumbnail images of a number of important frames determined as “Shop” are displayed. Similarly, when the “Product” tab is clicked, thumbnail images of a number of important frames determined as “Product” are displayed. Similarly, when the “performer” tab is clicked, thumbnail images of many important frames determined as “Person” are displayed.

  Each thumbnail image is associated with information indicating the reproduction position of the image and text data described in the meta information. The viewing user selects a desired tab, and one thumbnail image displayed in a list is displayed. When one is clicked, text information associated with the thumbnail image is displayed at the bottom of the screen. Further, when the thumbnail image is double-clicked, the video is reproduced from the frame position of the thumbnail image.

  In the above embodiment, the case where an important frame is extracted from the video of an information program that introduces a city has been described as an example. Since it can be extracted with accuracy and presented to the viewer as a list, the viewer can quickly obtain information on the desired product. In this embodiment, text information is associated with the thumbnail image of the important frame. For example, if a search is performed by inputting the name of a favorite product or performer in text, the favorite product or performer's name is displayed. You can watch the video quickly.

  DESCRIPTION OF SYMBOLS 1 ... Important information extraction apparatus, 2 ... Video distribution apparatus, 3 ... Meta information provision apparatus, 4 ... Television / monitor, 101 ... Video reception part, 102 ... Feature-value extraction part, 103 ... Shot division | segmentation part, 104 ... Appearance time length Detection unit, 105 ... In-screen information density distribution calculation unit, 106 ... Important frame candidate evaluation unit, 107 ... Important frame determination unit, 108 ... Database, 109 ... Information presentation unit, 110 ... Meta information acquisition unit

Claims (5)

In an important information extraction device that analyzes video and extracts important information,
Means for extracting features from the video;
Means for dividing the video into a plurality of shots based on the feature amount;
A database that stores representative features of important frames in which important objects appear;
Means for extracting a representative frame from each shot;
Means for extracting important frame candidates from the representative frame based on the feature amount;
Means for obtaining meta information of the video;
Means for comparing each feature amount of the important frame candidate with a representative feature amount of the important frame to evaluate each important frame candidate;
Means for extracting an important frame from the important frame candidates based on the meta information and the evaluation value of each important frame candidate,
The database stores unique features for each object type,
The evaluation means calculates the evaluation value for each object type by comparing the feature amount of each important frame candidate with each unique feature amount stored for each type of the object,
The meta information describes the number of appearances of important objects, the order of appearance, and the object type,
The means for extracting the important frame includes:
Means for extracting important frame candidates from a set of important frame candidates in combination according to the appearance order and the number of appearances of the important objects;
Means for adding the extracted evaluation value for each object type of each important frame candidate based on the type of each important object, and extracting the important frame candidate of the combination that maximizes the sum as an important frame. An important information extraction device characterized by that.
2. The important information extracting apparatus according to claim 1 , wherein the means for extracting the representative frame extracts a representative frame from a shot of a still image section. 3. The important information extracting apparatus according to claim 1, wherein the means for extracting the representative frame extracts a representative frame from a shot in the moving object follow section. Means for displaying the important frames as thumbnails in a list;
Each thumbnail image, important information extraction device according to any one of claims 1 to 3 information for reproducing the video from the position of each key frame, characterized in that it tied. In the important information extraction method that analyzes video and extracts important information,
A procedure for extracting features from the video;
Dividing a video into a plurality of shots based on the feature amount;
A procedure for extracting a representative frame from each shot;
A procedure for extracting an important frame candidate in which an important object appears based on the feature amount from the representative frame;
Obtaining the meta information of the video;
A step of evaluating each important frame candidate by comparing the feature quantity of the important frame candidate with a representative feature quantity of the important frame;
Extracting an important frame from the important frame candidates based on the meta information and the evaluation value of each important frame candidate,
The database stores unique features for each object type,
In the step of evaluating, the feature value of each important frame candidate is compared with each unique feature value stored for each type of object, and an evaluation value is calculated for each object type.
The meta information describes the number of appearances of important objects, the order of appearance, and the object type,
The procedure for extracting the important frame includes:
A procedure for extracting an important frame candidate from a set of important frame candidates in a combination according to the appearance order and the number of appearances of the important object;
A step of adding an evaluation value for each object type of each extracted important frame candidate based on the type of each important object, and extracting an important frame candidate of a combination that maximizes the sum as an important frame. The important information extraction method characterized by this.