JP4369308B2 - Representative image selection device, representative image selection method, and representative image selection program - Google Patents

Description

  The present invention relates to a technique for giving an index for using a moving image to a moving image obtained by photographing an object with a camera, and in particular, determines a target object from the moving image and determines a frame that can best separate the target object and the background. The present invention relates to a representative image selection device, a representative image selection method, and a representative image selection program that are selected to be representative images.

  The following are examples of conventional techniques for indexing video depending on the application, such as video search, editing, and quick viewing.

  The following Patent Document 1 “Method and apparatus for creating a list of representative images of video” describes a technique for selecting representative images of a video and listing the durations and locations of scenes of the representative images. . The method for selecting the representative image is manual or automatic, and in the case of automatic, frames are set at regular intervals or scene change is detected. In this case, even if a scene change is detected automatically, the frame of interest at the instant of the change does not necessarily include the object of interest clearly, and the accurate representative image is not necessarily obtained.

  The technique described in Patent Document 2 “Shot Detection Method and Representative Image Recording / Display Device” is also a technique for detecting changes in shots, collecting videos into appropriate units, and extracting representative images for each unit. is there. The image is selected based on zoom, pan, color histogram, frequency, and the like. Although a scene change can be detected, this is not always suitable for a representative image.

  The technique described in Patent Document 3 “Representative image display method, representative image display device and moving image search device using this device” extracts the contour of the moving object from each frame included in the scene. , A technique in which a composite image obtained by superimposing the contours is used as a representative image of the scene. However, as will be described later in the present invention, it is possible to evaluate the adaptability of each frame as a representative image, and to give an index for selecting a representative image according to a user's application to each frame of video. I can't.

The following Non-Patent Document 1 and Non-Patent Document 2 are contour extraction techniques, which can be applied to a target object contour extraction unit in the present invention described later. You cannot evaluate or index each frame of the video.
Japanese Patent Laid-Open No. 7-79404 JP 7-236115 A Japanese Patent Laid-Open No. 9-214666 Sakagami, Yamamoto, "Dynamic Net Model Active Net and its Application to Region Extraction", Television Society Journal, 1991, vol.45, No.10, p.1155-1163 Sakaida, Naemura, Kinji, “Extraction of moving image objects using background difference method and region growing method by spatio-temporal watershed”, IEICE Transactions DII, 2001, Vol.J84-DII, No.12 p.2541-2555

  As described above, in the prior art, as a method for selecting a representative image from a moving image, a scene change is detected and a scene switching frame is used as the representative image. However, a frame at the transition of a scene defined by changes in camera pan, zoom, color, etc. does not necessarily include the subject that is most interested in the scene.

  As the representative image, it is desirable to select an image that allows the content of the entire video to be grasped as a video digest even if only that image is viewed. If the original video is an image of a target object, the target object needs to be included. In addition, it is desirable that the user can instruct the selection method of the representative image because which of the frames including the target object is the representative image depends on the subsequent use.

  The present invention has been made based on the circumstances as described above, and can select a frame in which the object of interest is most clearly reflected without blurring as a representative image, and also according to the usage application of the user. An object is to enable selection of a representative image.

In order to achieve the above object, the present invention inputs a moving image and divides a region including a target object region, which is one or a plurality of moving regions, from each frame of the moving image into small regions. For each pixel in the small region, a feature amount including at least a feature amount based on motion is extracted, and using the extracted feature amount, each pixel in the small region is a target object region and a background region other than that region Classify which class belongs to. Thereafter, the classes having similar feature distribution shapes between the adjacent small areas are set as the same class, and the boundaries of the classes are connected between the small areas, thereby extracting the contour of the target object area. The average value of the distribution of the feature values is obtained for all the pixels in each of the target object region and the background region extracted in this way, and the distance between the two average values is used as the representative image of each frame. The number of points shown.

Further, as the score of the degree of matching, in addition to the distance, the degree of coincidence between the shape of the target object region and the predetermined template, the size of the target object region, or the length of the contour of the target object region Any one or more of a value compared with a reference value or a value obtained by comparing a contour shape of the object region of interest with a predetermined reference value may be obtained. Then, the score of the degree of fitness and information on the method for calculating the score can be assigned to each frame of the moving image as an index, and the indexed moving image can be output .

  The extraction of the contour of the target object region is performed as follows. First, the input moving image is divided into small regions including some or all of the one or more objects of interest for each frame. Then, one or a plurality of feature amounts are extracted for each pixel in each divided small area. The feature amount is, for example, luminance, color, optical flow, and the like. When extracting feature quantities, for example, whether to extract feature quantities based on motion using the processing target frame and neighboring frames, or to extract feature quantities not based on motion using the processing target frame, Both feature quantities are extracted.

  For each divided small area, when classifying which pixel each pixel belongs to, using all or part of the feature quantity, the likelihood indicating the separation of the classification is the highest. Classify that will show the value Next, for each adjacent small region, using the feature amount, the regions classified into the same class are associated, and based on this, the regions classified into the same class are joined together, The outline of the object of interest is extracted and separated from the background.

  By using the present invention as the representative image, the original image of the frame that can most clearly separate the object of interest from the background is used as the representative image. Can be selected. The representative image obtained in this way is easy to use because the object of interest is clearly visible.

  In addition, the contour extraction method can be applied when the foreground and the background are both moving, so it can be used for images where the object of interest and the background are moving while moving with a mobile camera or mobile device. Applicable.

  The apparatus of the present invention can be realized by a computer and a program, and the program can be recorded on a recording medium or provided through a network.

  According to the present invention, it is possible to extract a target object from a video, evaluate the suitability of a frame in the video as a representative image according to a criterion based on the target object, and add the suitability to the frame as an index. . This makes it possible to select a representative image according to the usage application of the user.

[First Embodiment]
First, a first embodiment of the present invention will be described with reference to FIG. FIG. 1 is a diagram illustrating a configuration example of a representative image selection device. FIG. 2 is a flowchart showing the processing. The representative image selection device 1 includes a moving image input unit 11, a target object contour extraction unit 12, a representative image matching degree calculation unit 13, an indexing unit 14, a moving image output unit 15, and a contour evaluation reference storage unit 16. Reference numeral 160 denotes a contour evaluation reference stored in the contour evaluation reference storage unit 16.

  The moving image input unit 11 inputs a moving image (step S1 in FIG. 2), and sequentially sends the moving image to the target object contour extracting unit 12 for each frame of the moving image. Since the following processing is performed for each frame, the input moving image may be a moving image captured by a video camera or a Motion JPEG format moving image captured by a continuous shooting function of a digital camera.

  The target object contour extracting unit 12 extracts the contour of the target object for each time-series frame of the image as shown in FIG. 1A (step S2 in FIG. 2). The target object area is a moving area by default. A moving region can be detected by a method such as performing local template matching between adjacent frames (for example, see “Basics of Digital Image Input” Corona, Chapter 3, pp.137).

  Using this region as the initial value of the target object region and the other region as the initial value of the background region, contour extraction of the target object region is performed. Examples of the contour extraction technique include a method using a dynamic contour model as described in Non-Patent Document 1 above.

  The contour evaluation reference storage unit 16 is means for storing the contour evaluation reference 160. The contour evaluation standard 160 is a standard prepared in advance for evaluating whether each frame can be a representative image, and is a standard for evaluating the output of the target object contour extraction unit 12. The contour evaluation standard 160 may be input from an external device.

  The representative image suitability calculating unit 13 calculates the suitability of each frame as a representative image based on the output of the target object contour extracting unit 12 and the contour evaluation reference 160 (step S3 in FIG. 2).

  For example, the representative image suitability calculation unit 13 calculates the final likelihood (goodness of fit of the feature amount distribution between the target object and the background), the coordinate value of each pixel, calculated by the target object contour extraction unit 12. , A flag indicating any one of the background region / target object region, the coordinate value of the pixel on the contour, and a data array in which the feature amount calculated in the region division process is associated with each other. Based on this, the adaptability of each frame as a representative image is calculated as a score. The goodness of fit indicates the closeness to the ideal representative image, and the score is obtained by substituting features into an evaluation function based on the distance from the image feature to the template in order to evaluate the goodness of fit. Value.

  For example, the index assigning unit 14 evaluates the image of each frame input from the representative image suitability calculating unit 13, the contour of the target object extracted from the frame, the degree of separation of the contour from the background, and the contour. The evaluation method and the resulting score are written in a data structure as shown in FIG. 3 using the evaluation method and the score as an index, thereby giving an index to the image of each frame (step S4 in FIG. 2). The moving image output unit 15 outputs the indexed moving image (step S5 in FIG. 2).

  By assigning an index to the image of each frame, for example, the user can search only the frames indicated by the hatched portion in FIG. .

  Further, the index assigning unit 14 evaluates the image of each frame input from the representative image suitability calculating unit 13, the contour of the target object extracted from the frame, the degree of separation of the contour from the background, and the contour. The evaluation method, the score of the result, and the condition for selecting the representative image input by the representative image condition input unit described later are written in the data structure as shown in FIG. 11 using the evaluation method, the score, and the condition as an index. In some cases.

  An example of processing of the representative image suitability calculation unit 13 using the contour evaluation reference 160 will be described with reference to FIGS. As an evaluation method based on the contour evaluation standard 160, (a) a method for evaluating the shape and size of the target object region, (b) a method for quantitatively evaluating the properties of the contour line, and (c) a target object region and a background region. Whether or not the degree of separation, the temporal change in the degree of separation, and the overall likelihood (goodness of fit of the feature quantity distribution between the target object and the background), which is the output of the target object contour extraction unit 12, is greater than or equal to a certain value. There are a method of evaluation, and (d) a method of combining a plurality of the above evaluation methods.

(A) Evaluation of shape and size of target object region FIGS. 4A and 4B are diagrams illustrating an example of evaluation of the shape and size of the target object region, which is the output of the target object contour extraction unit 12. It is. As shown in these figures, for example, an array in which a coordinate value of each pixel that is an output of the target object outline extraction unit 12 and a flag indicating whether the pixel belongs to a background area or a target object area corresponds to each other. From this, a binary image with a background area of “0” and a target object area of “1” is created, and template matching (for example, “Introduction to Computer Image Processing”, Research Institute Publishing, Chapter 5, pp. 149) is performed. . When a binary image template with the shape inside “1” and the background “0” is superimposed on the created binary image while changing its size and position, the overlapping area is maximized The ratio of the area of the overlapping portion to the area inside the shape is defined as the degree of coincidence.

  The score may be the degree of coincidence and the magnitude at that time, or may be a value calculated from them using a predetermined function. In addition, an evaluation function for features such as contour unevenness may be used (for example, see “Introduction to Computer Image Processing”, Soken Publishing, Chapter 3, 3.4 Graphic Shape Features). The method is not limited to the above example, and any method that can quantitatively evaluate the shape and size of the object region of interest may be used.

(B) Evaluation of outline property FIG. 4C is a diagram illustrating an example of evaluation of the outline property. Evaluate the length of the contour, whether the contour is a closed curve, the position of the end points, the smoothness of the contour, etc. The length of the contour is obtained by boundary line tracking or the like (for example, see “Image Processing Engineering Basics”, Kyoritsu Publishing Co., Ltd., Chapter 7, pp. 126).

  Whether it is a closed curve is determined by whether the start and end points match. As shown in FIG. 4C, the position of the end point follows a pixel on the boundary (contour line) between the target object region and the background region, and there is a pixel that takes the maximum or minimum value of x or y in the image region. Judge whether or not. The smoothness of the contour is obtained by an evaluation function using the primary difference or the secondary difference of the contour.

  Scores are given as a result of comparing these items with the reference values of these items. An appropriate function with each item as a parameter is prepared and calculated using this function. The method is not limited to the above example, and any method that can quantitatively evaluate the properties of the contour line may be used.

(C) Evaluation of the degree of separation between the target object region and the background region, the temporal change in the degree of separation, and whether or not the overall likelihood that is the output of the target object contour extraction unit 12 is greater than or equal to a certain value. It is a figure explaining the example of evaluation. It evaluates whether the degree of separation between the target object region and the background region, the temporal change in the degree of separation, and the goodness of fit of the feature quantity distribution between the target object and the background are above a certain value. Here, the degree of separation indicates the degree of overlap in the feature space between the distribution of feature values of the pixels belonging to the object region of interest and the distribution of feature values of the pixels belonging to the background region. The separation degree is high when the separation degree is low and the overlap is small. The score is defined as the degree of separation of the distribution of feature amounts of the target object area and the background area. In the example shown in FIG. 5B, the average of the feature amount distribution of the pixels in the background region and the average of the feature amount distribution of the pixels in the object region of interest in the image shown in FIG. The distance between the average values is used as the degree of separation. As the degree of separation, Mahalanobis distance (see, for example, “Intuitive Pattern Recognition” Ohm, pp. 121-122) can also be used.

  Also, it may be evaluated whether the goodness of fit (likelihood) of the feature quantity distribution between the object of interest and the background is a certain value or more. In addition, a function using them as input may be prepared. Moreover, you may evaluate how likelihood changes with time. When a sudden change in likelihood occurs between frames, such as likelihood small → large or likelihood large → small, this is detected. The score may be the amount of change in likelihood, or a function with the input as an input may be prepared. The method is not limited to the above example, and any method that can quantitatively evaluate the degree of separation between the object region of interest and the background region and the temporal change thereof may be used.

(D) Combination of the above evaluations In the embodiment of the present invention, a plurality of combinations of evaluations as in the above example may be performed. In addition, which criterion is used can be selected by the user according to the subsequent use.

[Second Embodiment]
Next, a second embodiment of the present invention will be described with reference to FIG. FIG. 6 is a diagram illustrating a configuration example of a representative image selection device according to the second embodiment. The representative image selection device 2 has a configuration in which a target object instruction unit 21 is added to the configuration of the representative image selection device 1 shown in FIG. The configuration other than the object-of-interest instruction unit 21 is the same as the configuration included in the representative image selection device 1 shown in FIG.

  The target object instruction unit 21 is a means for the user to specify a target object region. In the first embodiment, an example has been described in which the object-of-interest contour extraction unit 12 regards all moving regions as objects of interest by default. In the second embodiment, as shown in FIG. 6, an object-of-interest instruction unit 21 is provided, and a plurality of determination methods are provided. After the user is instructed by the object-of-object instruction unit 21, image processing is performed. By doing this, the target object can be determined.

  Examples of the instruction method include the following. (1) Select a moving region having a certain area or more, (2) Select a region having an absolute value of a motion vector having a certain value or more, (3) Pre-registered templates among the moving regions Template matching is performed to select a highly correlated area, and (4) the user selects a target object area with a mouse or the like by using a GUI (Graphical User Interface).

[Third Embodiment]
Next, a third embodiment of the present invention will be described. FIG. 7 is a diagram illustrating a configuration example of an attention object contour extraction unit according to the third embodiment of the present invention. The third embodiment is an example in which the target object contour extracting unit 12 shown in FIG. 6 is used as an example of the target object contour extracting unit 12 in the representative image selecting devices 1 and 2 in the first and second embodiments. It is.

  The target object contour extracting unit 12 includes a small region setting unit 121, a feature vector extracting unit 122, a small region dividing unit 123, and a region extracting unit 124, and a time series of images input from the moving image input unit 11. For each frame, the contour of the object of interest is obtained.

  First, the small area setting unit 121 divides an area including the target object into a plurality of small areas. For example, when there is an instruction input from the object-of-interest instruction unit 21, one or more small areas including the entire designated area are set. For example, as shown in FIG. 8, the image area is divided into N (N ≧ 2) small areas, and information that sets pixel coordinates and pixel values in units of these small areas is held. .

  The feature vector extraction unit 122 obtains a feature amount for the pixels in the small area. The feature amount is obtained for all pixels by luminance, color, optical flow, and the like. And the information which makes the coordinate of a pixel and the feature-value a pair is hold | maintained.

  Next, the small area dividing unit 123 obtains the probability that each pixel is included in each area based on the distribution of the feature amounts of the target object area and the background area in each small area, and seems to be the best fit. In other words, the target object region and the background region in the small region are divided again so that the likelihood becomes maximum, and the likelihood at that time is calculated.

A well-known general definition of likelihood is briefly described. When the probability density function is given by f (y, θ), the random variables Y _1, Y _2, ..., observed value y _1, y ₂ of Y _n, ..., with respect to y _n,
L = Π _i f (y _i , θ)
Is regarded as a function of θ, and this is called a likelihood function (see “Speech Recognition by Probability Model” (The Institute of Electronics, Information and Communication Engineers), p. 51). Various values that L can take are called likelihoods. That is, when assuming a probability model f (y, theta), observed value y _1, y _2, ..., the probability of y _n is obtained, the probability model f (y, theta) of observations to apply the It is a value that evaluates goodness.

  In the present embodiment, the object region and the background region are accurately separated by repeating the process of re-dividing the small region so that the likelihood becomes maximum. The number of repetitions is until the predetermined number of times is reached or the likelihood becomes equal to or greater than a predetermined value.

The processing of the small area dividing unit 123 will be described in more detail. The small region dividing unit 123 receives information that sets the coordinates of the pixel and the feature amount of each small region C _i (1 ≦ i ≦ N) from the feature vector extracting unit 122, and each feature amount for each small region. Using (F _{1 to} F _M ), processing for dividing each pixel into two classes is performed. Specifically, when using the feature value, the class to which the pixel belongs, the probability of belonging to the class, and the likelihood of the entire small region when classifying using the feature value ( A process of calculating a value indicating the degree of separation of classification: a higher value indicates that the distribution of the feature amount is better applied to the model). Then, information obtained by adding the estimated class for each pixel, the probability of belonging to the class, and the likelihood of the entire small area to the information obtained by combining the coordinates of the pixel and the feature amount is stored.

A procedure for dividing the small area C _i into two classes of area Z ₁ and area Z ₂ will be described. This division is performed based on the feature amount of the pixel. In the small area, the feature value is almost uniform in each of the object part and the background part, but it is usually considered to be distributed with some dispersion around a certain average value like a Gaussian distribution.

  A statistical method for estimating this distribution is effective in region segmentation. For example, assuming a Gaussian distribution, maximum likelihood estimation is performed to estimate the shape and parameter values of the region division using the mean and variance as parameters. For a plurality of small regions including the boundary between the object candidate region F ′ and the background candidate region B ′, it is finally divided into regions by repeatedly estimating parameters such as feature value average values and variance values and calculating likelihoods. Estimate the shape of

  As an algorithm, for example, an EM algorithm that repeats parameter estimation (E (Expectation) step) and maximum likelihood estimation (M (Maximization) step) (reference: information processing, vol.37, No.1, pp. 43-46, Jan, 1996).

The EM algorithm is an algorithm for the iterative search continue to iterative improvement solutions like the steepest descent method to determine the exact area The algorithm 2 between the region Z ₁ and region Z ₂ subregions C _i Divide into two classes. In the EM algorithm, parameters are estimated in parallel for all feature quantities in E step, and the total likelihood (sum of Q _i (F _m ) shown below) of all feature quantities is evaluated in M step. Perform maximum likelihood estimation as a function.

For example, the processing result obtained by dividing the i-th small region by M feature amounts F _{1 to} F _M is as follows:
(A) Feature quantities F _{1 to} F _M obtained for the k-th pixel: F _{1k to} F _Mk
(B) Information indicating whether the k-th pixel is classified into the region Z _i1 or the region Z _i2 by each feature amount m: Z _mk
(C) Probability that the k-th pixel belongs to the classified region Z _mk : P _mk (Z _mk )
(D) Likelihood in the entire small region C _i for each feature quantity: Q _i (F ₁ ) to Q _i (F _M )
However, Q _i (F _m ) = Σ _k log P _mk (Z _mk )
Is obtained as a set of values.

  Finally, the region extraction unit 124 converts the attention object region and the background region included in one small region, and the attention object region and the background region included in the adjacent small region into the distribution of the feature amount of each region. Integrate by considering close objects as the same area. Here, the area is merged. Two adjacent small areas connect the border lines on the adjacent sides, and the same side of the connected border line is the same class (in this case, two classes, the object area and the background area) That it was decided to belong to.

By integrating the small areas as follows, the object area of the entire image and the background area of the entire image are obtained. The integration process of each small area will be specifically described. As a rule of integration, small areas C _i neighboring are _two, the C _j, the two classes Z _i1 / Z _i2 in the small area C _i, two classes Z in the small area C _{j j1} / Determine the correspondence between classes with Z _j2 . And, the boundary line connecting classes is connected without contradiction.

In the integration process, first, with the distribution of the feature (e.g., average value), and two classes Z _i1 / Z _i2 in small areas C _i shown in FIG. 9 (A), the two in the small area C _j The correspondence between classes with class Z _j1 / Z _j2 is determined. For example, the correspondence relationship that Z _i1 and Z _j1 belong to the same class and Z _i2 and Z _j2 belong to the same class is determined.

9A and 9B, the small area C _i is divided into classes Z _i1 and Z _i2, and the small area C _j adjacent to C _i is divided into classes Z _j1 and Z _j2 . It is assumed that it is divided. Based on the distribution of the feature quantity of each class in each small area shown in the lower part of FIGS. 9A and 9B, the one with the closest feature quantity distribution shape is labeled as the same class. Thereafter, as shown in FIG. 9C, the boundary lines connecting the classes are connected without contradiction. The closeness of the distribution of feature values can be determined by comparing statistics such as average values or by Mahalanobis distance.

  As a result of the above processing, the target object contour extraction unit 12 calculates the time-series image, the contour information associated with each frame of the image, and the degree of separation from the background of the target object obtained at the time of contour extraction. Output.

[Fourth Embodiment]
Next, a fourth embodiment of the present invention will be described with reference to FIG. A representative image selection device 3 shown in FIG. 10 is obtained by adding a representative image condition input unit 31 and a representative image condition determination unit 32 to the configuration of the representative image selection device 2 shown in FIG. The configuration other than the representative image condition input unit 31 and the representative image condition determination unit 32 is the same as the configuration included in the representative image selection device 2.

  The representative image condition input unit 31 is a means for allowing the user to selectively input representative image selection conditions different from the evaluation criterion based on the contour given by the contour evaluation criterion 160. Examples of representative image selection conditions include specifying the background color and whether or not a specific object (such as a person, face, character, or telop) is included in the background. The user can select one or more of these conditions.

  The representative image condition determining unit 32 determines the condition of the frame input from the moving image input unit 11 according to the condition input by the representative image condition input unit 31. The representative image condition determination unit 32 has a determination processing function corresponding to the condition of the representative image condition input unit 31 and performs condition determination.

  For example, as a determination process corresponding to the condition of selecting a specific background color, a color histogram is obtained and compared with a color histogram of a specific background color. As a result of the determination by the representative image condition determination unit 32, when a selection condition such as “background color is blue” is detected for the current frame of interest, the selection condition is notified to the index assigning unit. The representative image condition determination unit 32 performs matching with a template of a specific object as a determination process corresponding to the condition that the specific object is included. As a result, when a condition for selection such as “a face is included” is detected for the current frame of interest, the selection condition is notified to the index assigning unit 14.

  This condition determination may be performed for all the frames of the input moving image, or may be performed only for the frames for which the score calculated by the representative image fitness calculation unit 13 is equal to or greater than a predetermined value. May be.

  The index assigning unit 14 evaluates the image of each frame input from the representative image suitability calculating unit 13, the contour of the target object extracted from the frame, the degree of separation of the contour from the background, and the contour. The method, the score of the result, and the condition for selecting the representative image input by the representative image condition input unit 31 are written in the data structure as shown in FIG. 11 using the evaluation method, the score, and the condition as an index.

[Fifth Embodiment]
Next, a fifth embodiment of the present invention will be described with reference to FIG. The representative image selection device 4 shown in FIG. 12 adopts a configuration in which a scene separation unit 41 and a scene combination unit 42 are added to the configuration of the representative image selection device 1 of FIG.

  The configuration other than the scene separation unit 41 and the scene combination unit 42 is the same as the configuration included in the representative image selection device 1. The scene separation unit 41 is a means for receiving, for example, a long-time video as shown in FIG. 13A and dividing it into short scenes such as scene 1 and scene 2 shown in FIG. It can be replaced with a conventional method of scene detection method from video (for example, the method described in Patent Document 1, Patent Document 2, and Patent Document 3). Other than this, it may be divided by a method such as dividing at a certain time.

  When the index assigning unit 14 assigns an index to the frame included in each of the scene 1 and the scene 2, and the frame indicated by the hatched portion in FIG. 13C is a representative image of each scene, the scene combining unit 42 As shown in FIG. 13D, scenes 1 and 2 which are divided and selected as representative images are combined in the original time order.

It is a figure which shows the structural example of a representative image selection apparatus. It is a processing flowchart of a 1st embodiment of the present invention. It is a figure which shows the example of the data structure of an index. It is a figure which shows the example of a process of the representative image adaptation calculation part using an outline evaluation reference | standard. It is a figure which shows the example of a process of the representative image adaptation calculation part using an outline evaluation reference | standard. It is a figure which shows the structural example of a representative image selection apparatus. It is a figure which shows the structural example of an attention object outline extraction part. It is a figure explaining the image division by a small area setting part. It is a figure explaining the integration process of each small area | region. It is a figure which shows the structural example of a representative image selection apparatus. It is a figure which shows the example of the data structure of an index. It is a figure which shows the structural example of a representative image selection apparatus. It is a figure explaining the 5th Embodiment of this invention.

Explanation of symbols

1, 2, 3, 4 Representative image selection device 11 Moving image input unit 12 Target object contour extraction unit 13 Representative image fitness calculation unit 14 Index assignment unit 15 Moving image output unit 16 Contour evaluation reference storage unit 21 Target object instruction unit 31 Representative image condition input unit 32 Representative image condition determination unit 41 Scene separation unit 42 Scene combination unit 121 Small region setting unit 122 Feature vector extraction unit 123 Small region division unit 124 Region extraction unit 160 Contour evaluation standard

Claims (11)

A representative image selection device for selecting a representative image from a moving image,
A moving image input means for inputting a moving image;
A small area setting means for dividing an area including an object area of interest which is one or a plurality of moving areas from each frame of the moving image into small areas;
Feature vector extraction means for extracting a feature quantity including at least a feature quantity based on motion for each pixel in the small area;
Sub-region dividing means for classifying each pixel in the sub-region using the feature amount to belong to a class of a target object region or a background region other than the target object region;
Region extracting means for extracting the contour of the object region of interest by connecting classes that are close to each other in the same class and having the same class between the adjacent small regions and connecting the boundaries of the classes between the small regions When,
The average value of the distribution of the feature amount is obtained for all the pixels in each of the target object region and the background region extracted by the region extracting means, and the distance between both average values is used as a representative image of each frame. representative image selection apparatus, comprising a representative image matching degree calculation means to score indicating a. The representative image selection device according to claim 1,
A representative image selection device comprising: an object-of-interest instruction means for designating an object-of-interest area that is one or a plurality of moving areas by input to the small area setting means . In the representative image selection device according to claim 1 or 2,
The representative image suitability calculating means includes:
As the score of the fitness, in addition to the distance, the degree of coincidence between the shape of the target object region extracted by the region extraction unit and a predetermined template, the size of the target object region, or the target object region Obtaining one or more of a value obtained by comparing the length of the contour and a predetermined reference value, or a value obtained by comparing the shape of the contour of the object region of interest and a predetermined reference value;
Furthermore, index assigning means for assigning information about the score of the fitness and the method for calculating the score to the frame as an index;
A representative image selection device comprising: a moving image output unit that outputs the indexed moving image . In the representative image selection device according to claim 3 ,
Representative image condition input means for inputting conditions for selecting a representative image;
Representative image condition determining means for performing condition determination on the moving image input by the moving image input means using a condition for selecting the input representative image;
The index assigning means receives the condition determination result by the representative image condition determining means, and uses the moving image as an index including information on the degree of matching of the frame of the moving image as a representative image and the condition for selecting the representative image. The representative image selection device is characterized in that the image is added to the frame. In the representative image selection device according to claim 3 or 4 ,
Scene separation means for dividing the moving image into a plurality of scenes;
The index assigning means assigns the index to a frame in each scene for each of the divided scenes. A representative image selection method in which a computer selects a representative image from moving images,
A moving image input process for inputting a moving image;
A small region setting process for dividing a region including a target object region, which is one or a plurality of moving regions, from each frame of the moving image into small regions;
A feature vector extraction process for extracting a feature amount including at least a feature amount based on motion for each pixel in the small region;
A small area dividing process of classifying each pixel in the small area to the class of which of the object object area and the background area, which is the other area, using the feature amount;
A region extraction process for extracting the contour of the object region of interest by joining classes that are close to each other in the same feature class between adjacent small regions and connecting the boundaries of the classes between the small regions When,
The average value of the distribution of the feature amount is obtained for all pixels in each of the target object region and the background region extracted in the region extraction process, and the distance between the two average values is used as a representative image of each frame. representative image selection method characterized by having a representative image matching degree calculation step of a number indicating the. In the representative image selection method according to claim 6,
A representative image selection method comprising: an object-of-interest instruction process that designates an object-of-interest area that is one or a plurality of moving areas by input with respect to the small area setting process . In the representative image selection method according to claim 6 or 7,
The representative image fitness calculation process includes:
As the score of the fitness, in addition to the distance, the degree of coincidence between the shape of the target object region extracted in the region extraction process and a predetermined template, the size of the target object region, or the target object region Obtaining one or more of a value obtained by comparing the length of the contour and a predetermined reference value, or a value obtained by comparing the shape of the contour of the object region of interest and a predetermined reference value;
And an indexing process for assigning information about the score of the fitness and a method for calculating the score to the frame as an index,
A representative image selection method comprising: a moving image output process of outputting the indexed moving image . In the representative image selection method according to claim 8 ,
A representative image condition input process for inputting a condition for selecting a representative image;
And a representative image condition determination process of performing condition determination on the moving image inputted by the moving image input step using the conditions for selecting a representative image that is the input,
The indexing process receives a condition determination result from the representative image condition determination process, and includes an index including information on a degree of matching of the frame of the moving image as a representative image and a condition for selecting the representative image. A representative image selecting method, characterized by being applied to a frame of In the representative image selection method according to claim 8 or 9 ,
Has a scene separation process of dividing the moving picture into a plurality of scenes,
In the index assigning process, the index is assigned to a frame in each scene for each of the divided scenes.   A representative image selection program for causing a computer to execute the representative image selection method according to any one of claims 6 to 10.

Images