JPH10224736A - Moving image index generating method and generator - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily grasp contents of a moving image and to easily retrieve a desired scene by selecting a frame of a moving image whose picked-up contents are meaningful as an index. SOLUTION: At first a frame being a still image corresponding to each time is extracted from a received moving image (step ST1). Then a dictionary relating to a desired video image is used to calculate an evaluation value of a partial area of the frame included in part or all of the moving image according to a predetermined calculation method (step ST2). Then an evaluation value of the frame is calculated based on the evaluation values calculated for all the areas included in the frame (step ST3). Furthermore, an image of a specific frame among the frames is selected as an index image for the display based on the evaluation value (step ST4).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and an apparatus for selecting, as an index, a frame image representing the contents of a part or the whole of a moving image.

[0002]

2. Description of the Related Art A moving image is composed of a sequence of still images (frames) corresponding to each time, and expresses movement by sequentially changing the displayed frame as time elapses. Unlike a still image, a moving image has a time axis, so it is difficult to quickly grasp the entire contents.

When all the frames are displayed on a single screen when grasping the contents of the whole moving image or searching for a desired part of the moving image, the efficiency is high. Since the number of frames included in a moving image is extremely large, displaying all frames is usually not efficient. For example, NTSC (National Television System Committee), one of the television standardization systems
In the system, about 30 frames are included per second, and 100,000 or more frames are included in one hour, and it is impossible to display these on one screen. Further, even if several frames are formed as one page and displayed on one screen, the number of pages increases, and eventually, it is more efficient to fast-forward and reproduce the moving image. .

On the other hand, since the display of all frames contains a lot of redundant information, the number of displayed frames can be reduced, and the efficiency of content grasp and retrieval can be improved. The frame selected for display at this time is called an index. For example, the index can be selected such as one frame per second or one frame per 30 seconds. At this time, if an index image is simply selected at a rate of one per several frames, an easy-to-understand display is not always possible. For example,
Some frames are important in terms of content, but were not selected for indexing because of their short display time, and conversely, frames that were not semantically significant but were still for a long time existed, and so were indexed in the same way. The inconvenience of becoming an image occurs.

In order to solve such inconveniences, a method has been considered in which a discontinuous frame called a cut is detected, and one index image is selected from frames included between the cuts. . By determining a frame section for selecting an index video in this way, it is possible to prevent a similar video from being selected as an index to some extent. As a technique for automatically detecting a cut, there are techniques described in JP-A-1-68084, JP-A-3-214364, JP-A-4-219828, and the like. Usually, after a certain frame is detected as a cut, the first frame after the cut is selected as an index.

[0006]

The method of selecting one index from the frames between cuts as described above is effective for determining a section for selecting an index. It remains as an unsolved problem which of the plurality of frames included between the two should be selected as the index video. That is, there has been a problem that no method has been provided for selecting one frame that reflects the contents from a predetermined part or the whole of the moving image.

The present invention has been made to solve the above problems.
By selecting a frame having a meaning in the content being photographed as an index, a moving image index generating method and a generating apparatus that can easily grasp the contents of the moving image and easily search for a desired scene when the moving image is displayed as an index. It is intended to provide.

[0008]

According to a first aspect of the present invention, there is provided a moving image index generating method for extracting a frame which is a still image corresponding to each time from an input moving image. Calculating the evaluation value of the partial region of the frame included in a part or all of the moving image by a predetermined calculation method using a dictionary relating to a desired video; and Calculating a frame evaluation value from an evaluation value calculated for the frame, and selecting an image of a specific frame from among the frames as an index image for display based on the evaluation value. It is characterized by.

According to a second aspect of the present invention, in the moving image index generating method according to the first aspect, the frame included in a part or the whole of the moving image is
Cutting out the moving object from the background;
Calculating an evaluation value by a predetermined calculation method for each of the cut-out objects; calculating an evaluation value of a frame from evaluation values of objects included in the same frame; and Selecting and outputting a specific frame as an index for display.

According to a third aspect of the present invention, in the moving image index generating method according to the first aspect, a camera which captures the moving image for a frame included in a part or the whole of the moving image. The method further includes a step of estimating a camera parameter representing a change in the movement and the direction, and a step of selecting a frame having a predetermined camera parameter as an index for display.

According to a fourth aspect of the present invention, in the moving image index generating method according to the first aspect, the similarity between any combination of frames included in a part or the whole of the moving image is determined. Calculating a frame using a predetermined calculation method, and detecting a frame that repeatedly appears by detecting a frame having a similarity greater than the threshold calculated by the predetermined method. And selecting a frame to be used as an index for display from the frames that appear repeatedly.

A moving image index generating apparatus according to a fifth aspect of the present invention provides a moving image index generating apparatus for extracting a frame which is a still image corresponding to each time from an input moving image,
A region evaluation value calculation unit that stores a dictionary relating to a desired video, calculates an evaluation value of a partial region of the frame included in a part or all of the moving image by a predetermined calculation method, and includes all regions included in the frame. Frame evaluation value calculation means for calculating an evaluation value of a frame from the evaluation value calculated by the area evaluation value calculation means for an area, and displaying an image of a specific frame from among the frames based on the evaluation value Index selecting means for selecting an index image at the time of (1).

According to a sixth aspect of the present invention, in the moving picture index generating apparatus according to the fifth aspect, the partial area of the frame is a moving object, and the index selecting means is configured to determine the moving object. It is characterized in that a still image of a frame in which a scene of a moving image changes drastically is selected as an index image based on the amount of motion.

According to a seventh aspect of the present invention, in the moving picture index generating apparatus according to the fifth aspect, the partial area of the frame is a partial area that appears repeatedly and has a high degree of similarity, and the frame evaluation value The calculating means calculates the similarity between frames as an evaluation value of the frame, and the index image selecting means selects a frame having an evaluation value higher than a predetermined threshold value as an index image having many repetitions. I have.

[0015]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of a moving picture index generating method and a generating apparatus according to the present invention will be described below in detail with reference to the accompanying drawings. First, a moving image index generation method according to a first embodiment and a moving image index generation device according to a second embodiment, which illustrate the basic concept of the present invention, will be described with reference to FIGS.

FIG. 1 is a flowchart showing an operation procedure of a moving image index generating method according to the first embodiment of the present invention. In FIG. 1, the moving image index generating method is executed at each time from an input moving image. Step ST1 of extracting a frame that is a corresponding still image, and calculating an evaluation value of a partial area of the frame included in a part or all of the moving image by a predetermined calculation method using a dictionary relating to a desired video. Step ST2, calculating a frame evaluation value from the evaluation values calculated for all the regions included in the frame, and step ST3. Calculating an image of a specific frame from the frames based on the evaluation value. Step to select as index image for display
ST4.

Next, a moving picture index generating apparatus according to a second embodiment showing the basic concept of the apparatus invention will be described with reference to the block diagram of FIG. In FIG. 2, the moving image index generation device according to the second embodiment stores a frame extraction unit 1 for extracting a frame that is a still image corresponding to each time from an input moving image and a dictionary relating to a desired image. A region evaluation value calculation means 2 for calculating an evaluation value of a partial region of the frame included in a part or all of the moving image by a predetermined calculation method, and the region for all the regions included in the frame Frame evaluation value calculation means 3 for calculating an evaluation value of a frame from the evaluation value calculated by the evaluation value calculation means 2, and an index image for displaying an image of a specific frame from the frames based on the evaluation value And an index image selecting unit 4 for selecting the index image.

The operation of the moving picture index generating apparatus according to the second embodiment corresponds to the processing procedure of the moving picture index generating method according to the first embodiment described above. . Next, FIG. 3 or FIG.
Each embodiment according to the lower concept will be described with reference to FIG.

A moving image index generating apparatus according to a third embodiment of the present invention is configured as shown in FIG. 3, for example. In FIG. 3, reference numeral 101 denotes a user interface unit, which is actually a keyboard, a mouse, a pointing device such as a touch panel, or the like. User is 10
By sending an instruction from 1, the system causes the system to reproduce, stop, display an index, and the like.

A control unit 102 controls the entire system. An image display unit 103 displays a moving image or an index image. Reference numeral 105 denotes a bus through which data is transmitted between the processing units. 1
Reference numeral 04 denotes a moving image recording unit, specifically, a large-capacity data recording medium such as a magnetic disk, a magnetic tape, and an optical disk. The moving image recording unit 104 may be replaced with a moving image server on the network or reception of a moving image broadcast.

Reference numeral 106 denotes an index video selection unit.
A process of selecting an index video of a moving image stored in the moving image recording unit 104 is performed. The method of the selection process here will be described in detail later. Reference numeral 107 denotes an index information recording unit which stores information on the index selected by the index video selection unit 106. Specifically, it is a recording medium such as a semiconductor memory, a magnetic disk, and an optical disk.

Reference numeral 108 denotes a cut detection unit which detects a cut between discontinuous frames of a video from a moving image stored in the moving image recording unit 104. Reference numeral 109 denotes a cut information storage unit, which is a recording medium such as a semiconductor memory, a magnetic disk, or an optical disk that stores the cut information detected by the cut detection unit 108.

When generating the index information, first, the cut detection section 108 detects a cut. The moving image to be processed is sent from the moving image recording unit 104 to the cut detection unit 108 via the bus 105.
The processing in the cut detection unit 108 can be performed by any cut detection method.
8084, JP-A-3-214364, JP-A-4-214
It is possible to apply the method described in, for example, Japanese Patent No. 219878. Information on the detected cut is output as cut information such as a frame number, time from the beginning of the moving image, and the number of bytes from the beginning, and is stored in the cut information storage unit 109.

Next, among the cut information stored in the cut information storage unit 109, a frame sandwiched between consecutive cuts is set as a partial moving image for index selection.
The index video selection unit 106 selects an index video for each of the plurality of partial moving images. Although the processing here will be described later, the information on the selected index video includes a frame number, a time from the beginning of the moving image,
Only the frame corresponding to the index information in a format such as the number of bytes from the head is reproduced from the moving image recording unit 104,
Video data consisting only of the index video may be generated.

When an index display command is input from the user interface 101, the index information stored in the index information storage unit 107 is read, and a video of a frame corresponding to the index information is read from the moving image recording unit 104. It is displayed on the video display unit 103. If video data consisting only of the index video has already been generated, the moving image recording unit 10
4, video data consisting of only index video is read directly and displayed on the video display unit 103.

Hereinafter, the processing in the index video selection unit 106 will be described in more detail with reference to FIG. FIG. 4 is a block diagram of the inside of the index video selection unit 106.

First, based on the cut information stored in the cut information storage unit 109, a frame between cuts for which an index video has not been selected is selected as a selection range of the index video. The frames within the selected range are sequentially read into the frame image storage unit 201 via the bus 105. Reference numeral 202 denotes a sample area cutout / normalization unit that cuts out a rectangular area of a part of the frame image stored in the frame image storage unit 201. It is easy to predetermine the scale and position of the rectangle to be cut out,
You may make it possible to change sequentially according to the input image. Usually, a rectangle of a combination of a plurality of scales and positions is cut out from one frame image as a sample area. For example, it is defined that the image is cut out from all positions on the frame in steps of 20 pixels in the horizontal direction, 50 pixels in the vertical direction, 50 pixels in the horizontal direction, and 50 pixels in the vertical direction in one pixel each. Since the cut-out sample regions have different scales, they are normalized to a predetermined scale.

The object dictionary recording unit 204 is a recording medium for recording an object dictionary. In the object dictionary, an image of an object to be found is learned in advance, and parameters for defining a statistical distribution are obtained. For example, a dictionary is created using human faces, animals, cars, and the like as objects. Bayesian estimation is well known as a dictionary creation technique. The object dictionary may be a set of representative patterns of the image of the object to be found.

The distance calculator 205 calculates the distance between the sample area normalized by a predetermined method and the data registered in the object dictionary. For example, when an average vector of a statistical distribution of objects or a covariance matrix is used as the object dictionary, a Euclidean distance or a Mahalanobis distance from the average can be used. When a representative pattern of an image of an object is registered in the object dictionary, the distance may be calculated as a distance between the normalized sample area and a representative pattern having the closest Euclidean distance or Mahalanobis distance.

The distance information calculated by the distance calculation unit 205 is stored in the distance information storage unit 203. Only the minimum value of the distance is stored and updated for each same frame, and when the distance calculation processing for all the sample regions in the frame is completed, the minimum value of the distance is stored in the distance information storage unit 203 as distance information representative of the frame. Is done. Distance information storage unit 2
Reference numeral 03 stores the minimum value of the distance in the frame, the minimum value of the distance information representing the frame calculated so far, the frame number of the frame, or information in place of the minimum value. When the distance calculation processing for all the frames in the index video selection range is completed, the distance information storage unit 203 selects, as the index video, the frame having the smallest distance value among the frames in the index video selection range. Then, information such as the frame number, the time from the beginning of the moving image, the number of bytes, and the like are sent to the index information storage unit 107 via the bus 105.

For example, it is assumed that a front face of a person has been learned and registered as an object dictionary. In this case, when the distance calculated by the distance calculation unit 205 is small, it means that there is a high possibility that a front image of a person is cut out as a sample area. FIG. 16 shows a state in which a frame including a frontal face is selected as an index in this example. In FIG.
The distance calculated by the distance calculation unit 205 has a minimum value in the center frame including the front face, and the calculated distance is large in the left and right end frames not including the front face. As a result, the frame including the central front face becomes the index.

By performing such processing, a frame in which an object meaningful as a person exists is displayed at the time of index display, instead of selecting an insignificant frame as an index, so that the contents can be easily grasped. Thus, it is possible to provide a display capable of performing a search efficiently.

The type and number of object dictionaries stored in the object dictionary recording unit 204 are not limited. Therefore, a plurality of objects to be displayed as an index can be prepared. For example, by registering a car dictionary in addition to a person's face, even when a person does not appear, a frame in which a car appears can be preferentially selected as an index.

When only a large distance is calculated in any frame, it is determined that none of the objects registered in the object dictionary is included, and an index image is selected by another method. It may be. For example, a simple method such as setting the first frame in the index video selection range or the middle frame may be used.

The processing described so far is performed by software,
It can be done in either hardware. When the processing is performed by software, the index generation of the present invention can be performed by the processing shown in the flowchart of FIG. 5, for example. In FIG. 5, an unprocessed frame is selected from the frame section (step ST
11) After extracting a sample area from the frame image (step ST12), the sample area is normalized to obtain a dictionary ten-distance (step ST13). If the distance obtained in step ST13 is the smallest, the current frame is set as an index candidate (step ST14). Then, step
In ST15, it is determined whether or not sample areas of all scales / positions have been tried, and if scales / positions that have not been tried remain, the process returns to step ST12, and detection is performed for all sample areas. . Step ST
If it is determined in step 15 that all the sample areas have been tried, it is determined in step ST16 whether all the frames in the frame section have been tried, and the processing has been performed on all the frames. If not, the processing of steps ST11 to ST16 is repeated. If it is determined that the processing has been performed for all frames, the index candidate is determined as an index video in step ST17.

Next, a moving image indexing apparatus according to a fourth embodiment of the present invention will be described with reference to FIG. Also in the fourth embodiment, the portions related to FIG. 3 described in the device of the third embodiment are described in the same manner, and thus the description is omitted here. Therefore, only the configuration of the index video selection unit 106 will be described.

FIG. 6 is a block diagram showing an example of the configuration of the index video selection unit 106 in the device according to the fourth embodiment. First, based on cut information stored in the cut information storage unit 108, a frame between cuts for which an index video has not been selected is selected as a selection range of the index video. The frames within the selected range are sequentially read into the frame image storage unit 401 via the bus 105. Reference numeral 402 denotes a sample area cutout / normalization unit,
A rectangular area of a part of the frame image stored in the frame image storage unit 401 is cut out. The scale and position of the rectangle to be cut out are exactly the same as those described in the third embodiment, and may be determined in advance or may be changed sequentially according to the input image. Since the cut-out sample regions have different scales, the sample region cut-out / normalization unit 402 normalizes the cut-out sample regions to a predetermined scale suitable for distance calculation to be performed later. Unlike the third embodiment, the sample cutout / normalization unit 402 of the fourth embodiment also performs a process of transmitting the scale and position information of the cutout region to the frame evaluation value calculation unit 405.

The object dictionary recording unit 406 is a recording medium for storing an object dictionary. As described in the third embodiment, the object dictionary is obtained by preliminarily learning the image of an object to be found and obtaining a parameter defining a statistical distribution. The distance calculation unit 403 calculates the distance between the object dictionary stored in the object dictionary storage unit 406 and the normalized sample area using a predetermined method. The calculated distance information is sent to the frame evaluation value calculation unit 405.

When the value of the distance sent from the distance calculation unit 403 is smaller than a predetermined threshold value, the frame evaluation value calculation unit 405
Then, an evaluation value of the area is calculated from the scale, position information, and distance information of the cutout area sent from 02. This evaluation value is set to be larger as the scale is larger, the position is closer to the center of the frame, and the distance is shorter. For example, the distance between the center of the region and the center of the frame is d, the scale (for example, the area) is s, and the distance calculation unit 403 is used.
D is the distance to the dictionary calculated in the above, and C1, C2, C
3, C4, C5 as constants, It shall be calculated as

The frame evaluation value calculation unit 405 accumulates the evaluation values of the above-described areas, calculates the sum of the evaluation values of the areas calculated for the areas in the same frame, and uses the sum as the frame evaluation value. . However, regarding the evaluation value of the area calculated for the overlapping area in the same frame,
The sum is calculated only for one evaluation value in these areas.

The frame evaluation value calculation unit 405 determines the maximum value for all frames within the index video selection range, the frame number of the frame giving the maximum value, or information in place of the frame number (time from start, number of bytes, etc.). Is stored. When the processing is completed for all the frames within the index video selection range, the frame number giving the maximum frame evaluation value or information in place of the frame number is sent to the index information storage unit 107, so that the index within the index selection range is obtained. The selection ends.

By performing the processing as in the fourth embodiment, an object to be picked up as an index is located near the center of the screen and a large object is easily selected as an index from the frames within the index selection range. Become. For example, if a person's front face is selected as the object, a frame that is larger in the center will be selected as an index than a frame that has a small face at the edge of the screen. The visibility is improved. FIG. 3 shows an example. Since the face is at the edge of the screen in the leftmost frame, the frame evaluation value is lower than that in the center frame in which the face is reflected in the center. Further, in the rightmost frame, since the face is reflected on the edge and is small, a lower frame evaluation value is calculated. In the case of the example of FIG.
The center frame is selected as the index.

On the other hand, C1 =
When C3 = C4 = 0 is selected, a frame in which many objects exist is easily selected as an index. When a frontal face of a person is adopted as an object as in the previous example, a frame in which two or three people are reflected is more easily selected as an index than a frame in which only one is reflected. Such an index is
This is a preferable index for grasping the characters.

The processing described so far can be performed by either software or hardware, as in the third embodiment. When processing is performed by software, for example, the index generation of the present invention can be performed by the processing shown in the flowchart of FIG. In FIG. 7, steps ST11 to ST13 and steps ST15 to ST17 are the same as the processing operations of the index image selection unit of the third embodiment shown in FIG. In the processing operation of the fourth embodiment, instead of step ST14 of the third embodiment, if the distance is smaller than the threshold value, a step ST for obtaining an evaluation value from the distance / position / scale is performed.
The third embodiment differs from the third embodiment in that step ST21 is provided between step ST15 and step ST16, and if the sum of the evaluation values of the entire frame is the largest, the step ST21 is performed using the current frame as an index candidate. Is different.

Next, a fifth embodiment will be described. Also in the fifth embodiment, the portions related to FIG. 3 described in the third embodiment are described in the same manner, and thus the description is omitted here. Therefore, only the processing of the index video selection unit 106 will be described.

FIG. 6 is an example of a block diagram of the index video selection unit 106 in the moving picture index generation device of the fifth embodiment. First, based on the cut information stored in the cut information storage unit, a frame between cuts for which an index video has not been selected is selected as a selection range of the index video.

For the frames within the index video selection range, the moving object separation unit 601 performs a process of separating the objects existing within the frames from the frames within the index selection range. For example, the amount of movement between frames is estimated as a vector for each small area of the video, and this is performed for the front of the frame. The motion information for each small area of the entire frame obtained by this processing is generally called an optical flow, and various methods have been developed (for example, PROCEEDING OF THEIEEE, p
p.917-935, VOL.76, NO.8, 1988.). Next, a continuous region having a similar motion is specified by clustering the vectors representing the obtained motion. By such processing, objects having different movements can be separated (for example, COMPUTER GRAPHICS AN
D IMAGE PROCESSING, pp. 301-315, VOL. 9, 1979.).

The object area calculation unit 602 calculates the area occupied by the object separated by the moving object separation unit 601. When a plurality of objects are separated from one frame, it is also specified which object has the largest area. The area here is the number of pixels occupied by the separated object.

Next, the object center-of-gravity calculation unit 603 calculates the position of the center of gravity of the object. This is performed for an object determined to have the maximum area in the object area calculation unit among the separated objects in one frame.

The position of the center of gravity calculated by the object center of gravity calculation unit 603 is sent to the distance calculation unit 605, and the Euclidean distance from the center position of the frame is calculated. The calculated distance is sent to the distance information storage unit 604.

The distance information storage unit stores the value of the minimum distance among the processed frames and information (frame number, time from the beginning, number of bytes, etc.) specifying the frame giving the minimum distance. I do. When the distance information about the new frame is sent from 605, the minimum distance update processing is performed. When the input of the distance information regarding all the frames within the index video selection range is completed, the specific information of the frame giving the minimum value of the distance is sent to the index information storage unit 107 via the bus 105, and the index selection processing ends.

By performing the processing described in the fifth embodiment, the largest moving object appearing in the index video selection range and the frame located at the most central position are selected as the index. First,
In the second embodiment, it is necessary to specify in advance what the object is. However, in the fifth embodiment, regardless of the type of object, if there is a large moving object in the frame, it is possible to select the frame in which the object is shown in the center as an index. For example, in a scene where a car crosses from one end of the screen to another, the frame in which the car is located at the center of the screen is selected as an index.

The processing described in the fifth embodiment can be performed by either software or hardware. When processing is performed by software, the index generation of the present invention can be performed by, for example, the processing shown in the flowchart of FIG. In FIG.
Step of selecting an unprocessed frame from the frame section
Before performing the processing in ST11, a step ST22 for separating the moving object included in the frame section is provided. Further, between step ST11 and step ST14, there are provided a step ST23 for calculating the center of gravity of the object having the largest area included in the frame image, and a step ST24 for calculating the distance between the center of the object and the center of the video. Other processing steps are the same as the processing steps in FIG.

Next, a moving picture index generating apparatus according to a sixth embodiment will be described. In the sixth embodiment as well, the parts related to FIG. 3 described in the third embodiment are described in exactly the same manner, and thus description thereof is omitted here. Therefore, only the process of the index video selection unit 106 will be described.

FIG. 10 is an example of a block diagram of the index video selection unit 106 in the generator according to the sixth embodiment. First, similar to some embodiments described above, based on the cut information stored in the cut information storage unit 109,
A frame between cuts for which the index video has not been selected is selected as a selection range of the index video.

For the frames within the index video selection range, the moving object separation unit 802 performs separation processing of the objects existing within the frames for the frames within the index selection range. This is the fifth
It is the same as 601 described in the embodiment,
This can be performed by clustering optical flows.

In the object gravity center calculation unit 801,
The center of gravity of the object separated in 2 is calculated. However, unlike the fifth embodiment, the center of gravity is calculated for all the separated objects. The object area calculation unit 803 calculates the area (the number of occupied pixels) of all objects separated in 802. The position information of the center of gravity and the area information are sent to the object evaluation value calculation unit 804 for each object.

The object evaluation value calculation unit 804 calculates an evaluation value for each object from the area information and the position information of the center of gravity. The object evaluation value is determined so as to increase as the area increases and to increase as the position of the center of gravity approaches the center of the screen. For example, assuming that the area is s 'and the distance between the center of gravity and the center of the screen is d', C6, C7, C8, and C9 are constants, And The object evaluation values are calculated for all objects separated from the same frame, and sent to the frame evaluation value calculation unit 806.

The frame evaluation value calculation section 806 calculates a frame evaluation value from the evaluation values calculated for the objects in the same frame. This can be performed, for example, by simply taking the sum, or can be calculated as the number of object evaluation values equal to or greater than the threshold value. The calculated frame evaluation value is sent to the frame evaluation value storage unit 805. The frame evaluation value storage unit 805 selects a frame having the largest frame evaluation value from all the frames within the index selection range, and stores information (frame number, frame number, The time from the beginning, the number of bytes, etc.) are sent, and the index selection processing from within the index video selection range ends.

By performing the index selection processing as in the sixth embodiment, among the frames within the index selection range, the moving objects are as many as possible, are present near the center of the screen as much as possible, and appear as large as possible. The frame is chosen as the index. By using a frame having such a property as an index, an object appearing in a moving image can be quickly recognized.

The processing described in the sixth embodiment can be performed by either software or hardware. When processing is performed by software, for example, the index generation of the present invention can be performed by the processing shown in the flowchart of FIG. The difference between the flowchart for explaining the operation of the fifth embodiment shown in FIG. 9 and the operation of the generation device according to the sixth embodiment shown in FIG. 11 is that instead of steps ST24 and ST14 in FIG. The point is that there is provided a step ST25 in which the sum of the obtained values is used as the evaluation value of the frame, and a step ST26 in which the current frame is used as the index candidate if the evaluation value of the frame is the largest.

Next, a moving picture index generation device according to a seventh embodiment will be described. Also in the device according to the seventh embodiment, the configuration of the portion related to FIG. 3 described in the third embodiment is completely the same, and therefore, redundant description will be omitted. Therefore, only the process of the index video selection unit 106 will be described.

FIG. 12 shows an index video selecting section 106 in the moving picture index generating apparatus according to the seventh embodiment.
Is a different example of the block diagram of FIG. As in the previous embodiments, based on the cut information stored in the cut information storage unit, a frame between cuts for which an index video has not been selected is selected as a selection range of the index video.

Camera parameters of frames within the index video selection range are estimated by the camera parameter estimating unit 1001. The camera parameters refer to parameters relating to the relative position and orientation between the subject and the camera when a moving image is captured, movement such as pan and zoom, change in magnification, and the like. The camera parameter estimating unit 1001 calculates a camera parameter estimating unit 1001 for a given frame from a plurality of consecutive frames.
Then, camera parameters in a certain frame are estimated from a plurality of consecutive frames. As a technique for this, the vanishing point of the motion vector is obtained using the Hough transform, the vanishing point having the largest number of votes among the plural vanishing points is taken as the vanishing point with respect to the background, and the constraint equation regarding the camera parameter is calculated. There are techniques such as preaching. The relationship between the camera parameter and the vanishing point and a method of obtaining the camera parameter are described in detail in, for example, "Kenichi Kanaya," Image Understanding ", Morikita Publishing". When the camera parameters are estimated, the values of the estimated parameters are sent to the camera parameter matching unit 1004.

The camera parameter matching section 1004 determines whether or not the parameter value sent from the camera parameter estimating section 1001 is the same as the camera parameter registered in the registered camera parameter storage section 1002. The camera parameter registered in 1002 may be a parameter value included in one frame, but it is more preferable to set it as a change point of the camera parameter. For example, as a frame at the time when the zoom-in is completed, a condition that a value of a parameter representing zoom (which is assumed to be a positive value at the zoom-in) is 0 and a differential value of the parameter in the immediately preceding frame is negative is set. A condition is registered such that a parameter representing pan is 0 and the absolute value of the differential value of the parameter of the immediately preceding frame is larger than a threshold as the start or end of pan. When the registered camera parameters match the estimated camera parameters, the camera parameter matching unit 1004 sends information indicating that the registered camera parameters have been matched to the matching information storage unit 1003 together with frame specific information.

The matching information storage unit stores all matching information until the processing for all the frames within the index video selection range is completed. If the selection of a plurality of indexes is permitted, all the specific information of the matched frame is sent to the index information storage unit 107, and the index selection processing ends.

When only one index is selected from the index video selection range, generally, one must be selected as an index from a plurality of frames stored in 1003. At this time, the processing described in the third to sixth embodiments is performed on the frame stored in the matching information storage unit 1003,
It is possible to select an index. If a simple determination is desired, a frame close to the center of the index video selection range may be selected, or a leading frame may be selected.

By performing the index selection processing as in the seventh embodiment, for example, when a face of a person includes a zoomed-in image, a frame in which the image of the zoomed-in face is greatly reflected is displayed. Since the selected moving image is selected as an index, the moving image and the image that the photographer is paying attention to are reflected in the index.

FIG. 18 shows an example in which the condition "the camera parameters in the horizontal and vertical directions are 0 and the differential value of the immediately preceding camera parameter in the horizontal direction exceeds a certain threshold value" is registered. This condition indicates the movement of the camera in the left-right direction, that is, the frame where panning has been completed. In the moving image described in the example of FIG. 18, the panning in the right direction continues, the panning ends when the face of the person finally comes to the center, and the camera stops. The horizontal and vertical camera movement speeds (camera parameters) are shown in the upper graph, and the differential values of the same speed are shown in the lower graph. In this example, in the right frame,
Since the horizontal movement speed is 0 and the immediately preceding differential value greatly exceeds the threshold value, it is selected as an index image.

The processing described in the seventh embodiment can be performed by either software or hardware. When processing is performed by software, for example, the index generation of the present invention can be performed by the processing shown in the flowchart of FIG. Step ST11
In, after selecting an unprocessed frame from the frame section, in step ST31, the camera parameters of the frame image are estimated. In step ST32, it is determined whether or not the camera parameters of all the frames in the frame section have been obtained. If it is determined that the camera parameters of all the frames have been obtained, a frame including the registered camera parameters is set as an index candidate (step ST33). Finally, when there are a plurality of index candidates, one of them is selected and used as an index image (step ST34).

Next, a moving picture index generating apparatus according to the eighth embodiment will be described. In the eighth embodiment as well, the portions related to FIG. 3 described in the index generation device of the third embodiment are described in exactly the same manner, and thus description thereof is omitted here. Therefore, the index video selection unit 10
Only the process 6 will be described.

FIG. 14 is an example of a block diagram of the index video selection unit 106 in the eighth embodiment. As in the previous embodiments, based on the cut information stored in the cut information storage unit, a frame between cuts for which an index video has not yet been selected is selected. The frames within the selected range are sequentially transmitted to the frame image storage unit 12 via the bus.
01 is read.

In the comparison frame image storage unit 1203, of the frames included in the moving image, the frame read into the frame 1201 and the frame located at a position separated from the frame by the number of frames or more are sequentially read. The similarity calculation unit 1204 includes the frame image storage unit 120
The similarity between “1” and the frame read into the comparison frame image storage unit 1203 is calculated by a predetermined method. For example, a method of simply adding the absolute value or the square of the difference between the pixel values at the corresponding position in the entire frame is used.

The threshold value calculation unit 1202 is
In step 5, a threshold value to be compared with the similarity is calculated. Normally, it is sufficient to set the threshold value as a constant, but it is also possible to change the threshold value according to the distribution of colors and luminance on the entire screen. When the comparison unit 1205 determines that the similarity is higher than the threshold value, the frame read into the frame image storage unit 1201 is used as an index candidate, and the similarity and the frame number for specifying the frame are used. Is sent to the index candidate storage unit 1206.

In the case where only one frame is a candidate, the index candidate storage unit 1206 selects this as an index image. However, usually, a plurality of frames are candidates or none of them are candidates. If there is no candidate, a frame selected by another method is used as an index image. For example, a frame selected by the method described in the third to seventh embodiments is determined as an index image.

If a plurality of frames are candidates, one of them is determined as an index image. At this time, the determination can be made by performing the method described in the third and seventh embodiments on only the candidate frames. Also, as a simple method,
A method of determining a frame that gives the maximum value of similarity or a method of determining a frame located in the middle among candidate frames may be used.

The technique described in the eighth embodiment makes it possible to select a scene that is repeatedly photographed because it is important as an index. For example, this corresponds to a case where a scene repeatedly appearing as a replay or a case where the same scene is displayed as a slow playback at a different playback speed in a sports video. These are all important scenes, and can be detected by the method according to the present embodiment. By selecting an index from these frames, the efficiency of search and understanding of contents can be improved.

The processing described in the eighth embodiment can be performed by either software or hardware. When processing is performed by software, for example, the index generation of the present invention can be performed by the processing shown in the flowchart of FIG. FIG.
In, first, a frame of the frame section spicy processing is selected (step ST11), and then, in step ST35,
Calculate the similarity with other frames. In step ST36, a frame whose similarity is larger than the threshold value is set as an index candidate. In step ST16, the processing operation of judging whether or not all the frames in the frame section have been tried is the same as the operation according to the third to seventh embodiments. If the processing has not been completed for all frames, steps ST11, ST35 and ST36 are performed for all frames, and if the processing has been completed for all frames, the process proceeds to step ST37. Step ST37
In, when there are a plurality of index candidates, one of them is selected and output as an index video.

Next, a method for displaying and outputting an index image generated by the above-described index generation method and apparatus will be described in detail with reference to FIGS. FIG. 19 is a conceptual diagram illustrating display of an index image in the moving image index generation method according to the ninth embodiment of the present invention. On a screen 10 of a display device such as a CRT (Cathode Ray Tube) or a liquid crystal display, a three-dimensional icon window 11 representing a certain time section of a moving image is displayed at the foremost side, and a representative frame of a representative frame is displayed behind it. A list display window 12 is displayed. On the side of the three-dimensional icon window 11, a component list window 13 for supporting a video editing operation is displayed. Using the component list window 13, an editing operation on the video can be performed. Editing work performed using the parts list window 13 will be described later.

In FIG. 19, each window 11,
Reference numerals 12 and 13 are displayed in a virtual three-dimensional space on the screen, and can be freely moved in the vertical, horizontal, and depth directions by a user's operation, and can be rotated in any direction. Can also. Here, the “virtual three-dimensional space” means a three-dimensional space projected and displayed on a screen displayed in two dimensions, and is a display method generally used in the field of virtual reality. That is.

A true three-dimensional display may be performed by using a display method corresponding to the position of the left and right eyes, that is, the position of the viewpoint, and the degrees of freedom for movement and rotation are determined by the respective windows 11 and 12. And 13 may be restricted so that they do not collide with each other or become inconvenient from the display of the screen. The display device including the moving image index generation method according to the present invention may be used automatically. It is also possible to control.

FIG. 20 is an explanatory diagram showing only the three-dimensional icon window 11 on the screen 10 of FIG. 19 in detail. In the figure, the three-dimensional icon window 11 has a representative frame 15 protruding from information. Each of the representative frames 15 is spaced apart by a distance in the depth direction that is proportional to the temporal length of the shot that generated it. In FIG. 20, the frames are displayed as a series of frames arranged over a distance in the depth direction. However, the present invention is not limited to this, and it is expressed that each of a series of frames is stored in a box-shaped container. Is also good.

It is also possible to make the three-dimensional icons box-shaped and to display an index for each box so that the icons can be easily viewed. May be arranged, or conversely, they may be arranged so that they are newly arranged from the front to the back.
In short, when the representative frames are displayed side by side, any detailed specification can be used as long as the representative frames are arranged in a three-dimensional space with a spatial distance proportional to the length of time of each shot. It can be implemented even if there is.

FIG. 21 is an explanatory diagram showing details of the representative frame list window 12 shown in FIG. FIG.
In the window 12, a frame display column 17 listing representative frames generated from each shot, and an information display displaying attributes, additional information, and the like of the shot indicated by the representative frame described in the frame display column 17 Column 18. The attribute / additional information indicates general information on the shot such as information on camera operation (zoom, pan, etc.) in the shot, presence / absence of a character, name of a character, shooting time, duration, the number of frames, and the like. If the user has input a comment (memorandum) on the shot, this may be included in the attribute / additional information.

Returning to FIG. 19, an example of an operation using this display method will be described. The winner is the three-dimensional icon window 11 or the representative frame list window 12
While looking at, select the shot that you determine is necessary. This selection operation is performed by, for example, clicking a target shot portion on the representative frame list window 12 with a mouse and pressing a “selection button” (not shown) or clicking again. Further, by positioning the mouse pointer on a target shot portion on the representative frame list window 12, moving the mouse pointer onto the parts list window 13 while holding down the mouse button, and releasing the end (drag and drop), Also, a desired shot can be selected.

The selection target may be the three-dimensional icon window 11 instead of the representative frame list window, or the time portion during the shot may be limited. In the parts list window 13, the shots selected and registered in this way and the parts in the shots are listed. The parts list window 13 is displayed assuming a recording area for registering, as a material, a material used later by the user for editing work from the entire moving image. The user picks up only necessary parts from the whole moving image in the parts list window 13 in advance, and rearranges the video materials registered in the parts list window 13 in a later operation. Documents can be created.

Here, the list of representative frames is not limited to the list based on cut detection. For example, among the shots determined by the cut detection, only the one showing a specific person is displayed, or only the one zoomed in is displayed. , Only the shot specified by the user may be displayed.

According to the moving image index generating method and apparatus according to the ninth embodiment, the result of cut detection is
With the three-dimensional display, it is possible to overcome the problem that the respective representative frames overlap and become difficult to see when displaying the three-dimensional icon, by a user's arbitrary movement / rotation operation. In addition, by arranging a plurality of windows in the three-dimensional space, not only can the window most necessary for the current work be moved to the foreground, but also by arranging the windows diagonally, There are various effects such as displaying many windows without occupying a large area.

Furthermore, by introducing the parts list window, the video material can be picked up in shot units and the next video material can be picked up without the troublesome work of directly searching for a necessary part from a long moving image. Are rearranged, so that the editing work environment can be improved and the flow of the moving image editing work can be made more efficient.

[0090]

According to the moving picture index generating method and the generating apparatus of the present invention, a frame in which a meaningful object such as a person exists is selected as an index instead of selecting a meaningless frame as an index. Become. In addition, a frame in which many objects such as persons exist is easily selected as an index.

Further, the frame in which the large object is located at the center is selected as an index. Furthermore, what appears in the moving image or what the photographer is paying attention to is reflected in the index. Furthermore, scenes that have been repeatedly photographed because they are important can be selected as an index.

Accordingly, since a frame having a meaningful content is selected as an index, it is easy to grasp the content of the moving image when the index display of the moving image is performed, and it is easy to perform a desired search. become.

[Brief description of the drawings]

FIG. 1 is a flowchart showing steps of a moving image index generation method according to a first embodiment as a basic concept of the present invention.

FIG. 2 is a block diagram showing a configuration of a moving image index generation device according to a second embodiment as a basic concept of the present invention.

FIG. 3 is a block diagram showing an overall schematic configuration of third to eighth embodiments of the present invention.

FIG. 4 is a block diagram illustrating a configuration of a selection unit according to a third embodiment of the present invention.

FIG. 5 is a flowchart illustrating a processing procedure according to a third embodiment of the present invention.

FIG. 6 is a block diagram illustrating a configuration of a selection unit according to a fourth embodiment of the present invention.

FIG. 7 is a flowchart illustrating a processing procedure according to a fourth embodiment of the present invention.

FIG. 8 is a block diagram illustrating a configuration of a selection unit according to a fifth embodiment of the present invention.

FIG. 9 is a flowchart illustrating a processing procedure according to a fifth embodiment of the present invention.

FIG. 10 is a block diagram illustrating a configuration of a selection unit according to a sixth embodiment of the present invention.

FIG. 11 is a flowchart illustrating a processing procedure according to a sixth embodiment of the present invention.

FIG. 12 is a block diagram illustrating a configuration of a selection unit according to a seventh embodiment of the present invention.

FIG. 13 is a flowchart illustrating a processing procedure according to a seventh embodiment of the present invention.

FIG. 14 is a block diagram illustrating a configuration of a selection unit according to an eighth embodiment of the present invention.

FIG. 15 is a flowchart showing a processing procedure according to the eighth embodiment of the present invention.

FIG. 16 is an explanatory diagram showing an example of an object dictionary relating to distance in the present invention.

FIG. 17 is an explanatory diagram showing another example of the object dictionary relating to the evaluation value in the present invention.

FIG. 18 is an explanatory diagram showing an example of an object dictionary of camera parameters according to the present invention.

FIG. 19 is an explanatory diagram showing a display method according to a ninth embodiment of the present invention.

FIG. 20 is an explanatory diagram showing details of a ninth embodiment of the present invention.

FIG. 21 is an explanatory view showing different details of the ninth embodiment of the present invention.

[Explanation of symbols]

 Reference Signs List 1 frame extraction means 2 area evaluation value calculation means 3 frame evaluation value calculation means 4 index selection means 101 user interface unit 102 control unit 103 video display unit 104 moving image recording unit 105 bus 106 index video selection unit 107 index information storage unit 108 cut Detection unit 109 Cut information storage unit

Continuing on the front page (72) Inventor Susumu Kubota 1 Toshiba R & D Center, Komukai Toshiba-cho, Saiwai-ku, Kawasaki-shi, Kanagawa

Claims (7)

[Claims] 1. A step of extracting a frame as a still image corresponding to each time from an input moving image, and a part or all of the moving image by a predetermined calculation method using a dictionary relating to a desired image. Calculating the evaluation value of the partial area of the frame included in the frame; calculating the evaluation value of the frame from the evaluation values calculated for all the areas included in the frame; based on the evaluation value Selecting an image of a specific frame from among the frames as an index image to be displayed. 2. A step of cutting out a moving object from a background with respect to the frame included in a part or the whole of the moving image, and calculating an evaluation value by a predetermined calculation method for each of the cut out objects. And calculating an evaluation value of a frame from evaluation values of objects included in the same frame; and selecting and outputting a specific frame as an index for display based on the evaluation value of the frame. The moving image index generation method according to claim 1, further comprising: 3. Estimating camera parameters representing changes in the movement and direction of a camera that is capturing the moving image for a frame included in a part or the whole of the moving image; Selecting a frame having the following as an index for display;
The moving image index generation method according to claim 1, further comprising: 4. A step of calculating a similarity between frames of an arbitrary combination with respect to a frame included in a part or the whole of the moving image by a predetermined calculation method, and calculating the similarity by a predetermined method. Detecting a frame that repeatedly appears by detecting a frame having a degree of similarity greater than the threshold, and selecting a frame to be an index for display from these repeatedly appearing frames, The moving image index generation method according to claim 1, further comprising: 5. A frame extracting means for extracting a frame which is a still image corresponding to each time from an input moving image, a dictionary relating to a desired image is stored, and a part of the moving image is stored by a predetermined calculating method. Or an area evaluation value calculating means for calculating an evaluation value of a partial area of the frame included in all of the frames; and an evaluation of the frame from the evaluation values calculated by the area evaluation value calculating means for all the areas included in the frame. A moving image index generating apparatus, comprising: a frame evaluation value calculating unit that calculates a value; and an index image selecting unit that selects an image of a specific frame from the frames as an index image when displaying the image based on the evaluation value. . 6. A partial area of the frame is a moving object, and the index selecting means uses a still image of a frame in which a scene of a moving image changes drastically as an index image based on a moving amount of the moving object. The moving image index generation device according to claim 5, wherein the moving image index is selected. 7. The partial area of the frame is a partial area that appears repeatedly and has a high degree of similarity, wherein the frame evaluation value calculating means calculates the similarity between frames as an evaluation value, and the index image selecting means includes: A frame having a higher evaluation value than a predetermined threshold value is selected as an index image having a large number of repetitions.
5. The moving image index generation device according to item 1.