JP2003348528A - Representative image selection method, representative image selection apparatus, representative image selection program and recording medium for representative image selection program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To select a frame in which the motion of an object in a video easy to recognize and no camera shaking is present in order to create a representative image from the video in a video database system, a video indexing means or a digital video editing system. <P>SOLUTION: An image change amount in a video block is calculated by an image change amount calculating means 111, and a frame in which the image change amount is minimum is calculated by an image change amount minimum frame calculating means 112. The frame in which the image change amount becomes minimum is found to detect an image having a high possibility of a characteristic attitude to become a node of the motion when positively moving a figure or an animal. Then, with the found frame in which the image change amount becomes minimum as a reference, a representative image candidate selecting means 113 selects candidates of the representative image in the video block. <P>COPYRIGHT: (C)2004,JPO

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video database system, a video indexing system, a digital video editing system, and the like, in which a representative image is created from a video. It relates to a method of selecting a frame.

[0002]

2. Description of the Related Art In recent years, with the increase in speed and cost of random access storage devices such as computers and hard disk devices, it has become widely practiced to convert video into computer-readable digital data and record and handle the data in the random access storage device. Is becoming increasingly common. There are various advantages to handling video by a method based on so-called video digitization as compared with a method of recording and handling moving images on a magnetic tape or the like.

One of the advantages is that "video indexing" for creating an index based on the content of a video can be suitably applied. The video index is a set of a label indicating the content of a certain section of the video and section information of the video section. The section information of the video section is represented by the start position and the end position of the video section. It is also indicated by the start position and the length of the section. In some cases, only the start position is indicated. The position in the video is represented by the elapsed time from the top of the video, the total frame number from the top frame of the video, and the like. The length of the video section is represented by an elapsed time, the number of frames, or the like. By using the index of a video, a desired video section can be selected without immediately viewing the entire video, and the desired video section can be immediately called from the random access storage device and browsed.

There are various labels representing the contents of a video section, but a reduced image of a certain frame in the video section is often used as a label for the video section for visual indication. At this time, the reduced image is called a representative image of the video section. Note that the number of representative images in one video section need not be limited to one, but may be plural. Also, a representative image may be created by processing one frame or a plurality of frames extracted from a video. Further, the representative image is not limited to a still image, and for example, a moving image may be used as a label representing the video section. Therefore, the case where a moving image is used as a label is also referred to as a representative image.

Various methods have been proposed for automatically indexing videos. For example, as a method of automatically extracting a video section, there is a method of separating a video by a so-called scene change, which is a portion where the video changes rapidly with time. Further, as a method of extracting a representative image of a video section, there are a method of selecting a first frame of the video section and a method of selecting a frame at a time when a predetermined time has elapsed from the top.

[0006] Various manufacturers such as video producers and broadcasters have attempted to manage a huge amount of videos and provide video sales services to customers by digitizing and indexing the videos owned by the companies. We are considering it.

[0007]

Incidentally, it is desirable that the representative image of the video section is a still image that best represents the contents of the video section. However, the conventional method of selecting the first frame of a video section or a frame at a time after a certain time has elapsed from the top does not always result in a still image that best represents the content of the video section, and this point is a major problem. Was.

For example, in a video section in which a person swings a golf club in one scene of golf, the instant of the swing may be appropriate as a representative image. However, since the above-mentioned conventional method does not have an effect of selecting a frame at the moment of a swing, a frame which is irrelevant to the swing and in which only a person stands or a frame in which only a landscape is reflected is selected as a representative image. May be lost. It is difficult to know from such a representative image that there is a swing image in the image section.

[0009] Further, in a video image of a fast-moving subject, the subject may be blurred. Although a blurred image is not suitable as a representative image, the conventional method has no effect of determining whether or not a subject is blurred, and thus has a problem that a frame with severe blurring may be selected as a representative image.

In order to solve the problem in selecting a representative image, Japanese Patent Laid-Open No. 9-93527 discloses a method of manually selecting the most appropriate frame while reproducing a video. However, according to this method, although an appropriate representative image can be obtained, there is a problem that human cost is required. In addition, there is also a problem that it takes time to manually select while reproducing the video.

An object of the present invention is to solve the problem of the above-mentioned conventional method. A method of automatically selecting a frame in which movement of a subject is easy to understand and has no blur from a video section as a representative image. The aim is to provide a way to make selection easier.

[0012]

SUMMARY OF THE INVENTION In the present invention, the movement of a person or an animal in the case of active movement is observed, and attention is paid to the fact that the movement or the movement is momentarily stopped at a turning point of the movement. Such a point at which the movement stops for a moment or becomes a minimum is hereinafter referred to as a "motion valley". For example, a series of movements of a person swinging a golf club can be decomposed into a swinging-up operation and a swinging-down operation of the golf club, and the boundary becomes a turning point and a movement valley. Not limited to this example,
In many cases, the active movement of a person or an animal can be broken down into a series of simple movements, and at the boundaries of these simple movements, movement valleys occur. Although it cannot be broken down into simple movements, some movements may be seamless, but it is considered impossible for organisms to continue to move without any breaks and without slow rhythms. .

In the present invention, attention has been paid to the fact that the movement of the person or the animal is well represented in the posture in the movement valley in the active movement of the person or the animal. For example, the position immediately after swinging the golf club, which is the turning point of the swinging motion of the golf club, and immediately before swinging down the golf club is a posture that clearly shows that the golf club is about to swing down. ing. Not limited to this example, the valley in the active movement of a person or an animal is a boundary between continuous simple movements, an end point of a certain movement and a starting point of a certain movement. I often take a posture. Even if the posture in the moving valley is not always characteristic, it is considered that there is a high possibility that the posture is characteristic as compared with a place other than the moving valley.

In a moving valley, the main body of the motion is stationary or the motion is slow, so that the blurring when photographing with a video camera or the like is small. As a measure of the degree of movement, the amount of change in the luminance of pixels between adjacent frames can be used, and can be calculated mechanically.

Based on the above considerations, the first aspect of the present invention is characterized in that a representative image candidate is selected on the basis of a frame at a point in time when the amount of change in video becomes temporally minimal.

The second aspect of the present invention is characterized in that, as a typical example in the case where the above-mentioned frame is used as a reference, the frame at the time when the amount of change in the image becomes temporally minimal is set as a representative image candidate. I do.

As a method of expressing motion by a still image, a plurality of images of a moving subject at different times are copied into one image.
There is a so-called strobe image. Focusing on the use of this strobe image, the third aspect of the present invention is to construct a strobe image by processing a plurality of frames before and after the frame at the time when the amount of change in the image becomes minimal in time. , Representative image candidates.

By the way, the amount of change in the image of a video does not occur only due to the movement of a subject, but also due to camera work (translation, rotation, zooming of a camera). Therefore, by estimating the camera work from the video, converting the video to cancel the camera work, and calculating the image change amount for the converted video, the movement of the subject is reflected well in the image change amount. Become.

A fourth aspect of the present invention is to estimate camera work (translation, rotation, and zooming of a camera) from an image, convert the image so as to cancel an apparent movement of a subject due to the camera work, and convert the image. The present invention is characterized in that the invention is applied to a video.

According to a fifth aspect of the present invention, a representative image candidate is selected by the first method of the present invention, and an appropriateness of each representative image candidate is determined as a representative image by a predetermined procedure. A representative image is selected from the representative image candidates.

According to a sixth aspect of the present invention, in the fifth aspect of the present invention, when determining the appropriateness of the representative image, | X−θ _x | is calculated based on the dent X shown in the following equation (1). It is characterized by giving greater relevance to smaller ones.

[0022]

(Equation 3)

Where f is the amount of change in the image as a function of time t, θ _x is a constant,
t ₁ and t ₂ represent the times when f is maximum on both sides of the minimum point of interest.

According to a seventh aspect of the present invention, in the fifth aspect of the present invention, f (t ₁ )
/ F to those (t ₀₎ Gayori large, giving greater relevance, _{and, f (t 2) / f} (t 0) Gayori the larger, characterized in providing a greater relevance. Here, t ₀ represents the time of the minimum point of interest.

[0025]

Embodiments of the present invention will be described below in detail. First, a first embodiment of a process for indexing a video using the representative image selection method of the present invention will be described with an example.

FIG. 1 shows an example of the configuration of a video index creation device for realizing this embodiment. The video indexing device 1
A representative image in each video section of a given video is selected, and a video index is created based on the selected representative image.
An information processing device 11 including a PU and a memory, a display device 12, an input device 13 such as a keyboard and a mouse,
It comprises a hard disk and other storage devices 14.
It is assumed that a video to be processed is digitized and stored in the storage device 14 in advance.

The information processing apparatus 11 includes an image change amount calculating unit 111 for calculating an image change amount in a video section, an image change amount minimum frame calculating unit 112 for obtaining a frame in which the image change amount is minimum, and an image change amount Representative image candidate selecting means 113 for selecting a representative image candidate of a video section with reference to a minimum frame, and representative image selecting means 1 for selecting a representative image from representative image candidates of the selected video section.
14.

FIG. 2 is a flowchart of a representative image selection process according to the present embodiment. This processing is to input a digitized video and output a set of a representative image and a video section as an index.

First, in step S201, the information processing device 11 reads a digitized video from the storage device 14, and divides the input video into video sections. As a method of dividing into video sections, a method of dividing by a so-called scene change in which a video changes abruptly in time can be suitably applied, but is not limited to this method, and another method may be used. As a result of the division,
It is assumed that the input video is divided into n video sections.
In order to store each video section, it is sufficient to store the start position and the end position in the video. The position in the video may be an elapsed time from the top of the video, a total frame number counted from the top frame of the video, or the like.

Next, in step S202, the number n of video sections is substituted for a variable N, and in step S203, 1 is substituted for a variable i. In step S204, the variable i
Is compared with the value of the variable N. If i ≦ N, the process proceeds to step S205; otherwise, the process ends. The loop is controlled by this conditional branch. In the loop, the processing is performed by focusing on the i-th video section in the i-th loop.

In step S205, the image change amount f is calculated for each frame of the i-th video section, and the result is stored in the array F. The image change amount f is calculated by, for example, the following equation (2) between adjacent frames.

[0032]

(Equation 4)

However, I ₁ and I ₂ are images of two adjacent frames. xs, xe, ys, ye are predetermined constants, and the points (x, x) satisfying xs ≦ x ≦ xe, ys ≦ y ≦ ye
y), a rectangular area in the screen is defined, and for the pixels in the rectangular area, the absolute values of the differences between the luminance values of the images are summed to obtain the image change amount f. The image change amount is not limited to the sum of the absolute values of the differences between the luminance values of the pixels, but may be another amount such as a color histogram. Further, the pixel to be subjected to the image change amount calculation may be the entire screen.

Assuming that the first frame of the video section is the first frame, the amount of change between the first frame and the second frame is F
[1], the change amount of the second frame and the third frame is substituted for F [2], and so on. That is, the change amount between the k-th frame and the (k + 1) -th frame is represented by F
Substitute [k]. Note that the frame number here is a frame number counted from the beginning of the video section, and does not match the frame number counted from the beginning of the video. In the present embodiment, the subscript of the array F is described to start from 1, but need not start from 1.

Next, in step S206, a subscript t at which F [t] is minimized is obtained from the array F, and
To be stored. A method for obtaining the minimum point will be described later.
Note that, in order to reduce the influence of noise, a smoothing process may be performed on the series of image change amounts stored in F before obtaining the minimum.

Assume now that the video section of interest for the sake of explanation is a scene in which a person swings a golf club as shown in FIG. As shown schematically in FIG. 3, the video section includes a scene in which a person swings a golf club and hits a golf ball, and then the camera zooms up to the face of the person. It is assumed that the video section is composed of ae frames.

FIG. 4 is a graph showing the amount of image change in the video section. In the graph of FIG. 4, the horizontal axis represents the suffix t, the vertical axis represents the value of F, and the line 401 represents the t-th. Represents the value F [t] of F in the frame. The horizontal axis may be interpreted as a frame number. In FIG. 4, the values of F are t = a2 and t = a2.
An example is shown in which the value of F becomes a maximum when a4 and the value of F becomes a maximum when t = a1 and t = a3.

FIG. 5 shows a plurality of frames arranged in the video section of interest. 5 in FIG.
01 is the first frame (head frame) of the video section,
502 is the a1 frame of the video section, 503 is the a2 frame of the video section, 504 is the a3 frame of the video section, 505 is the a4 frame of the video section, 506
Is the ae frame (final frame) of the video section.

Here, in the example of the video section, the relationship between the value of t, which takes an extreme value, and the movement of the subject in the video section will be described with reference to FIGS. In FIG. 5, the person who is the subject in the video section swings the golf club from the first frame 501 to the a2 frame 503. In the a1st frame 502, the swinging speed is a maximum, and as a result, as shown in FIG. 4, the image change amount is also a maximum in the a1st frame, that is, at t = a1. In the a1st frame 502, the moving speed of the golf club is fast and the shake is large. Also, the a2 frame 5
Numeral 03 indicates a transition from the swing-up operation to the swing-down operation, and the movement is temporarily reduced. As a result, as shown in FIG. 4, the image change amount is also minimal in the a2 frame, that is, at t = a2.

In FIG. 5, the person swings down the golf club and hits the ball from the a2 frame 503 to the a3 frame 504. The speed of swinging down in the a3 frame 504 is maximum, and as a result, the image change amount is also maximum in the a3 frame, that is, at t = a3 as shown in FIG. In the a3rd frame 504, the moving speed of the golf club is high and the shake is large. The a4th frame 505 in FIG.
In, the person starts to lower his arm, and his movement is temporarily reduced. As a result, the image change amount shown in FIG.
= A4, it is minimal.

In FIG. 5, the person has his arms down from the a4th frame 505 to the aeth frame 506. The boundary between the swinging motion of the golf club and the swinging down motion, and the boundary between the swinging down of the golf club and the lowering of the arm, which are the milestones of the motion of the person, appear as the minimum image change amount shown in FIG. .

Next, at step S207, step S
The number of the minimum points of F [t] obtained in 206 is substituted for a variable M. In step S208, the value of the variable M is compared with 0. If M> 0, the process proceeds to step S209; otherwise, the process proceeds to step S214.

In step S209, step S206
Based on the M frames at the minimum point of F [t] obtained in the above, L representative image candidates are selected. There may be various selection methods. For example, each of the frames at the minimum point may be used as a representative image candidate.
At this time, L = M (second embodiment described later).
A frame before or after a predetermined time from the minimum point may be set as a representative image candidate. Also, a plurality of frames appearing at predetermined time intervals before and after the minimum point may be set as representative image candidates. Further, a moving image from a certain minimum point to the next minimum point may be set as the representative image.

Next, in step S210, the selected L
The representative image candidates are displayed on the display device 12. On the other hand, the operator examines the displayed frame, selects one suitable image as the representative image of the video section, and specifies the selected image using the input device 13.

In step S211, the information processing apparatus 11
Inputs one frame selected by the operator from the input device 13 and substitutes the frame number for a variable S. The input method may be a method of pointing to a frame displayed by a mouse or the like, a method of inputting a frame number by a command, or a method of attaching a label to a frame and inputting a label by a command.

In step S212, a reduced image of the selected frame is created, paired with the position of the video section, and
The index is output to the storage device 14 as an index of the video section. The position of the video section may be the elapsed time from the top frame of the video or the total frame number from the top frame of the video.

In step S213, 1 is added to the value of the variable i, and thereafter, the process proceeds to step S204, and the process is similarly performed focusing on the next video section.

In step S208, if M> 0 is not satisfied, the head frame number 1 is substituted for the variable S in step S214, and the flow advances to step S212. This step S214 is a process when there is no extreme value in the image change amount F. In this case, since a representative image candidate cannot be obtained, the first frame of the video section is set as the representative image. If necessary, the representative image may be obtained by another method.
That is, in the present embodiment, the method of using the first frame of the video section as the representative image is used.
For example, a method disclosed in Japanese Patent No. 527 may be used.
When the variable i exceeds N in step S204 and escapes from the loop, the indexes of all video sections of the video are stored in the storage device 14.

FIG. 6 shows an example of an index output according to the present embodiment. As shown in FIG. 6, a set of a representative image and section information is stored in the storage device 14 as a video index for each video section serial number.

In the present embodiment, it is assumed that the entire video is stored in the storage device 14 in advance. However, a video input device is connected to the information processing device 11, and a video deck, a video camera, a May be input and processed sequentially to apply the present invention.

In this embodiment, the procedure of storing the image change amount of the entire video section in the array F in order to find the minimum of the image change amount has been described. Sometimes, only the image change amount of several consecutive frames is stored, and it is possible to sequentially determine whether or not the change is minimal.

In this embodiment, a method of selecting one representative image for one video section has been described. However, a plurality of representative images may be selected for one video section.

In this embodiment, the method of selecting a representative image from the candidates of the video section by the operation of the operator has been described. However, regardless of the selection of the operator, all the representative image candidates are set as the representative images. Alternatively, for example, a candidate having the youngest frame number may be selected as the representative image. Alternatively, some evaluation scale may be introduced, and the candidate with the highest evaluation may be selected as the representative image regardless of the operation of the operator.

The above is an explanation of the first embodiment of the processing for indexing a video using the representative image selection method of the present invention.

Next, a description will be given of a second embodiment of processing for indexing a video using the representative image selection method of the present invention. FIG. 7 is a flowchart of a representative image selection process according to the second embodiment of the present invention.

In the first embodiment described above, in step S209 shown in FIG. 2, it is necessary to select L representative image candidates based on M frames at the minimum point of the image change amount F [t]. On the other hand, in the second embodiment, in step S309 of FIG.
A frame in which [t] is minimal is selected as a representative image candidate. Other processing (steps S301 to S308, S
Steps S <b> 310 to S <b> 314 are the same as in the first embodiment, and a description thereof will not be repeated.

The above is the description of the second embodiment of the process of indexing a video using the representative image selection method of the present invention.

Next, a description will be given of a third embodiment of the processing for indexing a video using the representative image selection method according to the present invention. The third embodiment is obtained by adding a strobe image creation process to the first embodiment. FIG. 8 is a flowchart of the representative image selection processing according to the third embodiment.

In the third embodiment, a strobe image is created as a representative image. This strobe image transformation process
Steps S208 to S212 in the flowchart of FIG.
Step S214 is replaced by step S214 in the flowchart of FIG.
408 to S413. Other processes are the same as those of the first embodiment, and therefore, the third embodiment will be described with reference to the flowchart of FIG.

In the third embodiment, after the number of the minimum points of F [t] is substituted for the variable M (step S40).
7) In step S408, the value of the variable M is compared with 0. If M> 0, the process proceeds to step S409; otherwise, the process proceeds to step S413.

In step S409, step S406
For each of the minimum points of F [t] obtained in the above, a strobe image is created from a plurality of frames before and after. For example, constants p, q, and r are determined in advance, and p is calculated from a frame at a certain minimum point.
r seconds ago, (p-1) r seconds ago, ..., 2 r seconds ago, r seconds ago, 0
Seconds before, after r seconds, after 2r seconds, ..., after (q-1) r seconds, qr
Secondly, (p + q + 1) frames are extracted to create a strobe image. Either p or q may be 0. M strobe images are created for the M minimum points, and these are used as representative image candidates.

Next, in step S410, the created M
The strobe images are displayed on the display device 12. In response to this display, the operator examines the displayed strobe image,
One suitable image is selected and designated as a representative image of the video section.

In step S411, the information processing device 11
Selects one strobe image specified by the operator from the input device 13. The input method may be a method of pointing a frame displayed by a mouse or the like, or a method of attaching a label to a strobe image and inputting the label by a command statement.

In step S412, the selected strobe image and the position of the video section are paired and output to the storage device 14 as an index of the video section. The position of the video section may be the elapsed time from the top frame of the video or the total frame number from the top frame of the video. After step S412, the process proceeds to step S414, where 1 is added to the value of the variable i, and the process proceeds to step S404.

If M> 0 is not satisfied in step S408, the process proceeds to step S413, where the head frame of the i-th video section and the position of the video section are paired and stored in the storage device 14 as an index of the video section. Output. This step S413 is processing in the case where there is no extreme value in the image change amount F. In the present embodiment, the method of using the first frame of the video section as the representative image is used, but the representative image is obtained by another method. You may. For example, a method disclosed in JP-A-9-93527 can be used. After step S413, the process proceeds to step S414. Other processes are the same as in the first embodiment, and a description thereof will be omitted.

The above is the description of the third embodiment of the process of indexing a video using the representative image selection method of the present invention.

Next, a description will be given of a fourth embodiment of the processing for indexing a video using the representative image selection method according to the present invention. FIGS. 9 and 10 are flowcharts of the representative image selection process according to the fourth embodiment.

The processing according to the fourth embodiment is obtained by adding processing (steps S505 to S511) for deforming an image so as to cancel the movement of the camera to the first embodiment.
This image transformation process replaces step S205 in the flowchart of FIG. 2 with steps S505 to S511. Processing of other parts (steps S501 to S501)
Steps S504, S512 to S522) are the same as the processing in the first embodiment shown in FIG. 2, and therefore, the fourth embodiment will be described only for the image deformation processing part.

In the fourth embodiment, the variable i is compared with N in step S504, and if i ≦ N, all frames in the i-th video section are input in step S505. I ₁ , I ₂ ,..., I
_Let's call it _ae . Next, in step S506, 1 is substituted for a variable j.

In step S507, parameters for camera movement are extracted from the j-th frame I _j and the (j + 1) -th frame I _{j + 1} , and how the camera has moved between the two frames is estimated. For this estimation, for example, a method disclosed in JP-A-11-225310 is used. The method of estimating camera movement disclosed in Japanese Patent Application Laid-Open No. H11-225310 assumes that a point of a subject moves apparently on an image as shown in Expression (3) due to panning, tilting, and zooming of the camera. .

(X ′, y ′) = (ax ″ + b, ay ″ + c) (3) Equation (3) indicates that a point of a certain subject has coordinates (x ′, y ′) on an image in a certain frame A ), And in another frame B, a relational expression when projected at coordinates (x ″, y ″). Assuming that the apparent movement of the point is caused by the movement of the camera while the subject is stationary, the unknowns a, b, and c describe the movement of the camera from frame A to frame B. Called parameters. According to JP-A-11-225310, the values of the camera parameters a, b, and c can be determined by minimizing the mean square error. Step S5 above
In 07 obtains the camera parameters of the camera motion between the frame I _j to frame _{I j + 1, a j,} b j, c
_j .

In step S508, using the obtained camera parameters, the frame I _{j + 1} is deformed so as to cancel the movement of the camera from the frame I _j to the frame I _{j + 1} to obtain I ′ _{j + 1} . . Using the camera parameters a _j , b _j , and c _j obtained by the above method, the following equation (4) is used to calculate the point (x, y) on the frame I _{j + 1} on the frame I _j . (X ', y') can be obtained.

[0073] (x ', y') = (a j x + b j, a j y + c j) ... (4) Then, in step S509, the image I _j and the image I _{'j + 1}
Is obtained and substituted into F [j]. Then, in step S510, 1 is added to the value of the variable j.

In step S511, the value of variable j and the value of i
J> ae
If the value is -1, the image transformation process is terminated, and the process proceeds to step S512 where the same process as step S206 in FIG. 2 is performed. If j> ae−1 is not satisfied, the process proceeds to step S507 to form a loop. In this loop, stores the image change from frame I ₂ of the i-th image segment until frame I _ae in sequence F.

When the image deformation processing is completed, an image change amount F can be obtained in this video section as if the camera had not moved. As a result, the movement of the subject is well reflected in the image change amount without being affected by the movement of the camera.

The above is an explanation of the fourth embodiment of the processing for indexing a video using the representative image selection method of the present invention.

Next, a description will be given of a fifth embodiment of processing for indexing a video using the representative image selection method according to the present invention. FIGS. 11 and 12 are flowcharts of a representative image selection process according to the fifth embodiment.

The processing of the fifth embodiment is obtained by adding processing of mechanically selecting a representative image from representative image candidates to the first embodiment. This representative image selection processing is performed in steps S210 to S21 in the flowchart of FIG.
1 is replaced with the processing from step S610 to step S621 in FIGS. 11 and 12. Processing of other parts (steps S601 to S601)
Steps S609, S622 to S624) are the same as those in the first embodiment. Therefore, only the part relating to the representative image selection processing will be described with reference to the flowcharts in FIGS.

In the fifth embodiment, after selecting L representative image candidates in step S609 based on the minimum value of F [t], in step S610, F
The subscript t at which [t] is maximized is obtained and stored in the array TMAX. The method of finding the maximum of F [t] can be reduced to the problem of finding the minimum by changing the sign of F [t].

The following variables S and PMAX are used to search for a representative image candidate that is most appropriate as a representative image. In step S611, TMIN [1] is substituted for the variable S, and in step S612, 0 is substituted for the variable PMAX. In step S613, 1 is substituted for the variable k.

In step S614, TMAX [h] <
TMIN [k] and TMIN [k] <TMAX [h +
1] is checked for the existence of h. At the beginning and end of the sequence, h may not exist. If there is h that satisfies the condition, the process proceeds to step S615 to evaluate the suitability as a representative image. If there is no h that satisfies the condition, the evaluation of the suitability as the representative image is skipped, and the process proceeds to step S620.

At step S615, step S614
Is substituted into the variable H. Next, step S
In 616, TMAX [H] ≦ t ≦ TMAX [H + 1]
From the region that is, the dent X at t = TMIN [k]
Is obtained by the equation (1), and 1− | X−θ _x |
Store the value of. X is the section TMAX [h] ≦ t ≦ TM
AX [h + 1] represents the degree of F depression. If X is close to 0, the section is close to a stationary state and can be considered to have poor motion characteristics, and if X is close to 0.5 or 0.
If it is greater than 5, the section can only be considered moving and poor in motion characteristics. Closer to the middle of the appropriate value theta _x, since the compartment between can be regarded as rich in movement feature, 1- | X-θ _x | a and appropriateness of the representative image. θ _x is, for example, 0.25.

In step S617, the value of P and PMAX
To the value of. If P is larger than the appropriateness stored in PMAX, the process proceeds to step S618, where P
Replace the value of MAX. Otherwise, the process proceeds to step S620.

In step S618, SMIN is set to TMIN.
The value of [k] is substituted, and in step S619, PMAX
To the value of P. In step S620, the value of k + 1 is substituted for the variable k.

Next, in step S621, the value of the variable k is compared with the value of M. If k> M is not satisfied, step S61 is executed.
Then, the process returns to step 4 to evaluate the suitability of the next representative image candidate as a representative image. If k> M, escape the loop,
Proceed to step S622. Proceeding to step S622,
The variable S stores one frame number of a representative image candidate having the highest suitability as a representative image. Subsequent processing is the same as the processing after step S212 in FIG.

As an exceptional case, if the condition is not satisfied even in the inspection of the existence of h in step S614, the frame number of the youngest representative image candidate which is the initial value is stored in S. , Is used as the representative image. However, the present invention is not limited to this, and the representative image may be determined by another method.

Further, in the present embodiment, one representative image candidate having the maximum suitability as a representative image is selected. However, a plurality of representative images having higher suitability are preferentially selected. You may.

Further, even if the denominator of equation (1) is changed and X is defined as follows, there is essentially no difference, and the present invention can be applied effectively.

[0089]

(Equation 5)

Here, max (a, b) is a function that takes the larger value of a and b.

The above is an explanation of the fifth embodiment of the processing for indexing a video using the representative image selection method of the present invention.

Next, a description will be given of a sixth embodiment of the processing for indexing a video using the representative image selection method according to the present invention. FIGS. 13 and 14 are flowcharts of the representative image selection process according to the sixth embodiment.

In the processing of the sixth embodiment, the processing corresponding to step S616 in the flowchart of FIG. 12 in the processing of evaluating the suitability as a representative image of the fifth embodiment is replaced with the steps shown in FIG. It is replaced by the processing of S716, and the suitability as a representative image is evaluated based on the ratio between the minimum value of the image change amount and the maximum value on both sides thereof. Other than this processing (step S701)
To S715, S717 to S724) are the same as those in the fifth embodiment, and therefore only this process will be described with reference to the flowcharts in FIGS.

In the sixth embodiment, after the value of h is stored in the variable H (step S715), in step S716, from the area where TMAX [H] ≦ t ≦ TMAX [H + 1], t = TMIN [k ] Is determined as a representative image and stored in a variable P. To find P,
For example, P = p ₁ p ₂ p ₁ = F [TMAX [H]] / F [TMIN [k]] p ₂ = F [TMAX [H + 1]] / F [TMIN
[K]]. P need not be linear, and may be non-linear, such as P = p ₁ p ₂ ... Other when P = 0... P ₁ <θ ₁ or p ₂ <θ ₂ . θ ₁ and θ ₂ are predetermined thresholds. Step S716 is a process of evaluating whether or not the local minimum value at t = TMIN [k] is sufficiently smaller than the local maximum values on both sides. The smaller the value, the larger the value of P. Next, the process proceeds to step S717, and the subsequent processing is the same as the processing after step S617 in FIG.

The above is the description of the sixth embodiment of the processing for indexing a video using the representative image selection method of the present invention. By using the present invention, it is possible to select a representative image by mechanical processing, without going through the step of visually selecting and selecting a representative image by an operator. In the fifth and sixth embodiments, examples have been described in which each process for evaluating the suitability as a representative image is used individually, but both processes may be used simultaneously. For this purpose, for example, P = p ₁ p ₂ X, etc., so that both p ₁ , p ₂ , and X are reflected in the appropriateness as the representative image.

When the distance between the minimum point of interest and the next maximum point is larger than a predetermined threshold value or is smaller than a predetermined threshold value, such a minimum point is regarded as a representative image. A particularly low value such as 0 may be set for P to make it difficult to select.

In addition, operations that do not change the meaning, such as multiplying or adding a constant to X, p _1, or p ₂ , or converting with a monotonically increasing function, may be performed, or combined with other evaluation scales. It goes without saying that it may be used.

Here, an example of a method of processing for finding the minimum point of the sequence F [t] stored in the array will be described with reference to the flowchart of FIG. This processing corresponds to step S in FIG.
206 corresponds to the details. At the start of this processing, a sequence of image change amounts of the video section i of interest is stored in the array F. It is assumed that the number of frames in the video section is ae. Therefore, the number of elements of the array F is ae-1.

First, in step S801, the array TMI
Initialize N. In step S802, 1 is substituted for a variable m, and in step S803, 0 is substituted for a variable D. The variable D is a variable that stores the behavior of the immediately preceding F,
A value of 1 indicates an increase and a value of -1 indicates a decrease. A value of 0 indicates neither.

Next, in step S804, the variable g is set to 1
Is assigned. The variable g is a variable that stores a frame number for scanning the frame of the video section i.

In step S805, F [g + 1] and F [g + 1]
[G] is compared, and if F [g + 1]> F [g], that is, if F is increasing, the process proceeds to step S806. Otherwise, the process proceeds to step S812.

In the step S806, the value of the variable D is -1.
Judge. If the value of the variable D is -1, since F has decreased immediately before and is currently increasing, it is determined that F has a minimum value at t = g, and the process proceeds to step S807. Otherwise, the process proceeds to step S809.

In step S807, the value of the frame number g is substituted for TMIN [m]. This is a process of storing the minimum value found at the m-th position in the array TMIN. Reaching step S807 means that the sequence F is in ascending order in the frame of interest, and before that, it is in a state of neither descending order,
It can be regarded as a point changed from descending order to ascending order. Therefore, this is extracted as a minimum point. Subsequently, in step S808, the value of m + 1 is substituted for the variable m.

In step S809, 1 is substituted for a variable D. In step S810, the value of g + 1 is substituted for the variable g.

Next, in step S811, the value of the variable g is compared with the value of ae-2, and if g ≧ ae-2, the procedure ends. If not g ≧ ae-2, step S8
Returning to step 05, processing is performed for the next frame.

In step S805, F [g + 1]>
If it is determined that it is not F [g], step S812
In step S81, the value of F [g + 1] is compared with the value of F [g], and if F [g + 1] <F [g], the process proceeds to step S81.
Go to step S3, otherwise go to step S810.
In step S813, -1 is substituted for the variable D, and thereafter, the process proceeds to step S810. When this procedure is completed, the frame number at which the sequence F becomes the minimum is stored in the array TMIN, and the number of the minimum points is m-1.

Usually, the point at which the sequence changes from descending to ascending order is referred to as a minimum point. If there is a section having a constant value between the section in which the series is descending and the section in which the series is ascending, this is referred to as the minimum point. Whether or not to consider it may be arbitrarily defined by the practitioner. The present embodiment is configured based on the definition that this is regarded as a minimum point. If this point is regarded as the minimum point, the implementer may arbitrarily define which one point in the section where this value is constant is regarded as the minimum point. The present embodiment is configured based on the definition that the end of the section in which this value is constant is regarded as the minimum point.

The above processing can be realized by a computer and a software program, and the program is stored in an appropriate recording medium such as a portable medium memory, a semiconductor memory, and a hard disk which can be read by a computer. By reading from the computer.

[0109]

According to the present invention, when a representative image is extracted from a video section, a representative image that well represents the movement of a subject can be selected. In addition, by the method according to the present invention, an appropriate representative image can be automatically selected from the representative image candidates. In this case, the representative image is selected without the operator's visual observation and judgment. It becomes possible.

If an index of a video is created using the representative image selected in this way, there is an effect that an index in which the contents of the video can be easily understood can be created.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration example of an apparatus for implementing the present invention.

FIG. 2 is a flowchart of a representative image selection process according to the first embodiment.

FIG. 3 is a diagram illustrating an example of a video section to be processed;

FIG. 4 is a diagram showing a graph representing an image change amount in a video section.

FIG. 5 is a diagram illustrating a correspondence relationship between an image change amount and a video section.

FIG. 6 is a diagram illustrating an example of a video index.

FIG. 7 is a flowchart of a representative image selection process according to the second embodiment.

FIG. 8 is a flowchart of a representative image selection process according to the third embodiment.

FIG. 9 is a flowchart of a representative image selection process according to a fourth embodiment.

FIG. 10 is a flowchart of a representative image selection process according to a fourth embodiment.

FIG. 11 is a flowchart of a representative image selection process according to a fifth embodiment.

FIG. 12 is a flowchart of a representative image selection process according to a fifth embodiment.

FIG. 13 is a flowchart of a representative image selection process according to a sixth embodiment.

FIG. 14 is a flowchart of a representative image selection process according to a sixth embodiment.

FIG. 15 is a flowchart of a process for obtaining a minimum point.

[Explanation of symbols] 1 Video indexing device 11 Information processing device 12 Display device 13 Input device 14 Storage device 111 Image variation calculation means 112 Image change minimal frame calculating means 113 Representative image candidate selection means 114 Representative image selection means 401 Image change amount in t-th frame (F [t]) 501-506 1st-aeth frame

   ────────────────────────────────────────────────── ─── Continuation of front page    (72) Inventor Haruhiko Kojima             2-3-1 Otemachi, Chiyoda-ku, Tokyo Sun             Within the Telegraph and Telephone Corporation F term (reference) 5B075 ND12 NS01 UU40                 5C052 AA01 AB02 AB04 AC08 DD04                       EE03                 5C053 FA14 GB05 HA29 KA01 KA24                       LA06 LA11                 5L096 CA02 FA00 GA19 HA04

Claims (16)

[Claims] 1. A method for selecting a representative image in a video section, comprising: calculating an image change amount in a video section; obtaining a frame in which the image change amount is minimal; Selecting a candidate of a representative image of the video section based on a frame having a minimum value. 2. The representative image selection method according to claim 1, wherein in the step of selecting a representative image candidate of the video section, a frame having a minimum image change amount is set as a candidate of the representative image of the video section. Method. 3. The step of selecting a candidate of a representative image of the video section includes the steps of selecting a plurality of frames based on a frame having a minimal image change amount, and forming a strobe image from the selected plurality of frames. When,
Setting the created strobe image as a candidate for a representative image of the video section. 4. The step of calculating an image change amount in the video section includes the steps of estimating a camera movement parameter from an input video, and transforming each frame of the input image so as to cancel the camera movement using the camera movement parameter. 2. The method according to claim 1, further comprising the step of: 5. A step of obtaining appropriateness as a representative image by a predetermined evaluation criterion for each of the representative image candidates selected in the step of selecting a representative image candidate in the video section; Selecting a representative image from the representative image candidates based on adequacy. 6. In the step of determining the suitability as a representative image, f represents an image change amount, t ₁ and t ₂ represent frame numbers at which f is maximum on both sides of the representative image candidate frame,
When θ _x is a predetermined constant, the value X, Look, | X-θ _x | of what value is smaller, the representative image selection method according to claim 5, wherein providing the appropriateness of a larger representative image. 7. In the step of determining the suitability as a representative image, f represents an image change amount, t ₀ represents a frame number of the representative image candidate, and t ₁ and t ₂ represent f ₂ on both sides of the representative image candidate frame. Is the maximum frame number
A larger value of (t ₁ ) / f (t ₀ ) is given greater suitability as a representative image, and f (t ₂ ) / f
6. A representative image selecting method according to claim 5, wherein a value having a larger value of (t ₀ ) is given a greater suitability as a representative image. 8. A representative image selecting device for selecting a representative image in a video section, comprising: means for calculating an image change amount in a video section; means for obtaining a frame in which the image change amount is minimal; Means for selecting a candidate of a representative image of the video section based on a frame having a minimal image change amount. 9. The representative image selection unit according to claim 8, wherein said means for selecting a candidate of a representative image of the video section sets a frame having a minimum image change amount as a candidate of a representative image of the video section. apparatus. 10. A means for selecting a representative image candidate in the video section, means for selecting a plurality of frames based on a frame having a minimal image change amount, and means for creating a strobe image from the selected plurality of frames. 9. The representative image selection device according to claim 8, further comprising: a unit that sets the created strobe image as a candidate for a representative image of the video section. 11. A means for calculating an image change amount in a video section, comprising: means for estimating a camera movement parameter from an input video, and transforming each frame of the input image so as to cancel camera movement using the camera movement parameter. 9. The representative image selection device according to claim 8, further comprising: 12. A means for determining the suitability of each representative image selected by the means for selecting a representative image of the video section as a representative image based on a predetermined evaluation criterion. 9. The representative image selecting apparatus according to claim 8, further comprising: means for selecting a representative image from the representative image candidates based on suitability. 13. The means for determining the suitability as a representative image includes f representing an image change amount, t ₁ and t ₂ representing frame numbers at which f is a maximum on both sides of the representative image candidate frame, and θ _x
Is a constant, the value X, Look, | X-θ _x | of what value is smaller, the representative image selection apparatus according to claim 12, wherein providing the appropriateness of a larger representative image. 14. The means for determining the suitability as a representative image includes f representing an image change amount, t ₀ representing a frame number of the representative image candidate, and t ₁ and t ₂ representing f at both sides of the frame of the representative image candidate. Is the maximum frame number, f (t ₁ ) /
f (t ₀₎ value gives the appropriateness of a large representative image by what is more _{large, f (t 2) / f} (t 0) of the value that gives the appropriateness of a large representative image by a larger one 13. The representative image selection device according to claim 12, wherein: 15. A representative image selection program for causing a computer to execute the representative image selection method according to any one of claims 1 to 7. 16. A recording medium for a representative image selection program, wherein a program for causing a computer to execute the representative image selection method according to any one of claims 1 to 7 is recorded.