JP2003298981A - Digest image generating apparatus, digest image generating method, digest image generating program, and computer-readable storage medium for storing the digest image generating program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a digest image generating apparatus and method therefor capable of generating a proper digest image in response to a category. <P>SOLUTION: A plurality of feature extract sections 14, 16, 18, 20, 22 extract features of a plurality of kinds different from each other such as scene changes, a camera motion, the luminance and color of a designated area, a telop, and the sound from an original image, read a scene extract condition corresponding to a selected category from a library 26, produce a play list listing up feature information of scenes with a feature for satisfying the condition, and select a particular scene from the original image according to the play list to generate a digest image. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a summary image creating device, a summary image creating method, a summary image creating program, and a computer-readable storage medium storing the summary image creating program. Abstract image creating device for creating an image, abstract image creating method,
The present invention relates to a summary image creating program and a computer-readable storage medium storing the summary image creating program.

[0001]

2. Description of the Related Art Japanese Laid-Open Patent Publication No. 10-507554, Japanese Laid-Open Patent Publication No. 10-232884, and Japanese Laid-Open Patent Publication No. 11-751.
Japanese Patent Laid-Open No. 46-46 discloses a device for creating a summary image from a moving image in order to confirm the outline of a recorded TV program or movie in a short time.

[0002]

However, these devices may not be able to create an appropriate summary image depending on the program category such as news, music, sumo, etc. Although the above-mentioned JP-A-10-232884 and JP-A-11-75146 suggest that an appropriate summary image be created according to the category, no specific method is described.

An object of the present invention is to provide a summary image creating apparatus capable of creating an appropriate summary image according to a category, a summary image creating method, a summary image creating program, and a computer readable storage of the summary image creating program. Another storage medium is to provide.

[0004]

A summary image creating apparatus according to the present invention is a summary image creating apparatus for creating a summary image from a moving image, and includes one or a plurality of feature extracting means and a scene extracting means. The one or a plurality of feature extracting means extracts one or a plurality of types of features from the video or audio signal of the moving image. The scene extraction unit refers to the features extracted by the feature extraction unit and extracts a scene having a feature specific to the category of the moving image from the video or audio signal.

A summary image creating method according to the present invention is a summary image creating method for creating a summary image from a moving image, which is feature extraction for extracting one or a plurality of types of features from a video or audio signal of the moving image. And a scene extraction step of extracting, from the video or audio signal, a scene having characteristics specific to the category of the moving image with reference to the extracted characteristics.

A summary image creating program according to the present invention is
The computer is made to execute the above steps.

A computer-readable storage medium according to the present invention stores the summary image creating program.

The summary image creating apparatus, the summary image creating method, the summary image creating program, and the computer-readable storage medium storing the summary image creating program include:
One or a plurality of types of features are extracted from the video or audio signal of the moving image, and a scene having features unique to the category of the moving image is extracted by referring to the extracted features.

As a result, according to the present invention, it is possible to create an appropriate summary image according to a category even if the moving image has various categories.

[0010]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described in detail below with reference to the drawings. In the drawings, the same or corresponding parts will be denoted by the same reference symbols and the description thereof will be incorporated.

1. Configuration FIG. 1 is a block diagram showing a configuration of a summary image creating device according to an embodiment of the present invention. With reference to FIG. 1, the summary image creating apparatus creates a summary image from moving images stored in a data library 10. Specifically, the summary image creating apparatus includes an MPEG parsing unit 12 that decomposes an MPEG format video stream into a plurality of layers, and a scene change feature extraction unit that extracts scene change feature data related to a scene change from the video stream. 1
4, a camera motion feature extraction unit 16 that extracts camera motion feature data related to camera motion from the video stream, and designated area feature data related to luminance or color of a designated area in a frame of the video stream. Specified area feature extraction unit 18 to extract from inside
And a telop feature extraction unit 20 for extracting telop feature data related to the telop from the video stream,
The audio feature extraction unit 22 that extracts audio feature data related to audio from the video stream, and the features extracted by the plurality of feature extraction units 14, 16, 18, 20, and 22 are referred to And a scene extraction unit 24 for extracting a scene having a characteristic peculiar to the video stream and the audio stream.

The scene extraction unit 24 extracts a scene according to a condition library 26 for storing a plurality of scene extraction conditions corresponding to a plurality of categories and a scene extraction condition corresponding to a selected category among the plurality of categories. Scene identification information creation unit 28 that creates the scene identification information necessary to identify the desired scene, a playlist library 30 that stores a playlist that lists the created scene identification information, and the scene identification of the playlist. And a scene selection unit 32 that selects a scene specified according to information from a video stream and an audio stream.

2. Operation Next, the operation of this summary image creating apparatus will be described with reference to the flow chart shown in FIG.

The data library 10 stores digital data of recorded television programs such as news, music, baseball, drama, sumo, etc. as moving images. The data library 10 is actually formed on a DVD (digital versatile disk) or a hard disk of a computer.

First, the contents of the data library 10 are input to the summarized image creating apparatus (S1). The input content is MPEG encoded by an encoder (not shown).
Format is converted to a video stream (S
2), converted to a WAVE format audio stream. The video stream is input to the MPEG parsing unit 12 and decomposed into a plurality of layers such as I frame and P frame (S3). The I frame is input to the scene change feature extraction unit 14, the designated area feature extraction unit 18, and the telop feature extraction unit 20 (S4). The P frame includes a camera motion feature extraction unit 16 and a telop feature extraction unit 2
It is input to 0 (S4). The audio stream is input to the audio feature extraction unit 22 (S4). Then, a plurality of different types of features are extracted from the video stream and the audio stream by these feature extraction units (S5). The operation of each feature extraction unit will be described below.

(1) Scene Change Feature Extraction The scene change feature extraction unit 14 uses the DCT of the I frame.
(Discrete cosine transform) A luminance histogram is created based on the coefficients. Specifically, the DC component representing the average luminance of this block is extracted from the 64 DCT coefficients of the block of 8 dots × 8 dots. Then, a histogram of luminance in each frame is created. Then, the difference between the luminance histogram of each frame and the luminance histogram of the next frame is calculated, and the variance of the difference in the time direction is calculated. This distribution is the scene change feature data, and tends to increase when the scene changes significantly.

(2) Camera Motion Feature Extraction The camera motion feature extraction unit 16 affine-transforms the motion vector of the P frame to calculate camera motion feature data. From this camera motion feature data, determine the camera movement (motion) such as whether the camera that took the image is stopped, rotating up / down / left / right, zooming in, zooming out, etc. can do.

(3) Designated Area Feature Extraction The designated area feature extraction unit 18 uses the DCT coefficient of the I frame to set the luminance or color value in a predetermined range in one or more areas designated in advance in the frame. Specified area feature data effective for determining whether or not the area is inside is calculated.

FIG. 3 shows an example of a sumo wrestling screen.
In the case of the sumo category, it is necessary to incorporate such an action screen in the summary image. In sumo wrestling screens, the audience seats are generally displayed on the upper three-fifths of the screens, and the ring is projected on the lower two-fifths of the screens. Therefore, there are two designated areas 3 above and below in the frame.
Set 4, 36. In the approach screen, the upper designated area 34 tends to be dark because the audience seats are projected, and the lower designated area 36 tends to be bright because the ring is projected. Therefore, in the upper designated area 34, the proportion of pixels having low luminance is 80% or more, and in the lower designated area 36.
Then, if the ratio of pixels with high brightness is 90% or more, the screen can be determined to be the approach screen.

FIG. 4 is a flow chart showing a specific process for extracting a specified area feature.

First, it is determined whether all the pixels in the frame have been checked as described below (S10). If the check has not been completed for all pixels, the position of a pixel in the image and its brightness value are input (S1).
1).

Then, it is determined whether or not the below-described check has been performed for all designated areas (two designated areas 34 and 36 in the example of FIG. 3) in the frame (S12). If all the designated areas have not been checked, it is checked whether the position of the input pixel is within the designated area (S13). If it is not within the designated area, step S12
, And if there is, go to the next step S15 (S1
4).

If the position of the input pixel is within the designated area, it is checked whether the brightness value of the pixel is within a predetermined range (S15). In the case of the screen shown in FIG. 3, a predetermined range is set in the designated area 34 on the upper side around a low luminance value that is typical for a spectator seat, and in the designated area 36 on the lower side, a high brightness that is typical for a ring. Set a predetermined range around the value. If the brightness value is not within the predetermined range, step S12.
Returning to step S17, if it is within the predetermined range, the process proceeds to the next step S17.

Then, one pixel count is added (S17). Thus, it is possible to count the number of pixels having the brightness value within the predetermined range within the designated area.

If it is determined in step S12 that the above-mentioned checks have been completed for all designated areas, step S1
Returning to 0, and when it is determined in step S10 that the above-described check has been completed for all pixels in the frame, the process proceeds to the next step S18.

Next, the ratio of the number of pixels counted above to the total number of pixels in the designated area is calculated for each designated area (S18). Then, it is determined whether the calculated ratio of the number of pixels is equal to or more than a predetermined threshold value (S19). If it is equal to or more than the threshold value, the flag assigned to the designated area is set to "1" (S20), and if it is not equal to or more than the threshold value, the flag is set to "0" (S21).

Then, it is determined whether or not the number of frames in which flag = 1 continues is equal to or more than a predetermined threshold value (S22). If it is equal to or more than the threshold value, it is determined to be the start frame of the scene to be extracted (S23). On the other hand, if it is not equal to or more than the threshold value, it is determined whether or not the number of consecutive frames of flag = 0 is equal to or more than a predetermined threshold value (S24). If it is equal to or more than the threshold value, it is determined that it is the end frame of the scene to be extracted (S25).

Although the luminance information is used in the above example, color information may be used together with the luminance information or instead of the luminance information.

(4) Telop Feature Extraction The telop feature extraction unit 20 outputs telop feature data effective for determining whether or not there is a telop on the screen based on the DCT coefficient of the I frame and the motion vector of the P frame. calculate.

FIG. 5 is a flowchart showing a specific operation for extracting telop features.

First, it is determined whether or not the processing described below has been performed for all macroblocks in the frame (S3).
0). As shown in FIG. 6, in MPEG, one image is composed of a plurality of macroblocks. Each macroblock is composed of 16 dots × 16 dots.

If the processing of all macroblocks has not been completed, the DCT coefficient of the macroblock is input from the I frame (S31). As shown in FIG. 6, each macroblock is composed of four blocks. Each block is composed of 8 dots × 8 dots. In MPEG, the actual color is obtained by decoding 64 (= 8 × 8) DCT coefficients in each block.

Subsequently, the variance of 63 DCT coefficients excluding the DCT coefficient of the DC component (assigned to the dot in the upper left corner on the block shown in FIG. 6) is calculated for each block (S32). The variance of such DCT coefficient is 4
Calculate for one block.

Then, the maximum value is selected from the calculated four variances (S33). If the maximum value is larger than the predetermined threshold value, the flag assigned to the macroblock is set (S34). Since characters of one color or several colors are often displayed in the telop portion, the maximum value of the variance calculated here tends to increase.

If it is determined in step S30 that the above processing has been completed for all macroblocks, the ratio of flags in the designated area is calculated (S35). Since the telop is often displayed in the lower center of the screen, the designated area is set around that.

Then, if the flag ratio is higher than a predetermined threshold value, the flag assigned to the designated area is set (S36).

Then, the variance of the X-coordinate (horizontal direction of the screen) of the flagged macroblock is calculated (S37). Since the telop often extends in the horizontal direction of the screen, the variance of the X coordinate calculated here tends to be large.

Subsequently, the variance of the calculated variance in the time direction is calculated and it is checked whether or not it is stable (S3).
8). Since the telop is often displayed continuously for a certain period of time, the dispersion in the time direction calculated here tends to be stable.

Then, the flag of the designated area is set, and
It is determined whether or not the dispersion in the time direction is stable (S39). If the flag is set and the dispersion is stable, it is determined that the frame is the telop start frame, and the frame is held (S40). On the other hand, if not, it is determined whether or not the start frame is held (S41). If the start frame is held, it is determined to be the end frame of the telop, and the start frame and the end frame are output (S42).

(5) Speech Feature Extraction The speech feature extraction unit 22 determines the frequency and the variance of the voice obtained from the audio stream, the amplitude and the variance thereof, and the like.
The voice feature data effective for specifying the type of voice (conversation, music, etc.) is calculated.

As a result of the above-mentioned feature extraction, as shown in FIGS. 1 and 2, the feature extraction units 14, 16, 18, 20, 22 are extracted.
The respective characteristic data are output from (S6).

Then, the category of the moving image is selected according to the user's operation (S7).

FIG. 7 shows the contents of the condition library 26. The condition library 26 stores a plurality of scene extraction conditions corresponding to a plurality of categories. For example, in the category of news, it is necessary to create a summary image corresponding to a news headline. In this example, as the scene extraction conditions, "the voice contains a human voice" in relation to the voice feature data, "there is no scene change" in relation to the scene change feature data, and the camera motion feature data "There is no motion" in relation to ", and" that the telop appears in the latter half of the section on the lower side of the screen "is stored in relation to the telop feature data.

From such a condition library 26, scene extraction conditions corresponding to the category selected in step S7 are read.

Then, the scene specifying information creating section 28 satisfies the scene extraction conditions read from the condition library 26 based on the feature extraction data provided from the feature extracting sections 14, 16, 18, 20, 22. Create the scene identification information needed to identify the scene. As the scene identification information, the time from the first frame to the corresponding frame,
A frame serial number or the like can be used. The created scene identification information is sequentially recorded in the playlist library 30, and as a result, the playlist that identifies the scenes to be included in the summary image is completed. The play list is an index file that does not include image content and records indexes of frames forming a summary image.

When the summary image is actually created, the playlist is output from the playlist library 30,
The scene selection unit 32 selects a specific scene from the moving images according to this playlist (S9).

3. Other Embodiments In the above-described embodiments, the category is selected according to the user's operation. Since the user usually knows the category of the moving image, it is possible to create an appropriate summary image by allowing the user to select the category. However, when a plurality of categories of images are included in one moving image, the user has to select each category, which is troublesome.

Therefore, step S71 shown in FIG. 8 may be added instead of step S7 shown in FIG.
Here, the category is automatically selected.

FIG. 9 shows a method of automatically selecting a category. First, a plurality of types of features are extracted from a moving image by the same method as above. Since this moving image includes images of a plurality of categories, a characteristic peculiar to each category is extracted. These extracted features have a plurality of parameters peculiar to the time record and the feature extraction, and when the number of parameters is N, it can be replaced with an N-dimensional feature vector. It should be noted that one or a plurality of feature vectors serving as templates for each category are prepared in advance from the feature extraction results extracted from past programs and the like for programs in various categories and stored in a database.

A feature vector of a certain section of an arbitrary moving image is created, the distance and correlation coefficient of the feature vector in the database are obtained, and the feature vector having a short inter-vector distance or the highest correlation coefficient is selected. If the category of this feature vector is, for example, news, the category of this section can be determined to be news.

Further, according to the above-described embodiments, in most cases, an appropriate summary image can be created according to the category. However, even in the same category, there are cases where it is not possible to create an appropriate summary image because the characteristics extracted by the program vary.

Therefore, as shown in FIG. 10, the feature extraction conditions of each feature extraction unit are adjusted (S50), the features are extracted by the adjusted feature extraction conditions (S4), and the scene extraction conditions are further adjusted (S50). S51), the scene identification information may be created under the adjusted scene extraction conditions (S8). In FIG. 10, circled numbers indicate the order of processing.

In order to automatically adjust the feature extraction condition, the difference between the above-mentioned feature vector extracted by the initial feature extraction condition and one or a plurality of feature vectors in the database that are close to or have a high correlation is calculated. Then, the feature extraction conditions are finely adjusted so that the difference becomes small.

Next, the feature is extracted by using the finely adjusted feature extraction condition, the feature vector is obtained, and the feature vector in the database that is the closest or has a high correlation is selected. At this time, a slight difference remains, but this is stored as a feature vector difference. If the selected feature vector is, for example, news, the scene identification information is created using the news scene extraction condition. However, the scene extraction condition is created using the adjustment coefficient that is defined in advance based on the previously obtained feature vector difference. Fine-tune each parameter of.

The functions in the block diagram shown in FIG. 1 and the steps in the flow charts shown in FIGS. 2, 4, 5, 8 and 10 cause a computer to execute a predetermined program. Can be realized by This program can be recorded in a computer-readable recording medium such as a CD-ROM and distributed, but can also be distributed via the Internet or the like.

The embodiment of the present invention has been described above.
The embodiments described above are merely examples for carrying out the present invention. Therefore, the present invention is not limited to the above-described embodiments, and can be implemented by appropriately modifying the above-described embodiments without departing from the spirit thereof.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a summary image creating device according to an embodiment of the present invention.

FIG. 2 is a flowchart showing an operation of the image creating apparatus shown in FIG.

FIG. 3 is a diagram showing an example of a screen in which a designated area is set by a designated area feature extraction unit in FIG.

FIG. 4 is a flowchart showing an operation of a specified area feature extraction unit in FIG.

5 is a flowchart showing the operation of a telop feature extraction unit in FIG.

6 is an MPE used in the telop feature extraction shown in FIG.
It is a figure which shows the structure of the macroblock of G.

FIG. 7 is a diagram showing specific contents of a condition library in FIG.

FIG. 8 is a flowchart showing the operation of the summary screen creating apparatus according to another embodiment of the present invention.

9 is a diagram showing a method of automatic selection of categories in FIG.

FIG. 10 is a flowchart showing the operation of the summary screen creation device according to yet another embodiment of the present invention.

[Explanation of symbols]

10 data library 14 Scene change feature extraction unit 16 Camera motion feature extraction unit 18 Designated Area Feature Extraction Unit 20 Telop feature extraction unit 22 Speech feature extraction unit 24 Scene extractor 26 Condition Library 28 Scene Specific Information Creation Department 30 playlist library 32 Scene selection section 34,36 designated area

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Shingo Fujiwara             2-3-3 Minami 2-3, Chiyosaki, Nishi-ku, Osaka             OGIS Research Institute (72) Inventor Motohide Yasukawa             2-3-3 Minami 2-3, Chiyosaki, Nishi-ku, Osaka             OGIS Research Institute (72) Inventor Masahiro Inui             2-3-3 Minami 2-3, Chiyosaki, Nishi-ku, Osaka             OGIS Research Institute F term (reference) 5B075 ND12 NS01                 5C052 AA01 AC08 CC11 DD04                 5C053 FA06 FA14 FA23 GA11 GB37                       HA30

Claims (24)

[Claims] 1. A summary image creating apparatus for creating a summary image from a moving image, comprising one or a plurality of feature extracting means for extracting one or a plurality of types of features from a video or audio signal of the moving image, And a scene extracting unit for extracting a scene having a characteristic peculiar to the category of the moving image from the video or audio signal with reference to the characteristic extracted by the characteristic extracting unit. Image creation device. 2. The summary image creating apparatus according to claim 1, wherein one of the feature extracting means is a scene change feature extracting means for extracting a feature associated with a scene change from the video signal. A summary image creating device characterized by the above. 3. The summary image creating device according to claim 1, wherein one of the feature extracting means extracts a feature related to camera motion from the video signal. An abstract image creating device characterized by being extraction means. 4. The summary image creating device according to claim 1, wherein one of the feature extracting means relates to a luminance or a color of a designated area in a frame. A specified area feature extracting means for extracting the feature to be extracted from the video signal. 5. The summary image creating device according to claim 1, wherein one of the feature extracting means extracts a feature associated with a telop from the video signal. An abstract image creating apparatus characterized by being a telop feature extracting means for performing. 6. The summary image creating device according to claim 1, wherein one of the feature extracting means extracts a feature related to voice from the audio signal. A summary image creating apparatus characterized in that it is a voice feature extracting means. 7. The summary image creating device according to claim 1, wherein the scene extracting unit includes scene specifying information necessary for specifying a scene to be extracted. A summary image creating apparatus comprising: a scene specifying information creating unit to create; and a scene selecting unit selecting a scene specified according to the scene specifying information from the video or audio signal. 8. The summary image creating device according to claim 7, wherein the scene extraction means further includes a storage means for storing a plurality of scene extraction conditions corresponding to a plurality of categories, and the scene identification information. The means of creation is
A summary image creating apparatus, wherein the scene specifying information is created according to a scene extraction condition corresponding to a selected category among the plurality of categories. 9. The summary image creating device according to claim 8, wherein the scene extracting unit further selects one of the plurality of categories based on the features extracted by the feature extracting unit. An abstract image creating apparatus comprising a selecting means. 10. The summary image creating apparatus according to claim 1, further comprising: the feature extracting unit extracting the feature based on the feature extracted by the feature extracting unit. And a feature extraction condition adjusting means for adjusting a feature extraction condition applied when performing a summary image creating apparatus. 11. The summary image creating apparatus according to claim 8, further comprising a scene extraction condition adjusting unit that adjusts the scene extraction condition based on the feature extracted by the feature extracting unit. Summary image creation device. 12. A summary image creating method for creating a summary image from a moving image, comprising: a feature extracting step of extracting a plurality of different features from a video or audio signal of the moving image; A scene extraction step of extracting from the video or audio signal a scene having a characteristic peculiar to the category of the moving image with reference to the characteristic, a summary image creating method. 13. The method of creating a summarized image according to claim 12, wherein the feature extracting step extracts a feature associated with a scene change from the video signal. 14. The summary image creating method according to claim 12, wherein the feature extracting step extracts a feature related to camera motion from the video signal. Image creation method. 15. The method of creating a summarized image according to claim 12, wherein the feature extracting step includes extracting a feature related to a luminance or a color of a designated area in the frame. A method of creating a summarized image, characterized by extracting from the video signal. 16. The summary image creating method according to claim 12, wherein the feature extracting step extracts a feature associated with a telop from the video signal. A method for creating a featured summary image. 17. The method of creating a summary image according to claim 12, wherein the feature extracting step extracts a feature associated with voice from the audio signal. A method for creating a featured summary image. 18. The summary image creating method according to claim 12, wherein the scene extracting step includes scene specifying information necessary for specifying a scene to be extracted. A method of creating a summary image, comprising: a scene specifying information creating step to be created; and a scene selecting step of selecting a scene specified according to the scene specifying information from the video or audio signal. 19. The summary image creating method according to claim 18, wherein the scene specifying information creating unit stores the plurality of scene extracting conditions in association with a plurality of categories. Of the above categories, the scene specifying information is created according to a scene extraction condition corresponding to the selected category. 20. The summary image creating method according to claim 19, wherein the scene extracting step further includes a category selecting step of selecting one of the plurality of categories based on the extracted characteristics. A method for creating a summary image, which is characterized in that 21. The summary image creating method according to claim 12, further comprising a feature extraction condition applied when the feature is extracted based on the extracted feature. And a feature extraction condition adjusting step for adjusting the. 22. The summary image creating method according to claim 19, further comprising a scene extraction condition adjusting means for adjusting the scene extraction condition based on the extracted feature. Method. 23. A summary image creating program for causing a computer to execute the steps according to any one of claims 12 to 22. 24. A computer-readable storage medium storing the summary image creating program according to claim 23.