JPH0816610A - Animation retrieving system and animation retrieving data extraction method - Google Patents

Abstract

PURPOSE:To extract only a characteristic frame by applying both the switching of scenes and the level and the change of the sound volume as the standards for extraction of the retrieving data out of the AV data that includes the animation data and the voice data. CONSTITUTION:A scene switching detection part 1041 detects the scene switching of the AV data, and a sound volume detection part 1042 detects the sound volume out of the AV data. Then a retrieving data selection management part 1043 extracts the frames larger than a fixed sound volume level out of a detected scene as the retrieving data.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention allows a user to retrieve an image of a predetermined scene from a medium storing the image.
The present invention relates to a video search system and its method.

[0002]

2. Description of the Related Art In recent years, with the spread of multimedia systems, systems for handling AV data have been developed one after another. Here, the AV data refers to data having both moving image data representing moving images and audio data. AV
The mainstream data format is a format in which moving image data and audio data are managed in parallel on the basis of time. For example, Apple Computer's QuickTime puts video data and audio data in separate tracks, and while playing these tracks, it plays back video with audio.

First, encoding of moving image data will be described. The moving image data has an extremely large amount of information per unit time as compared with the audio data. However, in a continuous moving image, in general, the images before and after the image are very similar to the image of interest, and therefore when viewed as data, there is a great deal of redundancy. Therefore, it is possible to digitally perform high-efficiency encoding and significantly reduce the required amount of information. Currently, ISO (International Organization for Standardization) / IEC is used as a high-efficiency coding method.
MPEG (Moving Picture Cod) of (International Electrotechnical Commission)
The international standard draft MPEG1 established by the ing Experts Group) is the mainstream. Since this patent is for searching AV data, it is possible to handle a huge amount of AV data. Inevitably, the AV data may be highly efficient coded. Therefore, first, the MPEG1 encoding method will be described with reference to FIG.

As described above, in a continuous moving image, the preceding and following images are very similar to the image of interest. Therefore, in MPEG1, for example, when an image 10040 is to be encoded, the temporally preceding image 1001
The difference from 0 is taken, and the difference value is encoded to form a P picture (forward predictive encoded image) 10080. Further, when the image 10020 is to be encoded, the difference between the image 10010 in the front in time, the image 10040 in the rear, or the interpolated image formed from the front and the rear is calculated, and the difference value is encoded to B. A picture (bidirectional prediction coded image) 10060 is used. Similarly, when the image 10030 is to be encoded, the B picture 10070 is created from the images 10010 and 10040. By encoding in this way, the redundancy in the time axis direction is reduced and the amount of information is reduced. Further, instead of always encoding the difference, for example, by encoding the input image 10010 as it is,
It may be an I picture (intra-frame coded image) 10050.

Also, in MPEG1, not only the same difference in position is taken, but also motion compensation is used. This is because, in macroblock units, in the previous image of the image to be encoded, the block that has the smallest difference from the block to be encoded is searched for, and the difference from that is searched for to send further. This is a method of reducing data that cannot be obtained.
Here, the macro block is a block of 16 × 16 pixels. Actually, in the P picture, the one with a small data amount of the two, that is, the one that takes the difference from the prediction image after motion compensation and the one that does not take the difference, is selected and encoded in macro block units. But this still requires sending a lot of data about the part of the object that came out from behind. Therefore, in the B picture, one of the four is obtained not only by the difference between the decoded image and the interpolated image formed from the backward or both of the temporally forward direction after the motion compensation and the backward direction. Similarly, the one with the smallest data amount is selected and encoded in macro block units. This way most of the data need not be sent.

Also in MPEG1, moving picture data and audio data are put in separate tracks, and those tracks are simultaneously accessed to reproduce a moving picture accompanied by sound. Therefore,
For example, it is possible to extract moving image data corresponding to audio data at a certain time.

By the way, a system using such AV data, especially text, graphics, sound, etc.
In an authoring system that uses various data to create an application for presentation and the like, retrieval of AV data and its display are important problems.
This is because AV data requires a longer time to read data for reproduction than other data, and the AV data itself may have a longer reproduction time.

[0008] Conventionally, in a system for searching AV data, one of the mechanisms for extracting desired data is to use pre-registered browsing data individually or in accordance with a user's instruction. There was a browsing search that displayed a list of individual items. This search method is an excellent method for searching AV data because the content of the moving image can be grasped in a short time by looking at the browsing data.

By the way, a problem with the browsing search method is that it is necessary to set search data in advance, so that the setting work is very troublesome. In order to solve this problem, a scene switching occurrence portion is detected, one of frames before and after the scene switching is extracted as a search image into a search list, and a list is displayed at the time of search. JP-A-3-292257 There is a technology described in. Here, a frame is a still image of a minimum unit element that constitutes moving image data. A scene is defined as a set of frames in which a subject is continuously photographed. 2 which is different from the scene switching part
Refers to the connecting part of the scene. For example, a general television moving image is composed of 30 frames of still images for 1 second. In this method, it is also possible to extract a scene switching portion, which is detected next time, excluding a scene switching portion whose time is equal to or less than a certain time. If this method is used, one frame of a search image can be extracted for each scene and used as search data.

[0010]

However, in the technique described in the above publication, when the AV data has a large number of scenes, the number of images to be searched also increases, so that it takes time to search. There is a problem. In other words, merely using a frame in which scene switching has occurred as a search image is insufficient as a moving image search system and method.

An object of the present invention is to extract a plurality of frames of search images for each important scene and to extract fine search data for each important scene.

It is another object of the present invention to reduce the number of search data without extracting a search image from a scene that is not important.

[0013]

In order to solve the above-mentioned problems, according to the present invention, moving image data representing a moving image composed of a plurality of sequential image frames and the respective frames are dealt with. Sequential audio data to be input is input, and from the input moving image data, for each scene included in the moving image, at least one moving image data corresponding to the frame is searched. In a moving image search system for extracting as data, a scene switching detecting means for detecting a frame forming the scene for each scene based on the content of the inputted moving picture data, and a volume of the input audio data. For each scene detected by the corresponding frame, and for each scene detected by the scene switching detection means, a value equal to or more than a value predetermined by the sound volume detection means. At least one deflection volume is detected - Video de corresponding arm - the motor can be provided with a search data extracting means for extracting a search data for the respective scenes.

Further, moving image data representing a moving image composed of a plurality of sequential image frames and sequential voice data associated with each of the frames are input, and the input moving image data is input. Out of the
In a moving image search system for extracting moving image data corresponding to at least one frame as search data for each scene included in the moving image, each of the scenes is based on the content of the input moving image data. Each time, a scene switching detecting means for detecting a frame constituting the scene, a volume detecting means for detecting the volume of the input audio data for each corresponding frame, and a scene switching detecting means for detecting the volume. For each scene, based on the volume of each frame detected by the volume detecting means, a moving image corresponding to at least one frame in which the volume change with respect to a frame separated by a predetermined interval is a predetermined value or more. It is also possible to provide a search data extracting means for extracting the data as search data for each scene.

[0015]

First, the AV data will be described. The AV data is data including moving image data representing a moving image and audio data representing corresponding audio.

For the AV data, a scene switching occurrence portion and the volume of each frame are detected. Based on the detected information, necessary frames are extracted from the AV data, and search data is created and displayed.

That is, not only scene switching but also audio data is focused as a reference for generating moving image search data. The reason for this is that the high volume part and the high volume change part are considered to be more characteristic parts for AV data. The moving image search data is generated by selecting a necessary part for searching from the frames and combining them.

AV data is usually composed of a plurality of scenes, and the degree of importance of its contents varies depending on the reproduction time of each scene. Therefore, the moving image search data is extracted as follows according to the reproduction time of each scene.

From the scene having a short reproduction time, it is basically judged that it has no meaning and the search data is not extracted.
However, if the average sound volume of the scene is equal to or higher than a certain value, the search data is extracted for all the scenes, as having some meaning.

A scene having a long reproduction time is judged to have some meaning, and one characteristic portion is extracted. However, if it is desired to avoid a situation where only one characteristic portion is extracted even when the scene is very long, a plurality of characteristic portions can be extracted from the same scene according to a searcher's instruction. Furthermore, in this case, either of the following two functions can be selected.

(1) When it is desired to extract a plurality of characteristic portions, all frames having a certain volume or higher are extracted.

(2) When it is desired to extract only one characteristic part, the frame showing the maximum volume in the scene is extracted.

For a scene having an intermediate reproduction time, if the average sound volume is a certain value or more, it is assumed that it has some meaning, and the portion showing the maximum sound volume in the scene is extracted. Even if the average volume is less than a certain value, if there is a volume change portion that is equal to or more than a certain volume difference in the scene, it is determined to have some meaning, and the portion showing the maximum volume change in the scene is determined. Extract. The other scenes are judged to be meaningless and are not extracted.

The AV data is composed of a plurality of frames, and has volume characteristics such as maximum, minimum, and average of a certain size for each frame. To check the volume change in a scene, set the volume of a certain frame to
It must be compared to the volume of a different frame than that frame. Therefore, when investigating a change in volume in a scene, the procedure is as follows.

First, in the case of investigating a change in sound volume, if the immediately preceding frame is used as the object of difference detection, there are cases where the maximum sound volume difference cannot be accurately determined due to the influence of noise. Therefore, it is possible for the searcher to set how many frames before the volume is to be compared when the difference is detected.

Further, the detection of the maximum part of the volume change is carried out in the case where it is desired to detect the part in which the increase in the volume is maximum and in the part where the change in the volume is the largest, that is, the part where the absolute value of the volume difference is the maximum. There are two ways. The searcher can set which of these two methods is selected.

The search data is formed by combining the frames extracted from a plurality of characteristic scenes.
However, if the search data is too large when the search data is generated, the number of images that the searcher has to refer to increases, and the time required for the search also increases. Therefore, in order not to excessively extract the search data, the following is performed.

First, as for the number of frames constituting the extracted from each scene for extracting the search data, it is possible to set how many frames before and after the characteristic portion are extracted at the time of extraction so that the searcher can decide. To do.

If the search data is too large, the number of images that the searcher has to refer to increases, and the time required for the search also increases. Therefore, in order to reduce the amount of search data that is too large, the following is performed.

First, the searcher is allowed to set the maximum number of scenes that allow extraction of search data. If the number of scenes for which search data is extracted is greater than the number of set scenes, reduce the number of scenes for which search data is extracted,
Set less than the number of scenes. In this case, the searcher can set which of the scene with the smallest maximum volume and the scene with the smallest volume difference is deleted first.

Further, the searcher can set the maximum total number of frames that can be accepted as search data. If the total number of search data frames is larger than the set number of frames, the number of search data frames is reduced to a set number of frames or less.

In the above function of reducing the number of frames of the search data to the set number of frames or less, if the search data is too small, the contents may not be understood at the time of search. Therefore, to avoid this, 1
The minimum number of frames required for one scene can be set by the searcher so that the number of frames per scene is not too small and the contents cannot be understood.

[0033]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 is a block diagram of a moving image search data generation system according to an embodiment of the present invention. 1010 in FIG.
Is a main control unit that controls the entire moving image search data generation method of the present embodiment, and 1020 is an input unit that allows the user to instruct moving image file search from a keyboard or the like and to input parameters required when extracting the search data. Is. Reference numeral 1030 is a data storage management unit for recording and reading moving image files, 1040 is a search data generation unit for extracting characteristic portions in the moving image file, and 1050 is a search data generation unit 10.
A display unit for displaying search data managed by 40, 106
Reference numeral 0 is a communication unit for exchanging data between the above units.

In the above configuration, for example, the main control unit 1010 is a CPU, the input unit 1020 is a keyboard, the data storage management unit 1030 is a hard disk, the display unit 1050 is a CRT display, the search data generation unit 1040 is a ROM, The communication unit 1060 is a program stored in a RAM or the like and executed by a CPU, and can be configured by a well-known device by implementing the communication unit 1060 using a bus.

FIG. 11 shows an apparatus configuration diagram capable of realizing FIG. In FIG. 11, 11010 is a CPU (central processing unit), 11020 is a keyboard, 11030 is a hard disk, 11040 is a ROM (read only memory), 11050 is a display, and 11060 is a bus.

Returning to FIG. 1, 1031 and 1032 form a data storage management unit 1030. Reference numeral 1031 is a data storage unit that stores AV data, and 1032 is a data reading unit that reads data from the data storage unit 1031 under the control of the main control unit 1010 or the search data generation unit 1040. Reference numerals 1041, 1042, and 1043 configure a search data generation unit 1040. 1041 is a data reading unit 1
This is a scene switching detection unit that detects scene switching of the moving image file read out from 032. A volume detection unit 1042 detects the volume of each frame of the moving image file read from the data reading unit 1032. Reference numeral 1043 denotes a search data selection management unit that performs a predetermined process based on the information of the scene switching detection unit 1041 and the volume detection unit 1042, and manages the result, and holds the frame number or the frame data. .

First, the scene switching detection processing will be described. The scene switching detection processing can be performed using, for example, the correlation between two temporally consecutive frames and the information of the motion vector indicating the parallel movement of the entire image. Specifically, first, a two-dimensional small block is defined within one frame, and the correlation of each block (for example, the square of the error) between two temporally consecutive frames is determined to determine whether there is a correlation. Yes (square of error). When there is a correlation (the square of the error is small), it is considered as continuous scenes (scenes), and scene switching has not occurred. However, with this frame correlation alone, the scene caused by the parallel movement of the camera such as pan (horizontal movement) and tilt (vertical movement) will be judged as scene switching. A vector is detected, and it is determined whether or not the scene is switched based on the motion vector (continuous scene when the motion vector is detected). The detection of the motion vector can be obtained, for example, from the relationship between the spatial gradient of the frame image and the difference between the images. The method of detecting a motion vector is described in detail in the document "Digital signal processing of images: written by Takahiko Fukibe, Nikkan Kogyo Shimbun".

Here, a method of detecting a motion vector will be described. As a typical motion vector detection method,
There is a block matching method. The block matching method will be described below with reference to FIG. The block matching method uses one macroblock (16
This is a method of comparing a block of 16 pixels) with all macroblocks of the previous image and adopting the vector to the macroblock having the smallest image difference as the motion vector. For example, to obtain the motion vector of the macroblock 12010, the macroblock 12 read from the previous image
The positions of 020 to 12070 are gradually shifted (in this figure, they are shifted by 2 in the x-axis direction). Each read block is compared with the macro block 12010 to obtain the block with the smallest image difference (block 12040 in this figure). Block matching is performed for the number of blocks that can be candidates. In this figure, the vector (6,0) is adopted as the motion vector.

Therefore, for example, in the case of detecting a scene change in MPEG1, in the P picture, the number of blocks that have a difference from the motion-compensated predicted image and the number of blocks that do not have a difference are checked, and the difference is calculated. If the number of blocks is equal to or greater than a certain value, it is considered that the motion vector has been detected, and it may be determined that the scenes are continuous.

Therefore, it is judged that scene switching has occurred when there is no interframe correlation and no motion vector is recognized.

The volume detection processing will be described here. In the sound volume detection process, if the sound volume value for each frame is used as it is, it will be greatly affected by noise in each frame. Therefore, in order to remove noise for each frame, the volume value for each frame is, for example, three frames before and after the frame that is temporally continuous with the frame whose volume is to be obtained,
That is, the average value of a total of 7 frames including the frame for which the volume is to be obtained is taken and defined as the volume value of the frame. Of course, it is also possible to use a noise removal method other than this method.

The method of detecting the volume of each frame will be described below. Usually, in order to digitize an acoustic waveform, it is necessary to perform sampling, quantization, and encoding in order as shown in FIG. In sampling, the time is finely divided and the height of the waveform at that unit time is viewed. In quantization,
The height of the waveform is divided into a certain number of binary details and read. In encoding, an acoustic waveform is converted into a digital signal having a value obtained by quantization. In a general compact disc for music, the sampling frequency (when sampling, 1
The number of times per second is 44.1 kHz, and the number of quantization bits (fineness that separates the strength of sound when quantizing) is 16 bits.

Therefore, the volume of each frame can be obtained by defining, for example, the maximum value 14010 as the volume of the frame in the coded numerical value group within one frame time as shown in FIG. Of course, within the time corresponding to one frame, the average value or minimum value 14020 of the coded values may be defined as the volume of that frame.

In this embodiment, as the scene switching detection method and the volume detection method, the above-mentioned interframe correlation and motion vector,
Use the frame volume definition. FIG. 8 is a scene feature table that stores information for each scene that is necessary in the process of generating search data. In each row of the table, the scene end frame number, scene length, average volume, maximum volume frame address, maximum volume, maximum volume difference frame address, and maximum volume difference of each scene are stored. By realizing the table by the list, the size of the table can be arbitrarily changed.

The operation example of the block diagram of FIG. 1 will be described below with reference to the flowchart of FIG. FIG. 2 is a flowchart showing the operation of the moving image search data generation method. First, when the moving image file search process is started by an instruction from the input unit 1020, the searcher selects the moving image file to be searched by the input unit 1020 (step 2010). In order to search the instructed moving image file, the main control unit 1010 instructs the search data generation unit 1040 to create search data for the moving image file. In the search data generation unit 1040, the search data selection management unit 1043
Instructs the data reading unit 1032 to read the moving image file (step 2020). The search data selection management unit 1043 requests scene switching occurrence information and volume information of each frame to the scene switching detection unit 1041 and the volume detection unit 1042 in order to clarify the characteristic portion of the moving image file, and FIG. The scene feature table shown in is created (step 2030). Search data selection management unit 10
43 is based on the information obtained from the scene feature table,
The characteristic part in the moving image file is extracted to the extraction table of FIG. 9 (step 2040). In each row of the extraction table, the maximum volume of each scene, the maximum volume difference, the start frame number, and the end frame number of each scene that make up the search data are stored. By realizing the table by the list, the size of the table can be arbitrarily changed. The search data selection management unit 1043 selects search data from the extraction table based on the condition given by the input unit 1020 (step 205).
0). Finally, the main control unit 1010 instructs the display unit 1050 to display a list of search data, and the display unit 1050 displays a list of frames from the start frame number to the end frame number of each scene stored in the extraction table ( Step 2060). Data exchange between these units is performed via the communication unit 1060.

Regarding the scene feature table creation process, feature portion extraction process, and search data selection process in the flowchart of FIG. 2, flowcharts showing respective processing content examples are shown in FIGS. 3, 5, and 7. Further, FIG. 4 shows the scene change detection processing in the flowchart of FIG. 3, and FIG. 6 shows the same scene plural feature portion extraction processing in the flowchart of FIG. Hereinafter, with respect to the flowcharts of FIGS. 3 to 7, examples of processing contents thereof will be sequentially described.

An operation example of the scene feature table creation process (step 2030) shown in FIG. 3 will be described. First, when the volume difference between two frames is detected, the volume difference detection target frame number α that specifies the difference with the volume of the previous frame so that the volume difference with an arbitrary relative frame can be detected. , Is specified by the input unit 1020. Also, α
When detecting the frame with the largest volume difference from the previous frame, whether the frame with the largest volume increase or the volume change, that is, the frame with the largest absolute value of the volume difference is taken. Input section 1020
Specify by. (Step 3010). The current frame number for counting the number of frames from the first frame of the moving image file is set to 0 (step 3015). Read the first frame of the specified file (step 302
0), as a default value, 1 is set to a frame counter that counts the number of frames in one scene, and 0 is set to a variable total volume that represents the total volume of each frame in one scene. Variable maximum volume indicating where the frame is located The current frame number is set in the frame number, 0 is set in the variable maximum volume indicating the volume of that frame, and where the maximum volume difference from the α frame before is in one scene. The current frame number is set to the variable maximum volume difference frame number indicating that or, and 0 is set to the variable maximum volume difference indicating the difference between the volume of that frame and the frame before the α frame (step 3030). Add 1 to the current frame number (step 3035). It is checked whether the volume of the frame currently read is higher than the maximum volume (step 304).
0) If larger, replace the maximum volume frame number with the current frame number and the maximum volume with the volume value of the current frame (step 3050).

Next, it is necessary to check whether or not the frame before the α frame exists in the same scene before taking the volume difference from the frame before the α frame, so whether the value of the frame counter is α + 1 or more. It is checked (step 3060). When α + 1 or more, the volume can be compared with the volume before the α frame. Therefore, after detecting the volume before the α frame (step 3070), the frame with the largest increase in volume is taken as the maximum volume difference frame. Is checked whether the maximum volume difference is smaller than “(current frame volume) − (volume before α frame)” (steps 3080, 3090). When the frame having the maximum absolute value of the volume difference is taken as the maximum volume difference frame, it is checked whether it is smaller than the absolute value of “(current frame volume) − (volume before α frame)” (step 3080, 3100). If it is small, the current volume difference is the maximum volume difference, so the maximum volume difference frame number is the current frame number, and the maximum volume difference is the absolute value of "(current frame volume)-(volume before α frame)". , Respectively (step 3110). Further, if the value of the maximum volume difference is smaller than 0 (step 3120), α is subtracted from the maximum volume difference frame number (step 313).
0). This is because when the difference in the volume decreasing direction is taken, it is desired to extract the frame at the beginning of the decrease as the volume decreasing frame.

Next, the volume of the current frame is added to the total volume (step 3140) to check whether or not the next frame exists. If the next frame exists, the next frame is read (step 3160), and the scene switching detection process (step 3170) is performed to see if the scene switching flag is set (step 3180). If the flag is not set, add 1 to the frame counter (step 3
210) and add 1 to the current frame number again (step 3035). If the flag is set, a new line of the scene feature table 8010 shown in FIG. 8 indicates the current frame number as the scene end frame number 8015, the frame counter value as the scene length 8020, and the average volume 8030 as "( Total volume) / (frame counter) "
, The value of the variable maximum volume frame number as the maximum volume frame number 8040, the value of the variable maximum volume as the maximum volume 8050, the value of the variable maximum volume difference frame number as the maximum volume difference frame number 8060, and the maximum volume difference The value of the variable maximum volume difference is registered as 8070 (step 3190). If the end flag is set, the scene feature table creation processing is ended.
The process is continued by returning to the initial setting (step 3030) again. The operation example of the scene feature table creation process (step 2030) in FIG. 3 has been described above.

Next, an operation example of the scene change detection process (step 3170) shown in FIG. 3 will be described with reference to FIG.
First, it is checked whether there is a correlation between the previous frame and the current frame (step 4010) or whether there is a motion vector (step 4020). If there is no correlation or motion vector between the previous frame and the current frame, It is judged that the scene switching has occurred, and a scene switching flag is set (step 4
030), the scene switching detection process ends. If at least one of the correlations or motion vectors is detected, it is determined that the previous frame and the current frame are in the same scene, and the scene switching flag is lowered (step 40).
40), the scene switching detection process ends. The operation example of the scene switching detection process (step 3170) in FIG. 4 has been described above.

Next, an operation example of the characteristic portion extraction processing (step 2040) shown in FIG. 2 will be described with reference to FIG. First, scene lengths a, b, sound volumes c, d, e and maximum sound volume difference f are set by the input unit 1020 as parameters for selecting characteristic portions (step 5010). In addition, the input unit 1020 designates how many frames before and after the characteristic frame are extracted as the characteristic portion by the variable x.
Further, when the scene length is b or more, whether or not to extract a plurality of characteristic portions is designated by the input unit 1020 (step 5020). The first row of the scene feature table 8010 is read (step 5030), and the value of the scene counter is set to 0 (step 5040).

After that, cases are classified according to the length of the scene. First, to see whether or not the target scene is a short scene of a certain length or less, it is checked whether or not the scene length 8020 is a or less (step 5050). If the scene length 8020 is not a or less, then the target In order to see whether or not the scene is a scene longer than a certain length, it is checked whether the scene length 8020 is b or more (step 5080). If the scene length 8020 is a or less, it is further checked whether the average volume 8030 is d or more (step 5060). Average volume 803
If 0 is greater than or equal to d, it is determined that the scene length 8020 is a characteristic portion even if it is short, and all the frames of the scene are extracted in the extraction table.
The maximum volume difference 8050 is 0, the maximum volume difference 9030 is the maximum volume difference 8070, and the start frame number 904 is
0 to "(scene end frame number)-(scene length) +
1 ", and the scene end frame number 8015 is registered in the end frame number 9050 (step 5070).
After that, 1 is added to the scene counter (step 517).
0). When the average volume 8030 is smaller than d, it is determined that the part is not characteristic and nothing is done.

If the scene length 8020 is not b or more, it is determined that the scene is an intermediate scene length, and it is checked whether the average volume 8030 is e or more (step 5130). When the average volume 8030 is equal to or higher than e, it is determined that the part having the maximum volume is the most characteristic part in the scene, and the frame having the maximum volume frame number and the x frames before and after the frame are extracted to the extraction table. for,
The maximum volume 9020 is the maximum volume 8050, the maximum volume difference 9030 is the maximum volume difference 8070, and the start frame number 9040 is "(maximum volume frame number) -x".
And "(maximum volume frame number) + x" are registered in the end frame number 9050 (step 514
0), 1 is added to the scene counter (step 517)
0). If the average volume 8030 is smaller than e, then it is checked whether or not the maximum volume difference 8070 is f or more (step 5150). Maximum volume difference 8070 is f
In the above case, it is determined that the part having the maximum volume difference is the most characteristic part in the scene, and the frame having the maximum volume difference frame number and the x frames before and after the frame are extracted in the extraction table. , The maximum volume 9020 is the maximum volume 8050, the maximum volume difference 9030 is the maximum volume difference 8070, the start frame number 9040 is "(maximum volume difference frame number) -x", and the end frame number 9050 is "( Maximum volume difference frame number) + x ",
Each is registered (step 5160), and 1 is added to the scene counter (step 5170). The maximum volume difference is f
If it is smaller than the above, it judges that the scene is not characteristic and does nothing.

On the other hand, when the scene length is b or more,
It is determined that the scene is longer than a certain length, and it is first checked whether or not a plurality of characteristic portions are extracted from the scene (step 5090). When extracting a plurality of characteristic parts, the scene sub-counter that counts the number of those characteristic parts is set to 0 (step 5100), and the feature of the scene is extracted by the plural characteristic part extraction process in the same scene (step 5110). Extract the part and add it to the scene counter
(Scene sub-counter) -1 "is added (step 51
20) After that, 1 is added to the scene counter (step 51).
70). That is, step 5120 and step 5170
Considering together, it is equivalent to adding the value of the scene sub-counter to the scene counter. If a plurality of characteristic parts are not extracted from the same scene, it is determined that the part with the highest volume is the most characteristic part in the scene, and the frame with the highest volume frame number and x frames before and after it are extracted from the extraction table. To the maximum volume 9020, the maximum volume difference 9030, the maximum volume difference 8070, and the start frame number 9040.
(Maximum volume frame number) -x "," (maximum volume frame number) + x "are registered as the end frame number 9050 (step 5140), and 1 is added to the scene counter (step 5170).

After any of these steps, the scene feature table 8010 is checked to see if there is a next row (step 5180), and if so, that row is read again and the scene length is less than or equal to a. It is checked whether there is any (step 5050), and the process is continued. If it does not exist, the characteristic portion extraction processing ends. The operation example of the characteristic portion extraction process (step 2040) in FIG. 5 has been described above.

Next, an operation example of the same-scene multiple feature portion extraction process (step 5110) shown in FIG. 5 will be described with reference to FIG. First, prepare the same number of flags as the number of frames of a scene from which a plurality of characteristic parts are extracted, and assign a number from "(scene end frame number)-(scene length) +1" to the scene end frame number to identify each frame. One-to-one correspondence (step 6010). Read the first frame of the scene to be extracted (step 6020),
It is checked whether the volume of the current frame is greater than or equal to the volume c (step 6030). If the volume is equal to or higher than the volume c, flags corresponding to the current frame and x frames before and after the current frame are set.
After that, it is checked whether or not the next frame exists, and if there is, the frame is read (step 606).
0), it is checked again whether the frame volume is not less than the volume c (step 6030). If the next frame does not exist, it is checked how many temporally consecutive flagged portions are present and the value is set in the scene sub-counter (step 6070), and finally the frame corresponding to the flag is set. In order to extract the maximum volume 9020 to the maximum volume 9020, the maximum volume difference 90
The maximum volume difference 8070 is registered as the value 30, the start flag number of the continuous portion is set as the start frame number 9040, and the end number of the continuous portion is set as the end frame number 9050 for all the continuous portions (step 6080). The inner plural feature portion extraction process is ended. The operation example of the multiple feature portion extraction process (step 5110) in the same scene in FIG. 6 has been described above.

Next, an operation example of the search data selection process (step 2050) shown in FIG. 2 will be described with reference to FIG. First, the maximum allowable number of scenes that specifies the maximum number of scenes of the search data and the maximum allowable number of frames that specifies the maximum number of frames are specified. Further, in order to prevent the number of frames forming one characteristic portion from becoming too small and making it impossible to understand the contents, the minimum necessary limit frame number that specifies the lower limit of the number of frames forming one scene is specified. Further, which one of the characteristic part having the largest maximum volume and the characteristic part having the largest maximum volume difference is designated as the search data is designated (step 7010). Next, it is checked whether the value of the scene counter is larger than the maximum allowable number of scenes (step 7020). When the value of the scene counter is greater than the maximum allowable number of scenes, it is necessary to reduce the number of extracted scenes. Therefore, the characteristic scene with a large maximum volume is prioritized as the data for retrieval over the characteristic scene with a large maximum volume difference. Check whether or not (step 703)
0), if priority is given, the scene with the smallest maximum volume difference is deleted from the extraction table, and if it is not given priority, the scene with the smallest maximum volume difference is deleted from the extraction table (step 7040).
After (step 7050), 1 is subtracted from the scene counter (step 7055), the process returns to step 7020, and this is repeated until the value of the scene counter becomes equal to or less than the maximum allowable limit number of scenes. At this time, if there are a plurality of scenes having the smallest maximum volume or the maximum volume difference, the scene having the smallest end frame number is deleted.

When the number of extracted scenes is equal to or less than the maximum allowable number of scenes, first, the number of extracted frames which is the sum of "(end frame number)-(start frame number) +1" in each row of the extraction table is calculated. (Step 7058), 0 is set to the deletion prohibited scene counter (step 706).
0). The deletion-prohibited scene counter is a counter that counts the number of scenes in which the number of frames making up one scene is smaller than the minimum required limit number of frames. Next, it is checked whether the value of the deletion prohibited scene counter is equal to the value of the scene counter (step 7070). If they are equal, it means that there is no scene that can be deleted, and the search data selection process is terminated. If they are not equal, it is checked whether the number of extracted frames is larger than the maximum allowable limit number of frames (step 7080).
If the number of extracted frames is equal to or less than the maximum allowable limit number of frames, there is no need to delete the frames, and the search data selection process ends.

When the number of extracted frames is larger than the maximum allowable limit number of frames, it is necessary to reduce the number of extracted frames. Therefore, first, the scene length of the first scene is set to "
(End frame number)-(start frame number) +1 "(step 7090) to check whether the scene length is greater than the minimum required maximum number of frames (step 7100). If it is larger than that, it is possible to reduce frames from the scene. Therefore, in order to delete the front and rear frames of the scene one by one, 1 is added to the start frame number of the scene and 1 is subtracted from the end frame number (step 7110). ), And subtract 2 from the number of extracted frames (step 7115) If the scene length is not larger than the minimum required limit frame, further reduction of the number of frames will make the contents of the scene incomprehensible, so The frame deletion is regarded as impossible, and 1 is added to the deletion prohibition counter (step 7120). , The extraction table determine whether the next scene exists (step 7130). If present, it calculates the scene length of the next scene (step 71
40), the process is repeated from step 7100 until there is no next scene. When the next scene does not exist, the process returns to step 7070, and this processing is repeated until the deletion prohibited scene counter is equal to the scene counter and the number of extracted frames becomes equal to or less than the maximum allowable limit frame number. The operation example of the search data selection process (step 2050) in FIG. 7 has been described above.

In the above embodiment, the search data generation process is performed every time the search is performed. However, if the search data selection management unit 1043 shown in FIG. No need to do. At this time, the main control unit 1010 checks the search data selection management unit 1043 for the moving image file for which the browse instruction is received, and if the search data has already been generated, it is directly searched based on the search data. indicate. The search data generation unit 1040 creates search data only when the search data has not been generated.

Further, in the above-described embodiment, the user sets all the various parameters for generating the search data by using the input unit 1020 of FIG. 1, but is held in the search data selection management unit 1043. It is also possible to use the information of the scene feature table as a parameter.
For example, a value of 1.5 times the average volume of the scene can be used for c in FIG. This is step 5 for c
No setting in 010, multiple feature part extraction processing in the same scene 5
At the start of 110, c = (average volume) × 1.5 may be set. Also, in step 5010, it is possible to use a value of 10% of the scene length of the entire moving image file as the scene length b, and a value of twice the average volume of the entire moving image file as the volume e. In these cases, b =
If (total sum of scene lengths in scene feature table) × 0.1 and e = [(total sum of (scene length × average volume)} ÷ (total sum of scene lengths)] × 2 are set in step 5010 of FIG. good.

When currently handling AV data, tapes such as VTRs are the mainstream as a medium for storing video.
A tape as a storage medium has sequential access and can be said to be low-speed mass storage. In a system using such a medium, in order to enhance the effect of the search, the search data selection management unit 1043 may store the search data in another storage medium (such as a hard disk) that is capable of high-speed random access. Conceivable. According to this method, the moving image file can be searched at high speed, and the user can reproduce only the necessary scene from the tape.

In a digital video editing device or the like, an analog tape is A / D converted and stored in a digital medium (hard disk or the like) in the device (download).
There is work, and the search data generation process in this embodiment may be performed during this work. By doing so, high-speed search for moving image files will be possible immediately after downloading.

As described above, according to this embodiment,
It is possible to extract an image (moving image) for search based on the frame in which the scene switching has occurred and the audio information of the frame, and to search for arbitrary AV data (file).

As a result, unlike the prior art, it is not necessary for the user to specify the characteristic portion to be searched in advance and to create the search data before the search is performed, and the work amount of the user can be reduced. Is obtained.

[0066]

As described above, according to the present invention, for important (characteristic) scenes, a plurality of frames can be extracted as detection data for each scene, and the detection data can be extracted finely. .

Further, according to the present invention, for a scene that is not important, the number of search data can be reduced without extracting the search data from the scene.

[Brief description of drawings]

FIG. 1 is a block diagram showing an overall configuration of a moving image search data generation method according to an embodiment of the present invention.

FIG. 2 is a flowchart showing the operation of the embodiment shown in FIG.

FIG. 3 is a flowchart showing an operation of scene feature table creation processing in the flowchart of FIG.

FIG. 4 is a flowchart showing an operation of a scene switching detection process in the flowchart of FIG.

5 is a flowchart showing an operation of a characteristic scene extraction process in the flowchart of FIG.

6 is a flowchart showing an operation of a process for extracting a plurality of characteristic portions within the same scene in the flowchart of FIG. 5;

FIG. 7 is a flowchart showing an operation of search data selection processing in the flowchart of FIG.

FIG. 8 is a scene feature table created in the process of generating search data.

FIG. 9 is an extraction table created in the process of generating search data.

FIG. 10 is an explanatory diagram regarding an encoding method of MPG1.

FIG. 11 is a device configuration diagram in which the present invention can be realized.

FIG. 12 is an explanatory diagram regarding a block matching method.

FIG. 13 is an explanatory diagram related to a method of detecting the volume of each frame.

FIG. 14 is an explanatory diagram related to a method of detecting the volume of each frame.

[Explanation of symbols]

 1010 ... Main control unit 1020 ... Input unit 1030 ... Data storage management unit 1031 ... Data storage unit 1032 ... Data reading unit 1040 ... Search data generation unit 1041 ... Scene switching detection unit 1042 ... Volume detection unit 1043 ... Search data selection management Part 1050 ... Display part 1060 ... Communication part

Front Page Continuation (72) Inventor Akira Tanaka 1099 Ozenji, Aso-ku, Kawasaki-shi, Kanagawa Hitachi, Ltd. System Development Laboratory

Claims (12)

[Claims] 1. A moving image data representing a moving image composed of a plurality of sequential image frames, and each of the frames.
Sequential audio data associated with each of the frames is input, and at least one of the frames is included in the input moving image data for each scene included in the moving image.
In a moving image search system for extracting moving image data corresponding to a frame as search data, a scene switching for detecting a frame forming the scene for each scene based on the content of the inputted moving image data. Detection means, volume detection means for detecting the volume of the input audio data for each corresponding frame, and for each scene detected by the scene switching detection means, a value predetermined by the volume detection means A moving image search system, comprising: search data extracting means for extracting, as search data for each scene, moving image data corresponding to at least one frame in which the above sound volume is detected. . 2. A moving image data representing a moving image composed of a plurality of sequential image frames, and each of the frames.
Sequential audio data associated with each of the frames is input, and at least one of the frames is included in the input moving image data for each scene included in the moving image.
In a moving image search system for extracting moving image data corresponding to a frame as search data, a scene switching for detecting a frame forming the scene for each scene based on the content of the inputted moving image data. Detection means, volume detection means for detecting the volume of the input audio data for each corresponding frame, and for each scene detected by the scene switching detection means, for each frame detected by the volume detection means. On the basis of the sound volume of, the moving picture data corresponding to at least one frame whose sound volume change with respect to a frame spaced by a predetermined distance is equal to or more than a predetermined value is extracted as search data for each scene. A moving image search system comprising: a search data extracting unit. 3. The moving image search system according to claim 1 or 2, wherein the search data extracting means is based on the volume detected by the volume detecting means among the scenes detected by the scene switching detecting means. A moving image search system comprising: a calculating means for calculating an average volume in the scene, and extracting search data only for a scene in which the average volume calculated by the calculating means is equal to or higher than a predetermined volume. 4. The moving image search system according to claim 1, wherein the search data extraction means includes a detection means for detecting a frame having the maximum volume for each scene detected by the scene switching detection means. A moving image search system, wherein moving image data corresponding to a frame detected by the detection means is extracted as search data for each scene. 5. The moving image search system according to claim 2, wherein the search data extraction unit has a maximum change in volume with respect to each frame detected by the scene switching detection unit with respect to frames spaced apart by a predetermined interval. The video search system is characterized by extracting video data corresponding to a frame as search data for each scene. 6. A moving image data representing a moving image composed of a plurality of sequential image frames, and each frame.
Sequential audio data associated with each of the frames is input, and at least one of the frames is included in the input moving image data for each scene included in the moving image.
In a moving image search system for extracting moving image data corresponding to a frame as search data, a scene switching for detecting a frame forming the scene for each scene based on the content of the inputted moving image data. Detection means, volume detection means for detecting the volume of the input audio data for each corresponding frame, and for each scene detected by the scene switching detection means, a value predetermined by the volume detection means First search data extracting means for extracting moving image data corresponding to at least one frame in which the above sound volume is detected as search data for each scene, and detected by the scene switching detection means. With respect to each of the scenes, a change in volume for frames spaced by a predetermined interval is predetermined based on the volume of each frame detected by the volume detecting means. Second search data extracting means for extracting, as search data for each scene, moving image data corresponding to at least one frame having a value greater than or equal to the calculated value; and the first search data extracting means. Alternatively, the moving image search system is provided with a means for switching one of the second search data extracting means and enabling it by an instruction from the outside. 7. The moving image search system according to claim 1, wherein the search data extraction unit externally indicates a maximum number of frames to be extracted from each scene detected by the scene switching detection unit. The moving image search system, further comprising: means for limiting the number of frames to be extracted for each of the scenes, the number being equal to or less than the number instructed by the instruction means. 8. The moving image search system according to any one of claims 1 to 7, wherein the search data extraction means includes an instruction means for externally instructing a maximum value of the number of scenes for extracting the search data. A moving image search system comprising means for limiting the number of scenes from which search data is extracted to be equal to or less than the number designated by the means. 9. A moving image data representing a moving image composed of a plurality of sequential image frames, and each frame.
Sequential audio data associated with each of the frames is input, and at least one of the frames is included in the input moving image data for each scene included in the moving image.
In a moving image search system for extracting moving image data corresponding to a frame as search data, a scene switching for detecting a frame forming the scene for each scene based on the content of the inputted moving image data. Detection means, volume detection means for detecting the volume of the input audio data for each corresponding frame, and for each scene detected by the scene switching detection means, a value predetermined by the volume detection means A moving picture data corresponding to the frame in which the sound volume is detected and the frames in the vicinity thereof, as search data extracting means for extracting the moving picture data, and a moving picture characterized by comprising: Search system. 10. Moving image data representing a moving image composed of a plurality of sequential image frames and sequential audio data associated with each of the frames are inputted, and the inputted moving image data is inputted. In a moving image search system for extracting, as search data, moving image data corresponding to at least one of the frames for each scene included in the moving image, based on the contents of the input moving image data. Then, for each of the scenes, scene switching detection means for detecting a frame forming the scene, volume detection means for detecting the volume of the audio data to be input for each corresponding frame, and the scene For each scene detected by the switching detection means, a frame whose volume change with respect to a frame separated by a predetermined interval is equal to or more than a predetermined value, and its vicinity Video retrieval system, characterized in that it comprises a motor, a search data extracting means for extracting said as search data for each scene, the - videos de that corresponds to the frame. 11. A moving image data input by inputting moving image data representing a moving image composed of a plurality of sequential image frames and sequential audio data associated with each of the frames. A video search data extraction method for extracting video data corresponding to at least one frame as search data for each scene included in the video from the video, and the video data to be input. Based on the contents of the above, for each of the scenes, the frame that constitutes the scene is detected, the volume of the input audio data is detected for each corresponding frame, and for each detected scene, A moving image characterized by extracting moving image data corresponding to at least one frame in which a sound volume equal to or higher than a predetermined value is detected as search data for each scene. Search data extraction method. 12. Moving image data representing a moving image composed of a plurality of sequential image frames and sequential audio data associated with each of the frames are inputted, and the inputted moving image data is inputted. A video search data extraction method for extracting video data corresponding to at least one frame as search data for each scene included in the video from the video, and the video data to be input. Based on the contents of the above, for each of the scenes, the frame that constitutes the scene is detected, the volume of the input audio data is detected for each corresponding frame, and for each detected scene, A moving image corresponding to at least one frame in which the change in volume for a frame separated by a predetermined interval is equal to or more than a predetermined value based on the detected sound volume of each frame - Video Search data extraction method and extracting the data, as search data for the respective scenes.