JP3573493B2 - Video search system and video search data extraction method - Google Patents

Description

[0001]
[Industrial applications]
The present invention relates to a moving image search system and a moving image search method in which a user searches for a video of a predetermined scene from a medium storing the video.
[0002]
[Prior art]
In recent years, with the spread of multimedia systems, systems for handling AV data have begun to be developed one after another. Here, the AV data refers to data having both moving image data representing a moving image and audio data. As a format of AV data, a format in which moving image data and audio data are managed in parallel on a time basis has become mainstream. For example, in QuickTime of Apple Computer, moving image data and audio data are put on separate tracks, and a moving image accompanied by audio is reproduced while accessing these tracks.
[0003]
First, encoding of moving image data will be described. Moving image data has an extremely large amount of information per unit time as compared with audio data. However, in general, in a continuous moving image, the preceding and succeeding images are very similar to the image of interest, and therefore, when viewed as data, there is much redundancy. Therefore, high-efficiency coding can be performed digitally to greatly reduce the required information amount. At present, as a method of high-efficiency coding, an international standard draft MPEG1 defined by MPEG (Moving Picture Coding Experts Group) of ISO (International Organization for Standardization) / IEC (International Electrotechnical Commission) is mainstream. Since this patent is for searching for AV data, it is possible to handle a huge amount of AV data. Inevitably, there is a possibility that the AV data is subjected to high-efficiency encoding. Therefore, the encoding method of MPEG1 will be described first with reference to FIG.
[0004]
As described above, in a generally continuous moving image, the preceding and following images are very similar to the image of interest. Therefore, in MPEG1, for example, when an image 10040 is to be encoded, a difference from the temporally preceding image 10010 is obtained, and the difference value is encoded to obtain a P picture (forward predicted encoded image) 10080. When the image 10020 is to be encoded, the difference between the temporally forward image 10010, the backward image 10040, or the interpolated image created from the forward and backward is obtained, and the difference value is encoded to obtain the B value. A picture (bidirectionally predicted coded image) 10060 is assumed. Similarly, when trying to encode the image 10030, a B picture 10070 is created from the images 10010 and 10040. By encoding in this way, the redundancy in the time axis direction is reduced, and the amount of information is reduced. Also, instead of always encoding the difference, for example, the input image 10010 may be encoded as it is to obtain an I picture (intra-frame encoded image) 10050.
[0005]
In MPEG1, motion compensation is used in addition to simply taking the same position difference. This is done by searching for the block with the smallest difference from the block to be coded in the previous image of the image to be coded in macroblock units, and taking the difference from it to send further. This is a method to reduce unnecessary data. Here, the macro block is a block of 16 × 16 pixels. Actually, for a P picture, a difference between the predicted picture after motion compensation and a difference between the two pictures, which does not take a difference, is selected and coded in macroblock units. But still, a lot of data has to be sent for the part that comes out of the movement of the object. Therefore, in the B picture, one of four types, one obtained by taking a difference with an interpolated image created not only from the front but also from the back or both of them after the motion compensation already decoded, and the one not taking the difference. Those with a small number data amount are similarly selected and encoded in macroblock units. In this way, most data need not be sent.
[0006]
MPEG1 also stores moving image data and audio data in separate tracks, accesses those tracks simultaneously, and reproduces moving images with sound. Therefore, for example, it is also possible to extract moving image data corresponding to audio data at a certain time.
[0007]
By the way, in a system using such AV data, particularly in an authoring system for creating an application for presentation using various data such as text, graphics, sound, etc., retrieval and display of AV data are important problems. It becomes. This is because AV data takes a longer time to read data for reproduction than other data, and the reproduction time of the AV data itself may be longer.
[0008]
2. Description of the Related Art Conventionally, in a system for retrieving AV data, as one of mechanisms for extracting desired data, browsing (browsing) data registered in advance is individually or plurally displayed in accordance with a user's instruction. There was a browsing search to do. This search method is an excellent search method for AV data because the contents of a moving image can be grasped in a short time by looking at browsing data.
[0009]
By the way, as a problem of the browsing search method, since it is necessary to set search data in advance, the setting work is very time-consuming. To solve this problem, Japanese Patent Laid-Open Publication No. 3-292572 discloses a method of detecting a portion where a scene change has occurred, extracting one of the frames before and after the portion as a search image in a search list, and displaying the list at the time of search. There is a technique described in. Here, a frame is a still image of a minimum unit element that constitutes moving image data. A scene is defined as an aggregate of frames in which a subject is continuously photographed. The portion where the scene switching occurs indicates a portion where two different scenes are connected. For example, in a general television moving image, one second is composed of 30 frames of still images. In this method, it is also possible to extract a scene change portion excluding a scene change portion in which the time until the next detected scene change portion is equal to or less than a predetermined time. If this method is used, one frame of a search image can be extracted per scene and used as search data.
[0010]
[Problems to be solved by the invention]
However, the technique described in the above publication has a problem in that when the AV data has a large number of scenes, the number of images to be searched also increases accordingly, so that it takes a long time to search. In other words, simply using a frame in which scene switching has occurred as a search image is not sufficient as a moving image search system and method.
[0011]
SUMMARY OF THE INVENTION It is an object of the present invention to extract a plurality of frames of search images for important scenes and extract fine search data.
[0012]
It is another object of the present invention to reduce the number of search data for an unimportant scene without extracting a search image from the scene.
[0013]
[Means for Solving the Problems]
In order to solve the above problems, according to the present invention,
Moving image data representing a moving image composed of a plurality of sequential image frames, and sequential audio data associated with each of the frames are input, and are included in the moving image from the input moving image data. For each sceneInspectionSearch dataToIn the video search system to extract,
Based on the content of the moving image data to be input, for each of the scenes, a scene switching detection unit for detecting a frame constituting the scene,
Volume detection means for detecting the volume of the input audio data for each corresponding frame,
For each scene detected by the scene switching detection means, a volume equal to or greater than a predetermined value is detected by the volume detection means.WasflameincludingVideo data as the search data for each sceneFrom the sceneSearch data extraction means to be extracted;
Equipped,
The search data extraction means,
A moving image search system is provided, wherein the number of scenes from which the search data is extracted is limited to not more than the number specified as the maximum number of scenes from which the search data is extracted..
[0014]
Also,
Moving image data representing a moving image composed of a plurality of sequential image frames, and sequential audio data associated with each of the frames are input, and are included in the moving image from the input moving image data. For each sceneToSearch dataToIn the video search system to extract,
Based on the content of the moving image data to be input, for each of the scenes, a scene switching detection unit for detecting a frame constituting the scene,
Volume detection means for detecting the volume of the input audio data for each corresponding frame,
For each scene detected by the scene switching detection unit, a change in volume for frames separated by a predetermined interval based on the volume of each frame detected by the volume detection unit is equal to or greater than a predetermined value.HLaemincludingVideo data as the search data for each sceneFrom the sceneSearch data extraction means to be extracted;
Equipped,
The search data extraction means,
A moving image search system is provided, wherein the number of scenes from which the search data is extracted is limited to not more than the number specified as the maximum number of scenes from which the search data is extracted..
[0015]
[Action]
First, the AV data will be described. The AV data is data having both moving image data representing a moving image and audio data representing a corresponding audio.
[0016]
With respect to the AV data, a portion where a scene change occurs and a volume of each frame are detected. Based on the detected information, a necessary frame is extracted from the AV data, and search data is created and displayed based on the extracted frame.
[0017]
That is, attention is paid not only to scene switching but also to audio data as a criterion for generating moving image search data. This is because a part with a large volume or a part with a large volume change is considered to be a more characteristic part for the AV data. The moving image search data is generated by selecting a portion necessary for a search from the frames and combining them.
[0018]
The AV data is usually composed of a plurality of scenes, and the importance of the contents differs depending on the reproduction time of each scene. Therefore, moving image search data is extracted as follows according to the reproduction time of each scene.
[0019]
From a scene with a short playback time, the search data is not extracted because it is basically determined to have no meaning. However, if the average volume of the scene is equal to or higher than a certain value, the search data is extracted for all the scenes as having some meaning.
[0020]
A scene with a long reproduction time is determined to have some meaning, and one characteristic portion is extracted. However, if it is desired to avoid a situation in which only one characteristic portion is extracted even when the scene is very long, a plurality of characteristic portions can be extracted from the same scene in accordance with a searcher's instruction. Further, in this case, one of the following two functions can be selected.
[0021]
(1) When it is desired to extract a plurality of characteristic portions, all frames having a certain volume or higher are extracted.
[0022]
(2) When it is desired to extract only one characteristic portion, a frame indicating the maximum volume in the scene is extracted.
[0023]
If the average volume is equal to or higher than a certain value, a scene having an intermediate reproduction time is extracted as a part having the maximum volume in the scene as having some meaning. Even if the average volume is less than a certain value, if there is a volume change portion that is equal to or more than a certain volume difference in the scene, it is determined to have some meaning, and the portion showing the maximum volume change in the scene is determined. Extract. Other scenes are not extracted because they are determined to be meaningless.
[0024]
The AV data is composed of a plurality of frames, and has a volume characteristic such as maximum, minimum, and average of a certain size for each frame. When examining volume changes in a scene, the volume of one frame must be compared to the volume of a different frame. Therefore, when examining a volume change in a scene, the following is performed.
[0025]
First, when examining a change in volume, if the immediately preceding frame is used as a difference detection target, there may be a case where it is not possible to accurately determine a portion having the largest volume difference due to the influence of noise. Thus, the searcher can set how many frames before the volume to be compared when the difference is detected.
[0026]
Further, the detection of the maximum part of the volume change is performed in two ways: one is to detect the part where the volume increase is the largest, and the other is to detect the part where the volume change is the largest, that is, the part where the absolute value of the volume difference is the largest. is there. Which of these two types to select is set by the searcher.
[0027]
The search data is configured by combining frames extracted from a plurality of characteristic scenes. However, if the search data is too large when the search data is generated, the number of images that the searcher must refer to increases, and the time required for the search also increases. Therefore, in order to prevent search data from being extracted more than necessary, the following is performed.
[0028]
First, the number of frames constituting the scene extracted from each scene from which the search data is extracted is set so as to determine how many frames before and after the characteristic portion are to be extracted at the time of extraction so that the searcher can determine the number of frames.
[0029]
Further, if the search data is too large, the number of images that the searcher must refer to increases, so that the time required for the search also increases. Therefore, in order to reduce search data that is too large, the following is performed.
[0030]
First, the searcher can set the maximum number of scenes at which extraction of search data is allowable. When the number of scenes for extracting the search data is larger than the set number of scenes, the number of scenes for extracting the search data is reduced to be equal to or less than the set number of scenes. In this case, the searcher can set which of the scene with the smallest maximum volume and the scene with the smallest volume difference is deleted first.
[0031]
In addition, the maximum total number of frames allowable as search data is set by the searcher. When the total number of frames of the search data is larger than the set number of frames, the number of frames of the search data is reduced to be equal to or less than the set number of frames.
[0032]
In the function of setting the number of frames of the search data to be equal to or less than the set number of frames, if the size of the search data is too small, the contents may not be understood during the search. Therefore, in order to avoid this, the searcher can set the minimum number of frames necessary for one scene so that the number of frames per scene is not made too small and the contents cannot be understood. To
[0033]
【Example】
FIG. 1 is a block diagram of a moving image search data generation method according to an embodiment of the present invention. In FIG. 1, reference numeral 1010 denotes a main control unit that controls the entire video search data generation method according to the present embodiment. Reference numeral 1020 denotes a user's instruction to search for a video file from a keyboard or the like, and parameters required when extracting the search data. This is an input unit for inputting. 1030, a data storage management unit that records and reads a moving image file; 1040, a search data generation unit that extracts a characteristic part in the moving image file; 1050, a display that displays search data managed by the search data generation unit 1040 A communication unit 1060 performs data exchange between the above units.
[0034]
In the above configuration, for example, the main control unit 1010 is a CPU, the input unit 1020 is a keyboard, the data storage management unit 1030 is a hard disk, the display unit 1050 is a CRT display, and the search data generation unit 1040 is a ROM, RAM, or the like. By realizing the communication unit 1060 using a bus with a program stored and executed by the CPU, the communication unit 1060 can be configured with a conventionally well-known device.
[0035]
FIG. 11 shows a device configuration diagram capable of realizing FIG. In FIG. 11, reference numeral 11010 denotes a CPU (central processing unit), 11020 denotes a keyboard, 11030 denotes a hard disk, 11040 denotes a ROM (read only memory), 11050 denotes a display, and 11060 denotes a bus.
[0036]
Returning to FIG. 1, reference numerals 1031 and 1032 constitute a data storage management unit 1030. Reference numeral 1031 denotes a data storage unit that stores AV data, and 1032 denotes a data read unit that reads data from the data storage unit 1031 under the control of the main control unit 1010 or the search data generation unit 1040. 1041, 1042, and 1043 constitute a search data generation unit 1040. Reference numeral 1041 denotes a scene switching detection unit that detects scene switching of a moving image file read from the data reading unit 1032. Reference numeral 1042 denotes a volume detection unit that detects the volume of each frame of the moving image file read from the data reading unit 1032. A search data selection management unit 1043 performs a predetermined process based on the information of the scene switching detection unit 1041 and the volume detection unit 1042, and manages the result. The search data selection management unit 1043 holds the frame number or the frame data. .
[0037]
First, the scene switching detection process will be described. The process of detecting a scene change can be performed using, for example, a correlation between two temporally consecutive frames and information of a motion vector indicating a parallel movement of the entire image. Specifically, first, a two-dimensional small block is determined in one frame, and a correlation between the blocks (for example, the square of an error) is calculated between two temporally consecutive frames to determine whether there is a correlation. (The magnitude of the square of the error). If there is a correlation (the square of the error is small), it is considered to be a continuous scene (scene), and no scene switching has occurred. However, this frame correlation alone determines that a scene caused by a parallel movement of the camera such as pan (lateral movement) or tilt (vertical movement) is a scene switch. A vector is detected, and it is determined whether or not scene switching is performed based on the motion vector (when a motion vector is detected, a continuous scene). The detection of the motion vector is obtained, for example, from the relationship between the spatial gradient of the frame image and the difference between the images. The method of detecting a motion vector is described in detail in the document "Digital signal processing of images: written by Takahiko Fuukiki, Nikkan Kogyo Shimbun".
[0038]
Here, a method of detecting a motion vector will be described. As a typical method of detecting a motion vector, there is a block matching method. Hereinafter, the block matching method will be described with reference to FIG. The block matching method compares one macroblock (a block of 16 × 16 pixels) of an encoding target image with all macroblocks of a previous image, and adopts a vector to a macroblock having the smallest image difference as a motion vector. How to For example, when it is desired to obtain a motion vector of the macro block 12010, the positions of the macro blocks 12020 to 1270 read from the previous image are slightly shifted (in this figure, shifted by 2 in the x-axis direction). Each of the read blocks is compared with a macro block 12010, and a block (block 12040 in this figure) having the smallest image difference is obtained. Block matching is performed for the number of blocks that can be candidates. In this figure, the vector (6,0) is adopted as a motion vector.
[0039]
Therefore, for example, when a scene change is detected in MPEG1, the number of blocks having a difference from the motion-compensated predicted image and the number of blocks having no difference are checked in the P picture, and the number of blocks having the difference is determined. Is greater than or equal to a certain value, it may be considered that a motion vector has been detected and may be determined to be a continuous scene.
[0040]
Therefore, it is determined that scene switching has occurred when there is no inter-frame correlation and no motion vector is recognized.
[0041]
Here, the volume detection processing will be described. In the sound volume detection processing, if the sound volume value of each frame is used as it is, the influence of noise of each frame is greatly affected. Therefore, in order to remove noise for each frame, the sound volume value for each frame is, for example, three frames before and after which are temporally continuous with the frame for which the sound volume is to be obtained, that is, a total of seven frames including the frame for which the sound volume is to be obtained. Take the average value and define it as the volume value for the frame. Of course, it is also possible to use a noise removal method other than this method.
[0042]
Hereinafter, a method of detecting the volume of each frame will be described. Normally, in order to digitize an acoustic waveform, sampling, quantization, and encoding need to be performed in order as shown in FIG. In sampling, the time is divided finely, and the height of the waveform in that unit time is observed. In the quantization, the height of the waveform is read while being divided into a binary number of a certain number of digits. In the encoding, the acoustic waveform is converted into a digital signal having a value obtained by quantization. In a general music compact disc, the sampling frequency (the number of time steps per second when sampling) is 44.1 kHz, and the number of quantization bits (the fineness that separates the strength of sound during quantization) is 16 Is a bit.
[0043]
Therefore, the volume of each frame can be obtained by defining, for example, the maximum value 14010 as the volume of the frame in the group of encoded numerical values within one frame time as shown in FIG. Of course, within the time corresponding to one frame, the average value or the minimum value 14020 of the encoded values may be defined as the volume of the frame.
[0044]
In the present embodiment, the above-described inter-frame correlation, motion vector, and frame volume definition are used as a scene switching detection method and a volume detection method. FIG. 8 is a scene feature table for storing information for each scene required in the process of generating search data. Each row of the table stores the scene end frame number, scene length, average volume, maximum volume frame address, maximum volume, maximum volume difference frame address, and maximum volume difference of each scene. By realizing a table by a list, the size of the table can be changed arbitrarily.
[0045]
Hereinafter, an operation example of the block diagram of FIG. 1 will be described with reference to the flowchart of FIG. FIG. 2 is a flowchart showing the operation of the moving image search data generation method. First, when a search process of a moving image file is started by an instruction of the input unit 1020, a searcher selects a moving image file to be searched by using the input unit 1020 (step 2010). The main control unit 1010 instructs the search data generation unit 1040 to create search data for the moving image file in order to search for the specified moving image file. In the search data generation unit 1040, the search data selection management unit 1043 instructs the data reading unit 1032 to read the moving image file (step 2020). The search data selection management unit 1043 requests the scene switching detection unit 1041 and the volume detection unit 1042 for scene switching occurrence information and volume information of each frame in order to clarify the characteristic portion of the moving image file. (Step 2030). The search data selection management unit 1043 extracts the characteristic part in the moving image file into the extraction table of FIG. 9 based on the information obtained from the scene characteristic table (step 2040). Each row of the extraction table stores the maximum volume, the maximum volume difference, the start frame number, and the end frame number of each scene constituting the search data. By realizing a table by a list, the size of the table can be changed arbitrarily. The search data selection management unit 1043 selects search data from the extraction table based on the condition given by the input unit 1020 (Step 2050). Finally, the main control unit 1010 instructs the display unit 1050 to display a list of search data, and the display unit 1050 displays a list of frames from the start frame number to the end frame number of each scene stored in the extraction table ( Step 2060). The data exchange between these units is performed via the communication unit 1060.
[0046]
FIG. 3, FIG. 5, and FIG. 7 are flowcharts showing examples of the processing contents of the scene characteristic table creation processing, the characteristic part extraction processing, and the search data selection processing in the flowchart of FIG. Further, FIG. 4 shows a scene switching detection process in the flowchart of FIG. 3, and FIG. 6 shows a flowchart of an example of each process content extraction process in the flowchart of FIG. Hereinafter, the processing examples of the flowcharts in FIGS. 3 to 7 will be sequentially described.
[0047]
An operation example of the scene feature table creation processing (step 2030) shown in FIG. 3 will be described. First, when detecting the volume difference between two frames, the number of frames α for volume difference detection that specifies the number of frames before the volume difference is set so that the volume difference with any relative frame can be detected. , Input unit 1020. Further, when detecting a frame in which the volume difference from the frame before the α frame is the largest, the frame in which the increase in the volume is the largest is taken, or the change in the volume, that is, the frame in which the absolute value of the volume difference is the largest, is taken. Is specified by the input unit 1020. (Step 3010). The current frame number for counting the number of frames from the first frame of the moving image file is set to 0 (step 3015). The first frame of the specified file is read (step 3020), and 1 is set to a frame counter for counting the number of frames of one scene, and 0 is set to a variable total volume representing the total volume of each frame in one scene as an initial setting value. The current frame number is set as the variable maximum volume frame number indicating where the frame with the highest volume is located in one scene, 0 is set as the variable maximum volume indicating the volume of the frame, and 0 is set as the α frame before in the one scene. The current frame number is set to a variable maximum volume difference frame number indicating where the frame with the maximum volume difference is located, and 0 is set to a variable maximum volume difference indicating the difference between the volume of that frame and the volume of the frame preceding the α frame. (Step 3030). One is added to the current frame number (step 3035). It is determined whether or not the volume of the currently read frame is higher than the maximum volume (step 3040). If the volume is higher, the maximum volume frame number is replaced with the current frame number and the maximum volume is replaced with the volume value of the current frame (step 3050). ).
[0048]
Next, before taking the volume difference from the frame before the α-frame, it is necessary to check whether the frame before the α-frame exists in the same scene. Therefore, it is checked whether or not the value of the frame counter is α + 1 or more. (Step 3060). If the volume is greater than or equal to α + 1, the volume can be compared with the volume before the α-frame. Therefore, after detecting the volume before the α-frame (step 3070), the frame in which the increase in the volume is maximum is taken as the maximum volume difference frame It is checked whether or not the maximum volume difference is smaller than "(current frame volume)-(volume before α frame)" (steps 3080, 3090). When the frame having the maximum absolute value of the volume difference is taken as the maximum volume difference frame, it is checked whether the frame is smaller than the absolute value of “(current frame volume) − (volume before α frame)” (step 3080, 3100). When the volume is small, the current volume difference becomes the maximum volume difference. Therefore, the current frame number is used as the maximum volume difference frame number, and the absolute value of “(current frame volume) − (volume before α frame)” is used as the maximum volume difference. , Respectively (step 3110). Further, if the value of the maximum volume difference is smaller than 0 (step 3120), α is subtracted from the maximum volume difference frame number (step 3130). This is because, when taking a difference in the sound volume decreasing direction, it is desirable to extract a frame at the beginning of the decrease as a frame of the sound volume decreasing.
[0049]
Next, the current frame volume is added to the total volume (step 3140), and it is checked whether or not the next frame exists. If the next frame exists, the next frame is read (step 3160), and through the scene switching detection processing (step 3170), it is checked whether or not the scene switching flag is set (step 3180). If the flag is not set, 1 is added to the frame counter (step 3210), and 1 is added again to the current frame number (step 3035). If the flag is set, the value of the current frame number as the scene end frame number 8015, the value of the frame counter as the scene length 8020, and the value of the frame counter as the average volume 8030 in a new line of the scene feature table 8010 shown in FIG. The value of the variable maximum volume frame number as the maximum volume frame number 8040, the variable maximum volume value as the maximum volume 8050, and the variable maximum volume difference as the maximum volume difference frame number 8060. The value of the variable maximum volume difference is registered as the value of the frame number as the maximum volume difference 8070 (step 3190). If the end flag is set, the scene feature table creation processing is ended. Returning to step 3030), the processing is continued. The operation example of the scene feature table creation processing (step 2030) in FIG. 3 has been described above.
[0050]
Next, an operation example of the scene switching detection process (step 3170) shown in FIG. 3 will be described with reference to FIG. First, it is checked whether there is a correlation between the previous frame and the current frame (step 4010) and whether there is a motion vector (step 4020). If there is no correlation or motion vector between the previous frame and the current frame, It is determined that a scene change has occurred, a scene change flag is set (step 4030), and the scene change detection process ends. When at least one of the correlation or the motion vector is detected, it is determined that the previous frame and the current frame are in the same scene, the scene switching flag is lowered (step 4040), and the scene switching detection process is performed. To end. The operation example of the scene switching detection processing (step 3170) in FIG. 4 has been described above.
[0051]
Next, an operation example of the characteristic portion extraction process (step 2040) shown in FIG. 2 will be described with reference to FIG. First, scene lengths a and b, sound volumes c, d, and e, and a maximum sound volume difference f are set by the input unit 1020 as parameters for selecting characteristic portions (step 5010). In addition, the input unit 1020 specifies the number of frames before and after the feature frame to be extracted as a feature portion by using a variable x. In addition, when the scene length is b or longer, the input unit 1020 specifies whether to extract a plurality of characteristic portions (step 5020). The first line of the scene feature table 8010 is read (step 5030), and the value of the scene counter is set to 0 (step 5040).
[0052]
After that, the cases are classified according to the length of the scene. First, in order to see whether or not the target scene is a short scene of a certain length or less, it is checked whether or not the scene length 8020 is a or less (step 5050). In order to see whether or not the scene is longer than a certain length, it is checked whether or not the scene length 8020 is longer than b (step 5080). If the scene length 8020 is equal to or less than a, it is further checked whether or not the average volume 8030 is equal to or more than d (step 5060). If the average volume 8030 is equal to or greater than d, it is determined that the scene length 8020 is a characteristic part even if it is short, and the maximum volume 9020 is set to the maximum volume 9020 in order to extract all frames of the scene into the extraction table. 8050, the maximum volume difference 9030, the maximum volume difference 8070, the start frame number 9040, "(scene end frame number)-(scene length) +1", and the end frame number 9050, the scene end frame number 8015. After each registration (step 5070), 1 is added to the scene counter (step 5170). If the average volume 8030 is lower than d, it is determined that the part is not characteristic, and nothing is performed.
[0053]
If the scene length 8020 is not equal to or greater than b, it is determined that the scene is an intermediate scene length, and it is checked whether the average volume 8030 is equal to or greater than e (step 5130). When the average sound volume 8030 is equal to or higher than e, it is determined that the portion having the highest sound volume is the most characteristic portion in the scene, and the frame of the maximum sound volume frame number and the x frames before and after it are extracted in the extraction table. Therefore, the value of the maximum volume 8050 is set to the maximum volume 9020, the value of the maximum volume difference 8070 is set to the maximum volume difference 9030, "(maximum volume frame number) -x" is set to the start frame number 9040, and " (Maximum volume frame number) + x "is registered (step 5140), and 1 is added to the scene counter (step 5170). If the average volume 8030 is smaller than e, it is checked whether the maximum volume difference 8070 is equal to or greater than f (step 5150). If the maximum volume difference 8070 is equal to or greater than f, it is determined that the portion having the maximum volume difference is the most characteristic portion in the scene, and the frame of the maximum volume difference frame number and the x frames before and after the frame are extracted. In order to extract the values in the table, the value of the maximum volume 8050 is set to the maximum volume 9020, the value of the maximum volume difference 8070 is set to the maximum volume difference 9030, "(maximum volume difference frame number) -x" is set to the start frame number 9040, and the end is set. “(Maximum volume difference frame number) + x” is registered as the frame number 9050 (step 5160), and 1 is added to the scene counter (step 5170). If the maximum volume difference is smaller than f, it is determined that the scene is not characteristic, and nothing is performed.
[0054]
On the other hand, if the scene length is equal to or longer than b, it is determined that the scene is longer than a certain length, and it is first checked whether or not a plurality of characteristic portions are extracted from the scene (step 5090). When extracting a plurality of characteristic portions, 0 is set in a scene sub-counter for counting the number of the characteristic portions (step 5100), and the feature of the scene is obtained by a plurality of characteristic portion extraction processing in the same scene (step 5110). The part is extracted, "(scene sub counter) -1" is added to the scene counter (step 5120), and then 1 is added to the scene counter (step 5170). That is, considering step 5120 and step 5170 together, it is equivalent to adding the value of the scene sub-counter to the scene counter. If a plurality of characteristic parts are not extracted from the same scene, the part having the highest volume is determined to be the most characteristic part in the scene, and the frame with the maximum volume frame number and the x frames before and after it are extracted from the extraction table. , The value of the maximum volume 8050 in the maximum volume 9020, the value of the maximum volume difference 8070 in the maximum volume difference 9030, “(maximum volume frame number) −x” in the start frame number 9040, and the end frame number “(Maximum volume frame number) + x” is registered in 9050 (step 5140), and 1 is added to the scene counter (step 5170).
[0055]
After passing through any of these steps, it is checked whether or not the next line exists in the scene feature table 8010 (step 5180). If so, the line is read and if the scene length is again less than or equal to a. A check (step 5050) continues the processing. If not, the characteristic part extraction processing ends. The operation example of the characteristic portion extraction processing (step 2040) in FIG. 5 has been described above.
[0056]
Next, an operation example of the multiple feature portion extraction process in the same scene (step 5110) shown in FIG. 5 will be described with reference to FIG. First, the same number of flags as the number of frames of a scene from which a plurality of characteristic portions are extracted are prepared, and numbers from “(scene end frame number) − (scene length) +1” to scene end frame numbers are assigned. One-to-one correspondence is made (step 6010). The first frame of the scene to be extracted is read (step 6020), and it is checked whether or not the volume of the current frame is higher than the volume c (step 6030). If the volume is equal to or higher than c, a flag corresponding to the current frame and x frames before and after the current frame is set. Thereafter, it is checked whether or not the next frame exists. If so, the frame is read (step 6060), and it is again checked whether or not the frame volume is equal to or higher than the volume c (step 6030). If the next frame does not exist, it is checked how many times the flag is temporally continuous and the value is set in the scene sub-counter (step 6070). Finally, the frame corresponding to the flag is set. To extract the maximum volume 9020, the maximum volume difference 9030, the maximum volume difference 8070, the start frame number 9040, the start flag number of the continuous portion, and the end frame number 9050 The end numbers of the continuous portions are sequentially registered for all the continuous portions (step 6080), and the multiple feature portion extraction process in the same scene ends. The operation example of the multiple feature portion extraction process in the same scene (step 5110) in FIG. 6 has been described above.
[0057]
Next, an operation example of the search data selection process (step 2050) shown in FIG. 2 will be described with reference to FIG. First, a maximum allowable limit number of scenes specifying the maximum number of scenes of the search data and a maximum allowable limit number of frames specifying the maximum frame number are specified. In addition, in order to prevent the number of frames constituting one characteristic portion from becoming too small to make the contents incomprehensible, a minimum necessary limit number of frames for designating the lower limit of the number of frames constituting one scene is specified. Furthermore, it is specified which of the characteristic part with the largest volume and the characteristic part with the largest difference in volume has priority as the search data (step 7010). Next, it is checked whether or not the value of the scene counter is larger than the maximum allowable limit number of scenes (step 7020). When the value of the scene counter is larger than the maximum allowable limit number of scenes, it is necessary to reduce the number of extracted scenes. Therefore, a feature scene with a large maximum volume is prioritized as search data over a feature scene with a large maximum volume difference. It is checked whether or not the priority is higher (step 7030). If priority is given, the scene with the smallest maximum volume difference is deleted from the extraction table, if not, the scene with the smallest maximum volume is deleted (step 7040) (step 7050). After that, 1 is subtracted from the scene counter (step 7055), and the process returns to step 7020 again until the value of the scene counter becomes equal to or less than the maximum allowable limit number of scenes. At this time, if there are a plurality of scenes with the smallest maximum volume or maximum volume difference, the scene with the smallest end frame number is deleted.
[0058]
If the number of extracted scenes is equal to or less than the maximum allowable limit number of scenes, first, the number of extracted frames which is the sum of “(end frame number) − (start frame number) +1” of each row of the extraction table is calculated (step 7058). ), 0 is set to a deletion prohibition scene counter (step 7060). The deletion prohibition scene counter is a counter that counts the number of scenes in which the number of frames constituting one scene is smaller than the minimum necessary frame number. Next, it is checked whether the value of the deletion prohibition scene counter is equal to the value of the scene counter (step 7070). If they are equal, it means that there is no scene that can be deleted, and the search data selection process ends. If not, it is checked whether the number of extracted frames is larger than the maximum allowable frame number (step 7080). If the number of extracted frames is equal to or less than the maximum allowable frame number, there is no need to delete frames, and the search data selection process ends.
[0059]
If the number of extracted frames is larger than the maximum allowable limit number of frames, the number of extracted frames must be reduced. First, the scene length of the first scene is set to “(end frame number) − (start frame number). +1 "is calculated (step 7090), and it is checked whether or not the scene length is larger than the minimum necessary frame number (step 7100). If the scene length is larger than the minimum required number of frames, it is possible to reduce the number of frames from that scene, so add 1 to the start frame number of the scene and delete 1 Subtract 1 from the number (step 7110) and subtract 2 from the number of extracted frames (step 7115). If the scene length is not larger than the minimum necessary frame, if the number of frames is further reduced, the contents of the scene cannot be understood, so that it is considered impossible to delete a frame from the scene, and 1 is set in the deletion prohibition counter. Is added (step 7120). Next, it is checked whether or not the next scene exists in the extraction table (step 7130). If there is, the scene length of the next scene is calculated (step 7140), and this is repeated from step 7100 until the next scene no longer exists. If the next scene no longer exists, the process returns to step 7070, and this process is repeated until the deletion-prohibited scene counter is equal to the scene counter and the number of extracted frames is equal to or less than the maximum allowable frame number. The operation example of the search data selection process (step 2050) in FIG. 7 has been described above.
[0060]
In the above embodiment, the search data generation process is performed every time a search is performed. However, if the search data selection management unit 1043 in FIG. 1 has a function of retaining an extraction table after a search, it is not necessary to perform the process each time. Absent. At this time, the main control unit 1010 checks the search data selection management unit 1043 for the moving image file for which the browse instruction has been received, and if the search data has already been generated, directly based on the search data. indicate. Only when it is not generated, the search data generation unit 1040 creates search data.
[0061]
Further, in the above embodiment, various parameters for generating search data are set by the user using the input unit 1020 of FIG. 1. However, scene features held in the search data selection management unit 1043 are stored. It is also possible to use table information as a parameter. For example, a value 1.5 times the average volume of the scene can be used in FIG. This can be achieved by setting c = (average volume) × 1.5 at the start of the multiple feature portion extraction process 5110 in the same scene without setting c in step 5010. In step 5010, it is also possible to use a value of 10% of the scene length of the entire moving image file as the scene length b, or to use a value twice as large as the average sound volume of the entire moving image file as the sound volume e. In these cases, b = (sum of scene lengths in the scene feature table) × 0.1 and e = [{sum of (scene length × average volume)｝ ÷ (sum of scene lengths)] × 2 are shown in FIG. May be set in step 5010.
[0062]
When AV data is currently handled, a tape such as a VTR is mainly used as a medium for storing video. A tape as a storage medium can be said to be a sequential access and a low-speed mass storage. In a system using such a medium, in order to enhance the effect of the search, the search data selection management unit 1043 may store the search data in another storage medium (such as a hard disk) capable of high-speed random access. Conceivable. According to this method, a high-speed search of a moving image file becomes possible, and a user can reproduce only a necessary scene from a tape.
[0063]
Also, in a digital video editing device or the like, there is an operation of A / D conversion from an analog tape to digital media (a hard disk or the like) in the device and storing (downloading) the data. It may be performed sometimes. Then, after the download is completed, a high-speed search for the moving image file can be performed immediately.
[0064]
As described above, according to the present embodiment, a search image (moving image) is extracted based on a frame in which scene switching has occurred and audio information of the frame, and a search for arbitrary AV data (file) is performed. Becomes feasible.
[0065]
As a result, unlike the related art, it is not necessary for the user to specify the characteristic portion to be searched in advance and create the search data before performing the search, and the effect of reducing the user's workload can be obtained. .
[0066]
【The invention's effect】
As described above, according to the present invention, for an important (characteristic) scene, a plurality of frames can be extracted as detection data for each scene, and the detection data can be finely extracted.
[0067]
Further, according to the present invention, for an unimportant scene, the number of search data can be reduced without extracting search data from the scene.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating an overall configuration of a moving image search data generation method according to an embodiment of the present invention.
FIG. 2 is a flowchart showing the operation of the embodiment of FIG. 1;
FIG. 3 is a flowchart illustrating an operation of a scene feature table creation process in the flowchart of FIG. 2;
FIG. 4 is a flowchart illustrating an operation of a scene switching detection process in the flowchart of FIG. 3;
FIG. 5 is a flowchart showing an operation of a feature scene extraction process in the flowchart of FIG. 2;
FIG. 6 is a flowchart showing an operation of a process of extracting a plurality of characteristic portions in the same scene in the flowchart of FIG. 5;
FIG. 7 is a flowchart illustrating an operation of a search data selection process in the flowchart of FIG. 2;
FIG. 8 is a scene feature table created in the process of generating search data.
FIG. 9 is an extraction table created in the process of generating search data.
FIG. 10 is an explanatory diagram related to an MPG1 encoding method.
FIG. 11 is a diagram showing a device configuration that can realize the present invention.
FIG. 12 is an explanatory diagram relating to a block matching method.
FIG. 13 is an explanatory diagram relating to a method of detecting a volume of each frame.
FIG. 14 is an explanatory diagram relating to a method of detecting a volume of each frame.
[Explanation of symbols]
1010: Main control unit
1020 ... input unit
1030 Data storage management unit
1031: Data storage unit
1032: Data reading unit
1040 ... search data generation unit
1041 scene change detection unit
1042 ... Volume detector
1043... Search data selection management unit
1050 ... Display unit
1060 ... Communication unit

Claims (7)

Moving image data representing a moving image composed of a plurality of sequential image frames, and sequential audio data associated with each of the frames are input, and are included in the moving image from the input moving image data. In a video search system that extracts search data for each scene,
Based on the content of the moving image data to be input, for each of the scenes, a scene switching detection unit for detecting a frame constituting the scene,
Volume detection means for detecting the volume of the input audio data for each corresponding frame,
For each scene detected by the scene switch detection unit, moving image data including a frame in which a volume equal to or higher than a predetermined value is detected by the volume detection unit is used as the search data for each scene from the scene. Search data extraction means to be extracted;
With
The search data extraction means,
A moving image search system , wherein the number of scenes from which the search data is extracted is limited to the number specified as the maximum number of scenes from which the search data is extracted . Moving image data representing a moving image composed of a plurality of sequential image frames, and sequential audio data associated with each of the frames are input, and are included in the moving image from the input moving image data. In a video search system that extracts search data for each scene,
Based on the content of the moving image data to be input, for each of the scenes, a scene switching detection unit for detecting a frame constituting the scene,
Volume detection means for detecting the volume of the input audio data for each corresponding frame,
For each scene detected by the scene switching detection unit, a frame whose volume change with respect to a frame separated by a predetermined interval is equal to or more than a predetermined value based on the volume of each frame detected by the volume detection unit. A search data extraction unit that extracts moving image data from the scene as the search data for each scene,
With
The search data extraction means,
A moving image search system , wherein the number of scenes from which the search data is extracted is limited to the number specified as the maximum number of scenes from which the search data is extracted . The moving image search system according to claim 1 or 2 ,
The search data extraction means,
Among the scenes detected by the scene switching detection unit, comprising a calculation unit that calculates an average volume in the scene based on the volume detected by the volume detection unit,
A moving image search system, wherein search data is extracted only for scenes whose average sound volume calculated by the calculation means is higher than or equal to a predetermined sound volume. The moving image search system according to claim 1 ,
The search data extraction means,
For each scene detected by the scene switching detection means, comprising a detection means for detecting a frame having a maximum volume,
A moving image search system, wherein moving image data including a frame detected by the detection unit is extracted as the search data for each scene. The moving image search system according to claim 2 ,
The search data extraction means,
For each scene detected by the scene switching detection unit, extracting moving image data including a frame having a maximum volume change with respect to frames separated by a predetermined interval as the search data for each scene. Characteristic video search system. The moving image search system according to claim 1 or 2 ,
The search data extraction means,
A moving image search system , wherein the number of frames extracted from each scene detected by the scene switching detection means is limited to a number specified as a maximum number of frames extracted from each scene . Moving image data representing a moving image composed of a plurality of sequential image frames, and sequential audio data associated with each of the frames are input, and are included in the moving image from the input moving image data. In a video search system that extracts search data for each scene,
Based on the content of the moving image data to be input, for each of the scenes, a scene switching detection unit for detecting a frame constituting the scene,
Volume detection means for detecting the volume of the input audio data for each corresponding frame,
For each scene detected by the scene switching detection unit, moving image data including a frame in which a volume equal to or higher than a predetermined value is detected by the volume detection unit is used as the search data for the scene from the scene. First search data extraction means to be extracted;
For each scene detected by the scene switching detection unit, a frame whose volume change with respect to a frame separated by a predetermined interval is equal to or more than a predetermined value based on the volume of each frame detected by the volume detection unit. Second search data extraction means for extracting moving image data from the scene as the search data for the scene,
Means for switching and enabling one of the first search data extraction means and the second search data extraction means in accordance with an external instruction;
With
The first and second search data extracting means include:
A moving image search system , wherein the number of scenes from which the search data is extracted is limited to the number specified as the maximum number of scenes from which the search data is extracted .

Images