JP2007184674A - Digest making system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a means for making a digest video image efficiently for the program of a variety of genres. <P>SOLUTION: An application feature amount is calculated from the physical features of a video image by a plurality of feature extraction modules capable of plug-in. When the importance of a scene is determined from the application feature amount according to an importance calculation expression, correction of the calculation expression and alteration of parameters can be carried out easily, and the importance calculation expression is constituted interactively using a user interface where comparison before and after correction is facilitated. Information of importance calculation expression consisting of a set of the importance calculation expression thus obtained, and the feature extraction modules and parameters to be utilized is stored and distributed and reused for making the digest of a program in the same genre. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a video processing technique for processing a video, and more particularly to a technique for automatically creating a digest video as a summary from a given video.

  In recent years, it has become possible to record videos of a large number of broadcast channels for a long time by improving the performance of video information equipment. Also, the spread of satellite broadcasting and the like has increased the number of broadcast channels, and the number of programs provided to viewers is increasing more than double the conventional number. As a result, there is a need for viewers to efficiently select programs that suit their taste from a large number of programs.

  As a solution to this, it has been proposed to provide a digest video creation function in a receiver or a recording device. For example, (Patent Document 1) discloses a method of creating a digest video by extracting an important part of a program from the volume and telop display of the program.

  (Patent Document 2) discloses a method of creating a digest video based on event information transmitted along with a program.

JP 2000-23962 A Japanese Patent No. 3176893

  However, these conventional techniques can only create a digest video for a limited broadcast program. That is, in the former case, as a result of extracting an important part from a specified small number of information, a digest video can only be created for a program of a specific genre. In the latter case, since it is assumed that event information is attached to the program, it is not always possible to create a digest video for all broadcast programs.

  In order to solve the above-mentioned problem, the present invention provides a more effective digest creation device with a configuration having the following features.

  A first feature of the present invention is a method for creating a digest video for video data using video data and a feature amount extracted from the video data, video storage means for storing the video data, and the video data A basic feature storage means for storing a feature quantity calculated from a physical principle or a symbol processing principle from the image, and receiving the contents of the video storage means and the basic feature storage means as an input A plurality of feature extraction modules for calculating an applied feature amount corresponding to a sense, and an operation control unit for performing an arithmetic operation between the applied feature amounts calculated from the plurality of feature extraction modules and calculating an importance level for the video data And a digest creation method and apparatus for determining the content of a digest video based on the importance.

  According to a second aspect of the present invention, there is provided the digest creation method and apparatus according to the first feature, wherein the feature extraction module can be added from the outside in accordance with an operator's instruction. To do.

  A third feature of the present invention is the digest creation method and apparatus characterized in that the feature extraction module has means for changing a feature calculation parameter used for calculating the applied feature amount according to the degree of preference of the viewer. I will provide a.

  According to a fourth feature of the present invention, the calculation control unit applies an importance calculation formula of a predetermined format input from the outside between a plurality of applied feature amounts obtained from the plurality of feature extraction modules. The digest creation method and apparatus are characterized in that the importance is calculated according to the above.

  According to a fifth feature of the present invention, the importance calculation formula includes a description that specifies the plurality of feature extraction modules that the importance calculation formula requires for calculating the importance, and the plurality of feature extraction modules. Providing a digest creation method and apparatus characterized in that it includes a description for specifying a feature calculation parameter used in, and a description of a calculation formula for calculating the importance using applied feature amounts obtained from a plurality of feature extraction modules To do.

  According to a sixth aspect of the present invention, there is provided a digest creation method and apparatus characterized in that an operator inputs the importance calculation formula interactively.

  According to a seventh aspect of the present invention, there is provided a digest creation method and apparatus characterized in that the importance calculation formula is inputted from the outside through communication means.

  According to an eighth aspect of the present invention, there is provided a digest creation method and apparatus characterized by having a user interface for displaying the plurality of feature extraction modules and the importance calculation formula in association with each other.

  According to a ninth feature of the present invention, in the digest video creation method and apparatus of the eighth feature, when the user changes the feature extraction module, or the value of the feature calculation parameter given to the feature extraction module by the user is changed. In such a case, a digest creation method and apparatus characterized by having a user interface for displaying a difference between digest videos created before and after the change is provided.

  As described above, according to the present invention, it is possible to efficiently create a digest video for a video program spanning various genres. In addition, by sending and receiving importance calculation formulas through networks such as the Internet, digests created by other institutions and individuals can be used with each other, enabling viewers to view video programs more efficiently. It becomes possible.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a block diagram showing the entirety of a digest creation method and apparatus according to the present invention. The apparatus of the present invention is roughly divided into two parts, a recording / reproducing unit 180 and a digest creating unit 100.

  The recording / playback unit 180 inputs the broadcast video 190 and the media video 191 from the outside through the input unit 181. The input is recorded in the video recording means 184 or sent to the decoding unit 185 under the control of the recording / playback control unit 183.

  Further, the video already recorded in the video recording means 184 may be sent to the decoding unit 185. These controls are performed by the recording / playback control unit 183 according to an operation performed by the user using the operation unit 182. The output of the decoding unit 185 is displayed as a video on the display device 186 via the recording / playback control unit 183, and used for viewing by the user.

  The video data recorded in the video recording means 184 is data obtained by decoding the encoded data by the decoding unit 185 even if it is data encoded by MPEG-2 (Motion Picture Experts Group) or the like. It may be. The operation of the recording / reproducing unit 180 is not different from that of a general video recording device such as a VTR or a DVD recorder.

  In the present invention, in addition to the above, a digest creation unit 100 is provided. The digest creation unit 100 includes video data (attached moving image data, audio data, and symbol data recorded in the video recording means 184 or directly obtained from the recording / playback control unit 183. Also, the input unit 181 Either the encoded data obtained from the above or the data decoded by the decoding unit 185. The same applies hereinafter) 110 is input, and the reproduction position data 120 is provided to the recording / reproducing unit 180 through a predetermined process. To do. The processing performed by the digest creation unit 100 will be described below.

  The video data 110 is first input to the basic feature extraction unit 111. From this, a physical feature for the video data is calculated by a predetermined calculation, or a symbol feature based on the symbol processing is generated and stored in the basic feature storage unit 112. Here, the physical feature is a statistic calculated from video data based on a physical principle. Specifically, for example, as shown in FIG. 2, it consists of basic acoustic feature data 201 such as spectrum data, basic image feature data 202 such as region information, etc., and is stored in the physical feature storage means 112 in the form shown. The physical feature expresses the characteristics of the video data, but does not necessarily match the importance of the video in accordance with the viewer's feeling. In addition, the symbol feature is a symbol string included in video data, for example, a new symbol string obtained by decomposing part-of-speech such as a noun or a verb from a character string explaining a video such as subtitles, or a method for reading a character string. Phonetic symbol string. Specifically, for example, as shown in FIG. 12, subtitle part-of-speech information 1210 as subtitle basic feature data, appearance time 1220, part-of-speech 1221, and corresponding word 1222 are extracted from subtitle information 110 in the video data. “Subtitle part-of-speech feature extraction” is applicable. When the physical feature and the symbol feature are already attached to the video data 110 and sent, they may be directly input to the basic feature storage unit 112 or directly input to the basic feature extraction unit 111.

  The above physical features are converted into applied feature amounts 121-1 to 121-n by a plurality of feature extraction modules 120-1 to 120-n. As described above, the applied feature amount is a feature amount corresponding to the cognitive sensation of the video viewer, for example, whether the sound of a certain scene is music, human speech, background noise, Represents the difference. For example, in the case of detecting “musical feature” indicating music, a result is obtained by performing calculation according to the procedure shown in the flowchart of FIG. That is, first, a threshold value θ, which is a criterion for determining musicality, is acquired as a part of a parameter from the importance calculation formula information storage unit 140 described later (step 301). Next, the left and right in-phase power Ps of stereo sound is calculated with reference to the contents of the physical feature storage means 112 (step 302). Similarly, the left and right different phase power Pa is calculated (step 303), and it is determined whether or not the ratio of both exceeds the above threshold value θ (step 304), so that there is music (musicality = 1) or no music. (Musicality = 0) and the result are determined (steps 305 and 306). The obtained value is passed to the arithmetic control unit 130 described later and used for calculation of importance. This calculation is performed in response to a request from the arithmetic control unit 130.

  In the above example, the feature extraction module is realized by a software program. However, the feature extraction module may be realized by using dedicated hardware. Also, as shown in FIG. 1, any number of feature extraction modules may be used according to the purpose of the viewer, and they may be added or deleted as necessary.

  In the above example, the applied feature quantity is given as a logical value representing true / false, but the applied feature quantity may be a quantity having a physical dimension or a value such as continuous confidence used in fuzzy logic. .

  The applied feature amount obtained as described above is used by the arithmetic control unit 130 to calculate the importance based on the importance calculation formula information 141. This calculation is performed interactively with the user through the digest creation console 150.

  The above situation will be described with reference to FIGS. In the following description, the display device 151 of the digest creation console 150 may physically double as the display device 186 of the recording / playback unit 180, and the operation unit 152 of the digest creation console 150 is the operation unit 182 of the recording / playback unit 180. May be physically combined.

  FIG. 4 shows a calculation control screen 400 that the calculation control unit 130 displays on the display device 151 of the digest creation console 150. On the screen, a plurality of feature extraction modules used for calculating the importance are displayed in the form of module display units 401-1 to 401-n. The user inputs the importance calculation formula in the input field 411, and performs an operation between the applied feature amounts obtained from each module to obtain the final importance.

  Here, the importance calculation formula can also be loaded from the importance calculation formula information storage means 140 using the read button 412. Alternatively, the importance calculation formula may be loaded from the external server 171 through the external network 170. Further, when a new feature extraction module is required to calculate the importance, it can be added by a module addition button 404.

  Conversely, when a module is no longer needed, it can be deleted with the module deletion button 403. When inputting a parameter for each module, the module parameter input button 402 is used to input the module display section 401-n of the module. The module display unit 401-n includes a module name display unit 501 and a use parameter list 502 as shown in FIG.

  The user uses the scroll buttons 504 and 505 to select an appropriate parameter in the parameter list 502, for example, 503, and inputs a value from the operation unit 152.

  As described above, when the importance calculation method is determined, the user uses the calculation button 413 to calculate the importance. It is possible to name the importance calculation formula information 141 using the input field 414, and the importance calculation formula information created in the past using this name can be reused.

  FIG. 6 shows an example of the importance degree calculation formula information 141 used in the arithmetic control unit 130. The importance calculation formula information 141 includes a feature extraction module list 601 and a calculation formula 602 used in the calculation formula. In the feature extraction module list 601, a list of parameters necessary for each feature extraction module is given to each. In the calculation formula 602, specific values of each parameter are specified and the formula is described.

  FIG. 7 shows a calculation result screen 700 that displays the calculation result of importance by the arithmetic control unit 130. This screen is displayed on the display device 151 of the digest creation console 150 when the calculation by the calculation button 413 is completed. The calculation result screen 700 has a time display area 701, applied feature amount display areas 702-1 to 702-n corresponding to each of a plurality of feature extraction modules, and a result display area 703 showing the result of calculating the importance from these.

  Each area indicates each applied feature amount at the time indicated by the time display area 701 and the importance obtained finally. 704 and 705 are scroll buttons used when the display does not fit on the screen. Using this screen, the user confirms whether or not the importance calculation is as intended. If the result is good, the importance calculation formula information 141 is used to create a digest video.

  In the above description, no particular mention was made as to what time unit the importance is calculated. The time unit is calculated for each video frame, the program is calculated by dividing the program into fixed time intervals (for example, 10 seconds), the program is divided into several scenes by some method, and each scene unit is calculated. However, the present invention can be applied to any of these methods.

  In addition, when satisfactory results are not obtained in FIG. 7, the importance calculation formula may be interactively corrected using the change scene confirmation screen 800 shown in FIG. The change scene confirmation screen 800 has two result display areas below the time display area 701.

  That is, the pre-change result display area 803 displays the result based on the pre-change importance calculation formula 801, and the post-change result display area 804 displays the result based on the pre-change importance calculation formula 802. The portion where the difference between the two has occurred is displayed in the change portion display area 805, and change portion images 806, 807, etc. are displayed corresponding to each change portion.

  As a result, the user can more efficiently modify the importance calculation formula, the accompanying feature extraction module, and parameters.

  As described above, the importance is determined by the arithmetic control unit 130. This result is stored in the importance level storage means 160, and stored in the playback position storage means 162 in the format of the playback point list 900 shown in FIG. By providing the playback point list 900 as the playback position data 120, it is possible to provide a digest video to the viewer.

  In the above description, the case where the user becomes the operator himself and creates the importance calculation formula information 141 using the calculation control screen 400 or the like is shown. Such a creation operation requires a certain level of expertise regarding video information processing, and therefore cannot always be performed by a general viewer.

  Therefore, it is assumed that a general viewer often views the digest video by obtaining the importance calculation formula information 141 already created by another person and using it as it is. FIG. 10 shows an example of a screen for reading importance calculation formula information for this purpose. The importance calculation formula information reading screen 1000 shown in the figure is displayed on the digest creation console 150. The user inputs the name of the server in which the importance calculation formula information 141 is stored by operating the input button 1002 to the internal server name display unit 1001.

  This server may be the importance calculation formula information storage means 140 included in the digest video apparatus in FIG. In the latter case, the location of the server is specified using a URL or IP address used on the Internet. As a result of this input, an expression name list 1003 is displayed on the screen.

  These formula names are given in the input field 414 in FIG. 4 when the importance calculation formula information 141 is created by a third party. The user operates the cursor 1006 using the scroll buttons 1004 and 1005 to select a desired formula name, and the decision button 1007 can use the importance calculation formula information to which the formula name is attached.

  Thereafter, the digest is viewed using, for example, a recording / playback operation screen 1100 as shown in FIG. This screen has a button group 1110 used in a normal recording / playback device, that is, a rewind button 1111, a stop button 1112, a still button 1113, a play button 1114, and a fast forward button 1115, but in addition to this, a digest button 1101 is provided. When the digest button 1101 is pressed, a digest video created based on the importance calculation formula information 141 read in FIG. 10 is displayed on the display device 186 of the recording / playback unit 180.

  Note that as a result of pressing the digest button 1101, the importance calculation formula information reading screen 1000 may be displayed. Further, the importance level calculation formula information reading screen 1000 may be displayed as a result of pressing the reading button 412 in FIG.

  As a next embodiment, a case where a race portion is generated as a digest from a horse racing program using video data will be described. Note that the user interface for specifying the module to be used and the parameter is not referred to here as it conforms to the first embodiment, and is not referred to here. The basic feature extraction units 111-1 to 111-n, the calculation control unit 130, and the feature extraction module The processing and output in 120-1 to 120-n will be described.

  In the basic feature storage unit 112, as shown in FIG. 13, the sound power feature data 1303 as the sound basic feature data 1301 is the sound power feature extraction unit (one of 111) and the image color as the image basic feature data 1302 Assume that the composition information 1304 is extracted from the image color composition feature extraction unit (one of the other 111) and input image data 110 and prepared in the basic feature storage means. This method will be described below.

  The acoustic power feature extraction unit 111-n extracts the acoustic power feature based on the flow of FIG. Here, it is assumed that the input acoustic data includes sample data sampled (sampled) at sufficiently short intervals for each of the left and right, and these are referred to as acoustic samples. The sound power feature is extracted at a time interval longer than the sampling interval, and this interval is used as the sound basic feature feature interval for 1/30 seconds, that is, 30 sound power features per second are extracted as the basic feature.

  First, the acoustic time t is initialized to 0 (step 1401), the number counter is initialized to 0 (step 1402), and the sums sumL and sumR are initialized to 0 and 1403, respectively. If the acoustic time is within 1/30 (step 1404) and an acoustic sample exists (step 1405), the acoustic sample is input (step 1406), and the squared value is added to the sum (step 1407). The interval between the acoustic samples is added to t (step 1408), and 1 is added to c (step 1409). When these processes are larger than 1/30 which is the basic acoustic feature feature interval (step 1404), the time ts of the last input acoustic sample is acquired (step 1410). Next, the square root of the sum divided by the number counter c is taken (step 1411) and output to the basic feature storage means 112 together with ts (step 1412). These are repeated until an acoustic sample exists (step 1405).

  The image color composition feature extraction unit 111-n extracts image color composition features based on the flow of FIG. Here, the color composition is defined as a set in which one image is equally divided into four vertical and horizontal portions and the color average of each region obtained by dividing the image into 16 region images is obtained.

  Further, it is assumed that the moving image data of the input video data 110 is a series of images called frames, and the image color composition feature is processed for each image. Furthermore, in the following, the color of the pixels constituting the image will be described as a YCbCr system composed of luminance and color difference defined in many digital broadcasts, but it may be an RGB system or other color designation system.

  In addition, even if the interval between image frames is about 1/30 second, which is a typical broadcast frame interval in Japan, a process for degenerating them is provided in the previous stage for 1/2 second ( It may be reduced to 2 sheets per second).

  First, the size (number of vertical and horizontal pixels) of one image is acquired (step 1501), the limit coordinates for dividing into 4 × 4 partial areas are obtained (step 1502), and the luminance B and color differences Cb and Cr are calculated. All are initialized to 0 as an array of 16 so as to correspond to each region (step 1503). If a frame image is input (step 1504), the frame image is acquired (step 1505). The coordinate counters x and y are initialized to 0 (step 1506), and color information (luminance value and color difference) is acquired for each pixel in the image frame while increasing x and y (step 1510). Since the corresponding divided region number r is obtained from x and y (step 1511), the color information of the pixel at (x, y) is added to the color information of that region (step 1512). After performing these processes for all the pixels, the time tf of the frame image is obtained (step 1513), and the average value of the luminance array B [r] and the color difference arrays Cb [r] and Cr [r] that have been added Are determined as Mb [r], Mcb [r], and Mcr [r], respectively (step 1514). These values range from 0 to 255. In this color composition feature extraction, the luminance Mb [r] has 4 levels (2 bits), and the color differences Mcb [r] and Mcr [r] have 2 levels (1 bit). ) And is represented by 4 bits for one area. As a result, since all 16 areas can be expressed by 4 × 16 = 64 bits, the color composition is expressed as a 64-bit bit string while paying attention to the order of the areas (step 1515).

  This color composition and the time ts of the frame image obtained previously are output to the basic feature extraction storage means (step 1516), and processing is performed until no frame image is input (step 1504).

  Next, a method for specifically extracting a race portion of a horse race will be described by combining the processing of a plurality of feature extraction modules based on the basic features obtained so far. The race part in the horse racing program is combined with extraction according to the following three conditions.

  First, as condition (1) which is a race portion, it is assumed that the acoustic power is maintained at a certain value or more for a certain time or more. However, it is assumed that it does not completely maintain a certain value or more, but sometimes allows it to fall below the certain value.

  This condition (1) focuses on the use of running sounds and cheering of the spectators produced by the horse group to create a sense of realism during the race as a feature of program production in the race part. The traveling sound and cheering have been found by investigations that many horses and people have complicated mixing of sounds with completely different timing and frequency characteristics, so that the sound power tends to take a value above a certain value.

  In addition, it is known that although there are a plurality of such distances, the race takes at least about 1 minute and the race is completed in about 5 minutes at most. There are announcer voices and BGM music in corners that exist in programs other than races, but in many cases one person, at most two to three people speak at the same time, the time is 1 minute. It is known that this condition (1) is not satisfied and can be excluded to some extent because it is often less.

  As conditions (2) for being a race part, the colors on the screen should be a lawn color (green to yellow to brown) or a dirt (sand, earth) color (gray to black), especially at the bottom of the screen. Suppose that

  However, because the entire screen is not completely occupied by grass and dirt, it is evaluated as a percentage of the screen, taking into account the fact that the upper part of the screen is not evaluated and that it appears partially in other parts. .

  As a condition (3) for being a race part, in one race, the audience feels excited just before the goal and cheers are raised. And This method of detecting the sound excitement is similar to the condition (1). However, while the sound power continues for a certain value or more, the power increases and decreases with the voice of the included live voice (one person), but the cheer and the running sound are at the lowest part. For this reason, the lowest acoustic power is evaluated among a certain number of continuous acoustic powers.

  In order to judge and evaluate these three conditions, each condition is extracted as a feature quantity by the feature extraction module. Hereinafter, processing in each module will be described.

  First, acoustic continuous section feature extraction for evaluating the condition (1) will be described. FIG. 16 shows an outline of processing in the acoustic continuous section feature extraction module existing as one of the feature extraction module group 120-n, and FIG. 17 shows an operation flow thereof.

  As an overview, the application features that make the time series 1303 of the acoustic power, which is one of the basic features to be input, become a binary value (applicable is 1 and non-applicable is 0) as to whether the condition is met using parameters. Outputs the quantity 121-n. First, the acoustic threshold θ, the grace period τa, the shortest section length τmin, and the longest section length τmax are acquired as parameter information from the importance calculation formula information storage means 140 (step 1701). Next, the acoustic time t is initialized to 0 (step 1702). Next, an acoustic power value p at the acoustic time t is acquired (step 1704). If p is larger than θ (step 1705), t is substituted for the section start time s and the section end time e (step 1707). The counter c in the section below the threshold is set to 0 (step 1708). These processes are repeated while the time is smaller than the maximum time stored as the sound power basic feature (step 1703). t is increased by 1 (step 1709), and whether or not there is sound power is compared (step 1710), and the sound power p at time t is obtained (step 1711).

  If p is larger than θ (step 1712), t is substituted for the section end time e (step 1713) and the counter c is set to 0 (step 1714). However, if p is smaller than θ, the counter c is 1 (step 1715), if it is larger than the grace period τa (step 1716), if the section length es is between the shortest section length τmin and the longest section length τmax (step 1720) The applied feature value is set to 1 (step 1721), and if not (0) (step 1722), the process of finding a new continuous sound portion from the next sound time is continued (step 1723). When the acoustic time t is larger than the maximum acoustic power time, that is, when the processing for all the acoustic powers is completed (step 1703), the acoustic continuous section feature value as applied feature data is sent to the arithmetic control unit. Then, the process ends (step 1730).

  Note that these processing flows end when the sound continuous section feature value is passed to the calculation control unit 130 when all the sound times are completed, but the sound continuous section feature value is sent to the calculation control unit 130 for each sound time. After passing and finishing the calculation at each time, the processing of this flow may be continued again.

  Next, image color composition feature extraction for evaluating condition (2) will be described. FIG. 18 shows an outline of processing in the color composition feature extraction module existing as one of the feature extraction module group 120-n, and FIG. 19 shows an operation flow thereof. As an overview, the time series 1304 of the image color composition, which is one of the basic features to be input, is the number of evaluation objects, using the mask matrix M to be evaluated, the specified multiple colors C and the number of colors N as parameters. The ratio of the portion where any one of the specified colors exists is output as the applied feature amount 121-n.

  First, the area mask array M [16], the specified number of colors N, and the specified color array C [N] are acquired as parameter information from the importance calculation formula information storage means 140 (step 1901). Next, the image time t is initialized to 0 (step 1902), and the image color at time t is selected from the time series of the image color composition feature 1304 among the image basic feature data in the basic feature storage means 112 created earlier. The composition feature f [16] is acquired (step 1904).

  The counters i, s, and A are initialized to 0 (step 1905), and the area mask array is 1 for all the number of image areas (16 areas) (step 1910) (step 1911). The number A is added (step 1940), and if there is any of the specified number N of the specified color array C [N] (steps 1913, 1914, 1916), 1 is added to the counter s (step 1914) The next area is evaluated (step 1915). When the evaluation is completed for all 16 areas (step 1910), the ratio s / A in which any of the specified colors existed with respect to the total number of the specified areas is obtained, and this is the color in the image at time t. The composition feature value is used as an applied feature (step 1920).

  When processing for all image times is completed (steps 1921 and 1903), the color composition feature value is passed to the arithmetic control unit 130, and the processing is ended (step 1930). Note that these processing flows end when all the image times are completed, passing the color composition feature value to the calculation control unit 130, but pass the color composition feature value to the calculation control unit 130 for each image time, You may make it continue the process of this flow again after finishing the calculation at each time.

  Next, acoustic excitement feature extraction for evaluating the condition (3) will be described. (FIG. 20) is an outline of processing in the sound excitement feature extraction module existing as one of the feature extraction module group 120-n, and (FIG. 21) is an operation flow thereof.

  As an overview, an applied feature value 121-n that outputs a sound climax degree between 0 and 1 using a parameter τ is output as a time series 1303 of acoustic power, which is one of the basic features to be input.

  First, the movement section length τ is acquired as parameter information from the importance calculation formula information storage means 140 (step 2101). Next, while shifting the sound time t from 0 until the sound power reaches the maximum sound power time, Pmin that gives the lowest value among the sound power values in the section from t to t + τ is selected as the value at time t (step 2102).

  This value is between the minimum value and the maximum value of the sound power, but is normalized so as to correspond to 0 to 1, and this is set as the sound rise value at time t. (Step 2103). The acoustic climax value at all acoustic times (to be exact, up to the time of (maximum value−τ)) is passed to the arithmetic control unit 130 and the process is terminated (step 2104).

  Note that these processing flows are all finished when the acoustic climax value is passed to the arithmetic control unit 130 when the acoustic time is completed, but the acoustic climax value is passed to the arithmetic control unit 130 for each acoustic time. After completing the calculation at the time, the processing of this flow may be continued again.

  FIG. 22 shows an example of the importance calculation formula information 141 used by the arithmetic control unit 130 to extract a horse race. The importance level calculation formula information 141 includes a feature extraction module list 2201 and a calculation formula 2202 used in the calculation formula. In the feature extraction module list 2201, a list of parameters necessary for each feature extraction module is given to each. In the calculation formula 2202, formulas are described after specifying specific values of the respective parameters.

  The three feature extraction modules described so far, namely the acoustic continuous section feature extraction module, the color composition feature extraction module, and the acoustic swell feature extraction module, are named modules A, B, and C, respectively. And necessary parameter names are listed.

  In the calculation formula, each applied feature value output from each module is calculated for each time using the four arithmetic operators and constants.

  In the formula, there is a description of “sat”, which indicates internal processing for binarizing the applied feature value using the subsequent numerical value as a threshold value. Specifically, with respect to the application feature data 120-n output from module B, that is, the color composition feature extraction module (FIGS. 18 and 19), 0.6 is a threshold value, 1 is greater than this, and 0 is less than this Will be returned. In this way, importance can be calculated using appropriate parameters and applied feature values for each of the above parameters.

  FIG. 23 shows a calculation result screen 700 that displays the calculation result of importance by the arithmetic control unit 130. This screen is displayed on the display device 151 of the digest creation console 150 when the calculation by the calculation button 413 is completed. The calculation result screen 700 has a time display area 2301, applied feature amount display areas 2302-1 to 2302-3 corresponding to each of a plurality of feature extraction modules, and a result display area 2303 showing the result of calculating the importance from these. Each area shows each applied feature amount at the time indicated by the time display area 2301 and the importance obtained finally. 704 and 705 are scroll buttons used when the display does not fit on the screen. Using this screen, the user confirms whether or not the importance calculation is as intended. If the result is good, the importance calculation formula information 141 is used to create a digest video.

  In the above description, no particular mention was made as to what time unit the importance is calculated. The time unit is calculated for each video frame, the program is calculated by dividing the program into fixed time intervals (for example, 10 seconds), the program is divided into several scenes by some method, and each scene unit is calculated. However, the present invention can be applied to any of these methods.

  As a next embodiment, a case where a game board surface is generated as a digest from a Go / Shogi program using video data will be described. Note that the user interface for specifying the module to be used and the parameter is not referred to here as it conforms to the first embodiment, and is not referred to here. The basic feature extraction units 111-1 to 111-n, the calculation control unit 130, and the feature extraction module The processing and output in 120-1 to 120-n will be described.

  As in the second embodiment, the basic feature storage unit 112 already has the acoustic power feature data 1303, which is the basic acoustic feature data 1301, as shown in FIG. Assume that image color composition information 1304 as feature data 1302 is extracted from an image color composition feature extraction unit (another 111) and input image data 110 and prepared in basic feature storage means. Since this acquisition method is the same as that of the second embodiment, description thereof is omitted.

  In the case of Go / Shogi, it is assumed that the playing board, especially the board after the long review, is important as the digest. During the program, the board has almost no movement, and the board has the most appearance among the pictures with no movement. Since the explanation screen is displayed while the player is thinking, the display interval of the board is almost the time that the player thinks. Therefore, a portion having a long display interval is reproduced as a more important digest portion.

  From the above features, the following three feature extraction modules are defined and combined to determine the importance of Go shogi. That is, the feature extraction modules of condition (1) still image section feature, condition (2) mode image feature, and condition (3) appearance interval feature are defined and combined as described below.

  First, the processing of the still image section feature extraction module under the condition (1) is as follows. FIG. 24 shows an outline of processing in the still image section feature extraction module that exists as one of the feature extraction module group 120-n, and FIG. 25 shows its operation flow.

  As an outline, the time series 1304 of the image color composition which is one of the basic features to be input is a threshold value θstill of the norm (difference distance) of the color composition for determining as a still image, and the minimum time τmin that continues as a still image , And the mask matrix M as a parameter, a still state image is output as 1 and an image that is not a still image is output as an applied feature amount 121-n to be 0.

  In the processing flow, first, θstill, τmin, and matrix M are acquired as parameters (step 2501). The image time t is initialized to 0 (step 2502), and the image color composition information F [16] of the first image frame is acquired from the basic feature information storage means 112 (step 2504).

  T is substituted into the section start time s and the section end time e (step 2505), and the immediately preceding color composition information P [16] is initialized with F [16] (step 2506). Further, the time t is advanced (step 2507), the image color composition information F [16] of the next image frame is acquired (step 2509), and the counters d and i are initialized with 0 (step 2510). Next, i is counted for all 16 areas of the color composition (steps 2511 and 2515), and when mask M [i] is 1 (step 2512), F [i] and P [i] are compared. (Step 2513) If different, d is increased by 1 (Step 2514). If d is equal to or less than θstill per screen (step 2516), t is substituted for the section end time e (step 2517), and the immediately preceding color composition information P [16] is updated to the current F [16] (step 2516). 2518) Proceed to the next time (step 2507).

  If d is larger than θstill per screen (step 2516), it is determined that the still image state has ended, and if the section length es is τmin or more (step 2520), the feature amount of the section from image time s to e is calculated. It is set to 1 (step 2521), otherwise it is set to 0 (step 2522), and the loop is returned to the beginning to find a new still image section (step 2503).

  These are performed for all the image times, and the still image section feature value is passed to the calculation control unit (step 2530), and the process is terminated.

  The processing of the mode image feature extraction module under condition (2) is as follows. FIG. 26 shows an outline of processing in the most frequent image feature extraction module existing as one of the feature extraction module group 120-n, and FIG. 27 shows an operation flow thereof.

  As an overview, the time series 1304 of the image color composition, which is one of the basic features to be input, and the feature value for selecting the image frame to be processed, which is the application feature value 121-n of the other module, are input. , Which is one of the application feature data 121-n and has the most frequently appearing color features, using the internal mask dictionary with the region mask M [16], the norm threshold θdic and the number N of extracted symbols as parameters Are output as the most frequent image feature values.

  In the processing flow, first, M [16], θdic, and N are acquired from the importance calculation formula information storage unit 140 (step 2701). Next, the design dictionary is initialized and the design dictionary entry number DN is initialized to 0 (step 2702).

  Here, the pattern dictionary has one variable: DN: number of entries, Did [DN]: dictionary entry id (so that the original index can be understood at the time of sorting), and DL [DN]: total appearance time of the pattern And two arrays. The image time t is initialized to 0 (step 2703), and the picture dictionary registration process is performed (step 2705) while increasing the time (step 2706). The pattern dictionary registration process performs the following steps. First, selection information 2601 at time t is acquired and set as S (step 2731).

  When S is 1, color composition information f [16] is acquired from the basic feature storage means 112 in order to create a picture dictionary for the picture of the frame image at time t (step 2733). On the other hand, the color composition information g [16] at a time j that is earlier than the current time is acquired, and the mask M is 1 among all the areas of the color composition information, and the colors of the respective areas f [16] and g [16] By obtaining the number of regions d different from each other (steps 2741 to 2746), it is searched whether frames having the same color composition existed in the past (steps 2735, 2747, and 2748).

  If it exists, that is, d is equal to or smaller than θdic (step 2747), Fid [j]: Substitute the dictionary entry id for the pattern at time j into the value k (step 2754), and set the total appearance time of the corresponding pattern dictionary Increase by 1 (step 2755). If it does not exist, a dictionary entry is added (step 2751) and an entry is set (step 2752). The corresponding dictionary entry is indicated in Fid [t] at time t (step 2753). On the other hand, if S is not 1 (step 2732), -1 is set to Fid [t] because it is not the target frame image (steps 2750 and 2753).

  After completing the dictionary for all image times (steps 2704, 2705), sort (Did [DN], DL [DN]) in descending order using DL [DN] as a key (step 2710) . Finally, the most frequent image feature quantity at time t having the same Fid [t] as Did [DN] of the extracted N entries from the first dictionary entry (with larger DL [DN]) is set to 1 ( Step 2717) and the others are set to 0 (Step 2718). Thereafter, the completed mode image feature value 121-n is passed to the arithmetic control unit 130 (step 2720), and the process is terminated.

  Finally, the processing of the appearance interval feature extraction module for condition (3) is as follows. (FIG. 28) is an outline of processing in the appearance interval feature extraction module that exists as one of the feature extraction module group 120-n, and (FIG. 29) is its operation flow. As an outline, an input feature value for selecting an image frame to be processed that is an applied feature value 121-n of another module is input, and a maximum continuous playback time τplay is given as a parameter to obtain an image frame. The application feature data 121-n is output so that the appearance interval feature value increases as the selection end time increases from the end time of the immediately preceding selection time.

  What is the processing flow? First, τplay is acquired from the importance calculation formula information storage means 140 (step 2801). Next, the immediately preceding appearance time tp and image time t are initialized to 0 (steps 2902 and 2903). The selection information of the image frame at the input image time t is set to S (step 2905), and when S is not 1 (step 2906), the applied feature value at the image time t is set to 0 (step 2907), and the next time Proceed to step 2908.

  When S is 1, it is assumed that the processing target is selected (step 2906), and the section start time s and section end time e are initialized to t (step 2910). Next, while advancing the image time t (step 2911), the frame image selection information S is acquired (step 2913). If S = 1 (selection), the section end time e is updated to t (step 2915). ). When S is other than 1, the process goes out of the loop (step 2914). When the section length es is longer than τplay, the applied feature value of the image time s to (e-τplay) is set to 0, and the image time (e-τplay ) To e are applied feature values.

  When the section length e-s is equal to or shorter than τplay, the applied feature value from the section s to e is set to e-tp (step 2923). Then, the previous appearance time tp is updated with e (step 2924). When processing at all image times has been completed (steps 2904 and 2912), if necessary, after the last part has been processed (steps 2920 to 2923), it appears as an applied feature value 121-n The interval feature value is passed to the arithmetic control unit 130 (step 2930), and the process ends.

  FIG. 30 shows an example of the importance calculation formula information 141 used by the arithmetic control unit 130 to extract the game board surface of the Go / Shogi program. The importance calculation formula information 141 includes a feature extraction module list 3001 and a calculation formula 3002 used in the calculation formula. In the feature extraction module list 3001, a list of parameters necessary for each feature extraction module is given to each. In the calculation formula 3002, a specific value of each parameter is specified and the formula is described.

  The three feature extraction modules described so far, that is, the still image section feature extraction module, the mode image feature extraction module, and the appearance interval feature extraction module are named modules A, B, and C, respectively. And the necessary parameter names. The calculation formula indicates an actual value of a parameter to be given to each module, and indicates that the output of A can be configured as an input of B and the output of B can be configured as an input of C.

  The output of the last C is the importance. In this way, importance can be calculated using appropriate parameters and applied feature values for each of the above parameters.

  FIG. 30 shows a calculation result screen 700 that displays the calculation result of importance by the arithmetic control unit 130.

  This screen is displayed on the display device 151 of the digest creation console 150 when the calculation by the calculation button 413 is completed. The calculation result screen 700 includes a time display area 3101, applied feature amount display areas 3102-1 to 3102-3 corresponding to each of a plurality of feature extraction modules, and a result display area 3103 indicating the result of calculating the importance from these. Each area indicates each applied feature amount at the time indicated by the time display area 3101 and the importance obtained finally. 704 and 705 are scroll buttons used when the display does not fit on the screen.

  Using this screen, the user confirms whether or not the importance calculation is as intended. If the result is good, the importance calculation formula information 141 is used to create a digest video.

  In the above description, no particular mention was made as to what time unit the importance is calculated. The time unit is calculated for each video frame, the program is calculated by dividing the program into fixed time intervals (for example, 10 seconds), the program is divided into several scenes by some method, and each scene unit is calculated. However, the present invention can be applied to any of these methods.

  As described above, several feature extraction modules are installed as necessary, and each parameter, connection configuration, and arithmetic operation are given via the importance calculation formula information 140 to reproduce an appropriate digest. Therefore, the importance level can be generated in units of appropriate time and stored in the scene importance level storage means 160. On the basis of this, a portion having a degree of importance or higher designated by the playback position instruction means 161 is selected, and the playback position information 120 is given to the recording / playback control section 183, so that one of the videos recorded in the video recording means 184 is obtained. It is possible to perform digest reproduction for reproducing a part.

  The virtual device configuration method according to the present invention has been described mainly using video and audio content. However, the configuration using a functional module that processes general-purpose data can be used, and any configuration equipped with a communication unit can be used. It can also be applied to industrial equipment. Therefore, it may be used for many industries.

1 is a block diagram showing an overall digest video creation method and apparatus according to the present invention. The figure which showed the usage of the physical feature memory | storage means 112 Flow chart showing an embodiment of the feature extraction module 120-n The figure which shows the Example of the calculation control screen 400 by the calculation control part 130 The figure which shows the Example of the module display part 401-n in the calculation control screen 400 The figure which shows the structural example of the importance calculation formula information 141 The figure which shows the Example of the calculation result screen 700 by the arithmetic control part 130 The figure which shows the Example of the change scene confirmation screen 800 by the arithmetic control part 130 The figure which shows the Example of the reproduction | regeneration point list | wrist 900 The figure which shows the Example of importance calculation formula information reading screen 1000 The figure which shows the Example of the recording / reproducing operation screen 1100 The figure which shows the Example of the basic feature extraction part 111-n (caption part-of-speech feature extraction) The figure which shows the Example of the basic feature extraction part 111-n (acoustic power feature, image color composition feature) Flow chart of embodiment of basic feature extraction unit 111-n (acoustic power feature) Flowchart of embodiment of basic feature extraction unit 111-n (image color composition feature) Processing outline diagram of the embodiment of the feature extraction module 120-n (acoustic continuous section feature extraction module) Flow chart of embodiment of feature extraction module 120-n (acoustic continuous section feature extraction module) Processing outline diagram of the embodiment of the feature extraction module 120-n (color composition feature extraction module) Flow chart of embodiment of feature extraction module 120-n (color composition feature extraction module) Processing outline diagram of the embodiment of the feature extraction module 120-n (sound swell feature extraction module) Flow chart of embodiment of feature extraction module 120-n (acoustic swell feature extraction module) The figure which shows the structural example of the importance calculation formula information 141 (horse racing) The figure which shows the Example of the calculation result screen 700 by the arithmetic control part 130 (horse racing) Processing outline diagram of embodiment of feature extraction module 120-n (still image section feature extraction module) Flow chart of embodiment of feature extraction module 120-n (still image section feature extraction module) Processing outline diagram of embodiment of feature extraction module 120-n (mode image feature extraction module) Flow chart of embodiment of feature extraction module 120-n (mode image feature extraction module) Process outline diagram of embodiment of feature extraction module 120-n (appearance interval feature extraction module) Flow chart of embodiment of feature extraction module 120-n (appearance interval feature extraction module) The figure which shows the structural example of the importance calculation formula information 141 (Go and Shogi) The figure which shows the Example of the calculation result screen 700 by the arithmetic control part 130 (Go and Shogi)

Explanation of symbols

121-1 to 121-n Applied features, 400 Computation control screen, 401-1 to 401-n Module display, 700 Calculation result screen, 800 Change scene confirmation screen, 900 Playback point list, 1000 Importance information of importance calculation formula screen,
1101 Recording / playback operation screen

Claims (9)

An apparatus for creating a digest for the video data using the video data and the feature amount extracted from the data,
Video storage means for storing the video data;
Basic feature storage means for storing feature amounts calculated from the video data using a physical principle or a symbol processing principle;
Using the contents of the video storage means and the basic feature storage means as inputs,
A plurality of feature extraction modules for calculating applied features corresponding to the cognitive sensation of the video viewer;
Performing an operation between applied feature amounts calculated from the plurality of feature extraction modules;
Computation control unit that calculates the importance of the video data,
A digest creation device that determines the content of a digest video based on the importance. The digest creation device according to claim 1,
The digest extraction apparatus characterized in that the feature extraction module can be added from the outside in accordance with an operator's instruction. The digest creation device according to claim 1,
The feature extraction module has a means for changing a feature calculation parameter used for calculating the applied feature amount according to a degree of preference of a viewer. The digest creation device according to claim 1,
The calculation control unit calculates the importance by applying an importance calculation formula of a predetermined format input from the outside between a plurality of applied feature amounts obtained from the plurality of feature extraction modules. A digest creation device. The digest creation device according to claim 4,
The importance calculation formula includes a description that specifies the plurality of feature extraction modules that the importance calculation formula requires for calculating the importance, and a description that specifies feature calculation parameters used in the plurality of feature extraction modules. A digest creation device including a description of a calculation formula for calculating the importance using applied feature amounts obtained from a plurality of feature extraction modules. The digest creation device according to claim 5,
The digest creation apparatus, wherein the importance calculation formula is interactively input by an operator. The digest creation device according to claim 5,
The digest creation apparatus is characterized in that the importance calculation formula is input from the outside through communication means. In the digest preparation apparatus of Claim 4,
A digest creation apparatus comprising a user interface for displaying the plurality of feature extraction modules and the importance calculation formula in association with each other. The digest creation device according to claim 8,
When the user changes the feature extraction module, or when the user changes the value of the feature calculation parameter given to the feature extraction module, it has a user interface that displays a difference between digest videos created before and after the change. A digest creation device.

Images