JP2004363831A - Apparatus and method for moving picture processing - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem that when a moving picture is divided by using its attached information, it has not been possible to select the attached information used for division. <P>SOLUTION: Attached information which is usable for division is made selectable and relations between division results based upon selected attached information and the attached information used for the division are displayed in two dimensional array. A user while adding, changing, or deleting division conditions through a user interface of the two-dimensional array display is able to determine final division conditions. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a technique for dividing a moving image by using a change point of supplementary information of the moving image.
[0002]
[Prior art]
Generally, as seen in Patent Documents 1 and 2,
[0003]
-Operation of the video imaging device (pan, tilt, zoom, etc.)
・ Imaging device status (focus, white balance, etc.)
-Text data given by the user (title, person name, place name, etc.)
・ Features obtained by analyzing images (color, brightness, etc.)
Techniques have been developed for automatically dividing a moving image at a change point of metadata or a point at which the metadata satisfies a predetermined condition by using (hereinafter, these supplementary information is collectively referred to as metadata). By automatically dividing a moving image, playback from a desired location, editing of a moving image, and automatic summarization of the moving image are enabled.
The concept of automatic division of a moving image based on metadata will be described with reference to FIG.
[0004]
FIG. 1A shows, as an example of a metadata item for a certain moving image, a date, a zoom, a color distribution in a frame, and a place name, which are grouped in section units. The date is divided by the change point of the date, and the section before and after is set. Zoom is a section for each operation. The color distribution is obtained by calculating the color distribution in units of frames, dividing the color distribution at points that greatly change between frames, and setting the preceding and subsequent sections as sections. The section name provided by the user is the section name as it is. As described above, the metadata is grouped into sections based on the change points of the metadata, the sections to be added, and the like.
[0005]
FIG. 1B shows a result of dividing a moving image using the metadata, date, Zoom, color distribution, and place name division points (change points) arranged in units of sections. As shown in FIG. 1, for a plurality of types of metadata items, a result obtained by dividing a moving image when a segmentation point appears in any one of the metadata items is a division result. Thumb-A to Thumb-J in FIG. 1B are thumbnail images formed by reducing the first and last image frames of the moving image and the image frame at each division point.
[0006]
The division result is displayed by, for example, a GUI (Graphical User Interface) as shown in FIG. In FIG. 15, reference numeral 5000 denotes a division result window. Title bar 5001 informs the user that it is a “split result window”. Numeral 5002 indicates that this column is a division point. Reference numeral 5003 denotes a thumbnail corresponding to a division point. Time is displayed as going from top to bottom. Note that Thumb-A is a thumbnail corresponding to the start point of the moving image, and Thumb-J is a thumbnail corresponding to the end point of the moving image. Reference numeral 5004 indicates that this column has a division length. Numeral 5005 indicates the division length in time for each division. Note that Thumb-J is a thumbnail at the end point of the moving image, and has no division length, and thus is displayed as “−−−−−−−−”.
[0007]
[Patent Document 1]
JP-A-08-163488
[Patent Document 2]
Japanese Patent No. 3192666
[0008]
[Problems to be solved by the invention]
In the above-described conventional technique, a moving image can be automatically divided using the segment points of various metadata items. For example, in the example shown in FIG. 1, a moving image is divided at any of the four metadata items. Automatic division has not been realized by using a dividing point for an arbitrary metadata item. That is, since the metadata item used for the automatic division cannot be selected and is fixed, it is very difficult to divide it into a desired section.
[0009]
Here, the importance of selecting a metadata item will be described.
First, the importance of a metadata item used for automatic division may change depending on the content of a moving image. As an example, a case where a place name and a person name are given as metadata items will be described. First, consider a case where a wedding reception is photographed. Basically, we are shooting events that take place in the reception hall, so it does not make much sense to change the place name. On the other hand, various people, such as grooms, brides, friends, relatives, and company bosses, can be photographed. Therefore, it is expected that an appropriate section can be obtained by dividing by a person's name. Next, consider the case of shooting on a family trip. Since many sightseeing spots are visited, sections organized by place names are important. On the other hand, the photographed person is usually limited to family members, and therefore, only the name of the person is equal to the number of family members. Therefore, it is expected that an appropriate section can be obtained by dividing by a place name, rather than by a person name.
[0010]
Also, depending on the intention at the time of editing, the importance of the metadata item used for automatic division may change even if the same moving image is targeted. For example, in the case of the above-mentioned family trip shooting, moving pictures are usually grouped for each sightseeing spot (place) as described above, but if you want to make a moving picture only for a specific family member (for example, a child), You need to split by name.
[0011]
In addition, important metadata is different depending on a photographer's difference, even for a moving image having the same content. For example, some photographers make heavy use of the zoom operation, and some photographers make heavy use of the pan operation. In each case, the importance of zoom and pan differs.
[0012]
Therefore, when a user who edits a moving image knows the contents of a target moving image well (for example, when the user himself shoots), it is important for the user to select a metadata item. However, in the conventional method, this means is not provided, and the intended division result cannot be obtained even if the automatic division is performed, so that time and labor such as combining and further dividing occur.
[0013]
Further, even when the contents of the moving image are not known, the same problem occurs. In this case, in order to confirm the importance of the metadata item, a means for listing the division points corresponding to the respective metadata is further required. However, it is a matter of course that the conventional method does not provide such means, and in such a case, it is even more difficult to obtain an appropriate division result.
[0014]
The present invention has been made in view of the above problems, and provides a means for appropriately selecting a metadata item to be used for automatic division so that reproduction and editing operations can be easily performed. Aim.
[0015]
[Means for Solving the Problems]
In order to achieve the above object, a moving image processing apparatus according to the present invention includes an additional information collecting unit that obtains additional information regarding a moving image, a display unit that presents a user interface capable of selecting additional information, And division information generating means for generating division information for dividing a moving image into a plurality of sections based on selected supplementary information selected via a user interface.
[0016]
Further, an object of the present invention is to provide an additional information collecting step of acquiring additional information on a moving image, a display step of presenting a user interface capable of selecting additional information, and a step of selecting additional information through the user interface. A division information generating step of generating division information for dividing a moving image into a plurality of sections based on the selected incidental information.
[0017]
Furthermore, the object of the present invention is also achieved by a computer program that causes a computer device to function as the moving image processing device of the present invention, and a computer-readable recording medium that stores the computer program.
[0018]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. In this embodiment, a description will be given of a process of generating division information for automatic division using metadata and creating a moving image playlist using the result. This is an example of the processing to be performed, and it goes without saying that the division information can be used for other processing.
[0019]
FIG. 2 is a block diagram illustrating a control configuration of the moving image processing device according to the present embodiment. 2, a CPU 201 executes various controls in the moving image management device of the present embodiment. The ROM 202 stores a boot program executed when the apparatus is started and various data. The RAM 203 stores a control program to be processed by the CPU 201 and provides a work area when the CPU 201 executes various controls. The keyboard 204 and the mouse 205 provide various input operation environments for the user.
[0020]
The external storage device 206 includes, for example, at least one of a hard disk, a floppy disk, an optical disk, a magnetic disk, a magneto-optical disk, a magnetic tape, and a nonvolatile semiconductor memory. The display 207 includes a display or the like, and displays a processing result of the moving image management device, a GUI screen, and the like to a user. A network interface (NIC) 208 is a communication interface with an external device such as a USB, IEEE 1394, Ethernet (registered trademark), or modem, and enables communication with another device that is locally or network-connected. Further, the bus 211 connects the respective blocks constituting the moving image management apparatus.
[0021]
In the configuration of FIG. 2, the external storage device 206 does not need to be built in the moving image management device, and may be connected so as to be accessible via a network interface.
[0022]
FIG. 3 is a block diagram illustrating a functional configuration of the moving image processing device according to the present embodiment. The storage unit 300 is configured by the external storage device 206 and stores moving images, metadata, division information, and a moving image playback list. The moving image is captured by another imaging device such as a video camera or a digital camera, and a device capable of reading a recording medium on which the captured image is recorded, for example, a removable recording medium such as a semiconductor memory or an optical disk is used as the external storage device 206. It may be used or read from the NIC 208 into the storage unit 300 from another device via a network.
[0023]
Metadata that has been given in advance by another device can be captured in the same manner as a moving image. Note that the metadata at this time is stored in a format described later. Further, the metadata can be provided by the main moving image processing apparatus. The division information and the moving image reproduction list are generated and stored in the main moving image processing apparatus.
[0024]
The division information determination unit 301 generates division information based on the metadata read from the storage unit 300 and the metadata obtained from the metadata extraction unit 304, and determines a desired division. The user operation unit 302 acquires an instruction or input from a user via the keyboard 204, the mouse 205, or the like. The display unit 303 includes a display unit 207, and displays a division result or the like generated by the division information determination unit 301 to a user. The metadata extraction unit 304 analyzes moving images and metadata stored in the storage unit 300 and extracts new metadata. The metadata extraction unit 304 is activated when necessary according to the operation of the user operation unit 302. You. The reproduction order specification unit 305 specifies the reproduction order of the divided sections determined by the division information determination unit 301 by specifying the order. The reproduction unit 306 reads and reproduces a moving image from the storage unit 300 in accordance with the reproduction at the time of confirmation until the division information is determined by the division information determination unit 301 and the reproduction order specified by the reproduction order specification unit 305. , On the display unit 303.
[0025]
There are three main methods for adding metadata to a moving image. In this embodiment, any method may be used. Also, it may be provided by another method.
[0026]
(Method 1) Method to be applied to an imaging device that captures a moving image
Information such as the operation of the imaging device and the state of the imaging device during shooting can be obtained and provided from operation buttons such as a zoom button of the imaging device, and a clock and various sensors included in the imaging device. Items include shooting date and time, zoom, focus, white balance, gain, shutter speed, pan, tilt, AE mode, and the like. Further, information obtained from a device external to the imaging device (for example, coordinate information obtained from a GPS sensor) may be added.
[0027]
(Method 2) Method to be given by the user
After the shooting, the user can designate the section in the moving image while playing back the moving image, and input the information on the name, location, and the like of the person being shot, thereby giving the information.
[0028]
(Method 3) A method of analyzing and adding moving images
The captured moving image can be analyzed to extract a feature amount, and this can be added as metadata. Items include color distribution, luminance, presence of a face area, and the like in a moving image frame.
[0029]
For example, as for the color distribution, each frame is divided into a plurality of blocks vertically and horizontally as shown in FIG. 16, and the average value of each of the RGB channels is calculated for each block. Metadata of the distribution. Further, the sum of squares of the difference between each of the RGB channels between the corresponding blocks of the previous frame and the current frame is obtained, and this is set as the inter-frame similarity distance. An example of the calculation formula is shown below.
[0030]
Σ (i = 1 to k) ｛(P1 _iR -P2 _iR ) ² + (P1 _iG -P2 _iG ) ² + (P1 _iB -P2 _iB ) ² ｝ Equation 1
However,
i: Indicates the block being processed
K: number of divided blocks
P1 _iR : Average value of the R channel of the i-th block of the previous frame
P1 _iG : Average value of G channel of the i-th block of the previous frame
P1 _iB : Average value of the B channel of the ith block of the previous frame
P2 _iR : Average value of the R channel of the i-th block of the current frame
P2 _iG : Average value of G channel of i-th block of current frame
P2 _iB : Average value of B channel of the i-th block of the current frame
[0031]
The smaller the inter-frame similarity distance obtained by Equation 1, the more similar the color distribution of the two frames, and the larger the distance, the less similar. Then, for example, a frame having an inter-frame similarity distance equal to or greater than a certain threshold, that is, a frame having low similarity to the previous frame is regarded as a change point. A section before and after the change point is defined as a section, and this is defined as metadata in section units.
[0032]
Similarly, the luminance of each frame is divided into blocks as shown in FIG. 16, an average luminance in the block is obtained, and this feature amount can be used as metadata in frame units.
[0033]
For the presence of the face region, the color components near the center of the frame are analyzed, and when the skin color component is larger than a predetermined threshold, the face is estimated to be present, and the first frame and the last frame that are estimated to be present are changed. A method of adding metadata as a point can be cited. Another method may be used, such as detecting the presence of a face by template matching or the like.
[0034]
In the present embodiment, it is assumed that only metadata is collected into one file and stored in the storage unit 300.
[0035]
FIG. 11 is a diagram illustrating an example of a description of metadata stored in the storage unit 300. In the example shown in FIG. 11, the metadata is described in an XML (extensive Markup Language) format. Reference numeral 1401 denotes a header portion relating to the XML document, in which an XML declaration such as an XML version and a character encoding method is described. Reference numeral 1402 denotes a document body. The document body is formed between a start tag such as "<" and ">", such as 1403, and a "<//" and ">", such as 1404, between the start tag and the corresponding end tag. It is composed of elements composed of contents 1405 sandwiched.
[0036]
The start tag and the end tag indicate what content is described between the tags. Each element can be described hierarchically. In this example, “metadata” exists under the “metadata set” that is the root element (the highest level), and “target video”, “section”, and (In this example, “Zoom”), a “path” exists under “target video”, and a “start point” and a “length” exist under “section”. Under “Zoom”, “direction”, “start magnification”, and “end magnification” exist. As described above, the “metadata” is listed and described under the “metadata set”.
[0037]
Then, first, the position of the moving image file targeted for each “metadata” is described. This has “path” as an element of “target moving image”, and describes, for example, the file path of the target moving image file stored in the storage unit 300. Next, a “section” of each “metadata” is described, which is described by a “start point” and a “length”. The “start point” is described by a time code, and the “length” is described by a time from a start point to an end point. At this time, if it is desired to describe metadata on a frame basis, 0 is described in “length”. The specific content of the metadata corresponding to “section” is described next, and in this example, “Zoom” is described. “Zoom” has elements of “direction”, “start magnification”, and “end magnification”, and “direction” is described as in when zooming in and out when zooming out. “Start magnification” is the zoom magnification at the start of the operation, and in this example, 1 indicating 1 × is described. “End magnification” is the zoom magnification at the end of the operation, and is 2.5 times in this example.
[0038]
FIGS. 4A to 4C are flowcharts illustrating a moving image division processing procedure of the moving image processing apparatus according to the present embodiment. The moving image dividing process is executed, for example, when a user selects a moving image to be divided from the user operation unit 302 and instructs the dividing process.
[0039]
First, in step S401, the division information determination unit 301 refers to the storage unit 300, checks metadata items assigned to the target moving image from the metadata file, and generates a list of metadata items that can be used for automatic division. create. At this time, metadata items that do not exist in the metadata file and can be newly added by the moving image processing device are also included in the list.
[0040]
Next, in step S402, a list of metadata items created in step S401 and usable for automatic division is displayed on the display unit 303. Next, in step S403, it is determined whether or not the user has completed selection of the metadata item to be used via the user operation unit 302. Keep waiting until done.
[0041]
Here, an example of a user interface displayed to display and select available metadata items in step S402 will be described with reference to FIG. Reference numeral 500 denotes an available metadata selection window. The title bar 501 informs the user that the screen being displayed is an “available metadata selection window”. In the metadata item display area 502, a list of available (selectable) metadata items is displayed.
[0042]
In the present embodiment, what is underlined in the metadata item display area is the already selected metadata. The selection of an item can be performed by an arbitrary method usually employed in a GUI. For example, the pointer 504 is operated by the mouse 205, and a desired item is indicated by the pointer 504 and clicked to perform a selection operation. Can be. Further, in order to specify the order of display in the division determination window described later, the display order may be changed by dragging an item or the like. Note that a plurality of items can be specified as illustrated in the figure. Also, what is displayed in italics indicates metadata items that can be added by the main moving image processing apparatus. As a result, metadata can be extracted for items in this list even if they are not assigned in advance. The OK button 503 can be operated by clicking with the mouse 205 or pressing the corresponding key of the keyboard 204, and the selection of the available metadata item is completed by operating the OK button 503. The pointer 504 is, for example, a pointer of the mouse 205, and the user can give various instructions to the moving image processing apparatus by operating the pointer.
[0043]
When the metadata item to be used for division is set by operating the OK button 503 (YES in step S403), the selected items include items that need to be extracted (the above-described items displayed in italics). It is checked whether it is (Step S404). If not, skip steps S405 and S406 and proceed to step S407. If there is, in step S405, the moving image is analyzed by the metadata extracting unit 304, and the corresponding metadata item is extracted. For the extraction, for example, the method described as the above-described method 3 can be used. Then, in step S406, it is added to the metadata file of the storage unit 300 in a predetermined format (in the present embodiment, the XML format described above). This is to make the extracted metadata usable by other devices.
[0044]
Next, in step S407, the division information determination unit 301 reads the metadata read from the storage unit 300 and the metadata extracted by the metadata extraction unit 304 (however, this exists only when YES in step S404). And generate division information.
[0045]
Here, the division information will be described.
In the present embodiment, the division information is assumed to be one file and stored in the storage unit 300. In storing, it is preferable to attach a special extension or prepare a special directory so that the information can be identified as division information. The division information is a work file for creating a moving image playlist. Therefore, for example, when dubbing a moving image file for distribution to another user, the file does not need to be copied. In such a case, it is possible to avoid copying only the file of the division information by judging the extension or the directory.
[0046]
FIG. 12 is a diagram illustrating an example of the description of the division information stored in the storage unit 300. In the present embodiment, the division information is described in an XML format, similarly to the metadata shown in FIG. Since the basic description of the data structure is the same as that of FIG. 11, the description is omitted here and each element is described. The “divided information set” includes an element called “divided information”, which is information for each divided section, and is described by arranging a number of pieces of divided information as illustrated. The “split information” includes elements such as “target image”, “section”, “thumbnail”, and “item row”. The “target moving image” is information on a moving image that is a target of the division information. The description of the contents is omitted because it is the same as that of the metadata in FIG.
[0047]
“Section” is information on a divided section. The description of the contents is omitted because it is the same as that of the metadata in FIG. “Thumbnail” is information on a thumbnail representing a section. Like the “target moving image”, the file path has an element “path”, and describes a file path in which a thumbnail file generated in the process of step S408 described below is stored. The “item row” includes an element “in-section item information” which is information corresponding to each item in the section. The “item information in section” exists for the number of items, and is described in the item line in the order of display in the division determination window FIG. 6 (described later).
[0048]
In the “item information within section”, “item name”, “grant target”, “division criterion”, and elements corresponding to the contents of the metadata can be described as necessary. The “item name” describes a metadata item name. If the “grant target” is TRUE, it indicates that the metadata in the “item name” exists in the section, and if it is FALSE, it indicates that it does not exist. If the "division criterion" is TRUE, it is a metadata item that has become the division criterion of the section, and if it is FASLE, it indicates that the division criterion is not. In this example, in the element corresponding to the content of the metadata, the “item name” is the place name, and the “place name” which is the metadata element is copied from the metadata file. At this time, even if the metadata file includes “country”, “prefecture”, “city”, “town”, “chome”, “street number”, etc. as elements of “place name”, FIG. Only the elements to be displayed on the division determination window (described later) need to be copied. In this example, only "city" is copied.
[0049]
Here, the details of the division information generation processing performed in step S407 in FIG. 4A will be described with reference to the flowchart in FIG.
First, in step S1601, if any of the target metadata items is a frame unit (the “length” of the “section” is set to 0), it is determined as the section unit. For example, when the above-described color distribution is provided in units of frames, the distance between the feature amounts is obtained by the above-described method, and a point where the distance equal to or more than a predetermined value is a change point, and the preceding and following points are defined as a section. I do.
[0050]
Next, in step S1602, the division section for each item is determined based on a predetermined condition with reference to the metadata of the section unit. Some examples of the predetermined condition are given below. For example, in the case of the date described above, the section to which the metadata is added is used as a date division section as it is. For example, in the case of Zoom, if only the zoom-in section is set, the “direction” is referred to and only the IN section is adopted. Further, for example, in the case of Zoom, if there is a condition that an operation end point is set as a change point, there is a case where a section before and after the operation end point is set as a section.
[0051]
Next, in step S1603, the logical sum of the divided sections for each item is calculated, and this is set as the final divided section (that is, the division is performed each time a change point (section point) appears in any item). Next, in step S1604, attention is first paid to the leading divided section. Next, in step S1605, attention is paid to the first item in the order of display on the division determination window (described later) in FIG. The display order on the division determination window (described later) may be a predetermined order or an order specified in the available metadata selection window (FIG. 5).
[0052]
Next, in step S1606, it is determined whether or not the metadata item of interest is added in the division section of interest. This is done by referring to the “section”, which is an element in each “metadata” in FIG. 11, and a tag (“Zoom”) indicating the content of the metadata using the time code of the divided section of interest. Can be determined. If so, in step S1607, “grant target” in “division information” is set to TRUE. If it has not been assigned, it is set to FALSE in step S1608. Next, in step S1609, it is determined whether the item is a reference for division. This is based on whether the “start point” of the “section” in the “metadata” matches the “start point” of the “section” in the “split information” when the “grant target” is TRUE. Can be determined. If they match, in step S1610 the “division criterion” is set to TRUE. If they do not coincide with each other and, of course, if the section is not a target section in the first place, the “division reference” is set to FALSE in step S1611.
[0053]
Next, in step S1612, it is checked whether there is any metadata content that can be displayed. This can be realized by previously determining the contents of the metadata items that can be displayed according to the capabilities of the processing device and checking whether the metadata items correspond to the metadata items. In the example of FIG. 12, only "city" can be displayed for "place name". If there is one that can be displayed, in step S1613, only the corresponding element (“city” in the example) is copied from the “metadata” to the division information.
[0054]
Next, in step S1614, attention is paid to the next metadata item, and if there is the next metadata item, steps S1606 to S1613 are repeated. If the last metadata item to be displayed has been reached, the process advances to step S1615. In step S1615, attention is paid to the next divided section. If the next divided section exists, steps S1605 to S1614 are repeated. If not, the division information has been created, and this processing ends.
[0055]
Returning to FIG. 4A, in step S408, the division information determination unit 301 creates a thumbnail. Referring to the storage unit 300, a frame corresponding to the “start point” of the “split information” is extracted from the moving image, and reduced to create a thumbnail. The created thumbnail is stored in the storage unit 300 as a separate file in a format such as JPEG or bitmap, and the file path is stored in the division information as described above.
[0056]
Next, in step S409, the division result is displayed on the display unit 303 in a two-dimensional array of the time axis and the metadata item axis. Next, in step S410, the division information created on the RAM 203 is stored in the storage unit 300 by, for example, adding a dedicated extension as described above. This is to allow other devices to use the division information and to return the division determination window of FIG. 6 to the initial state as described later. Then, by the processes of steps S411 to S427 shown in FIG. 4B, the division point is determined according to the user operation via the user operation unit 302.
[0057]
Here, a division determination window, which is an example of a user interface for displaying a division result and determining a division point in step S409, will be described with reference to FIG.
[0058]
Reference numeral 600 denotes a division determination window. The title bar 601 informs the user that the window being displayed is a “division determination window”. “Thumb-A” or the like as indicated by 602 is a thumbnail of the frame of the division point (the top frame of the division section). Such a thumbnail is arranged and displayed at a corresponding section when the vertical axis is divided into sections (time progresses from top to bottom) and when the horizontal axis is a metadata item (hereinafter, the horizontal axis direction) A group is called a section row, and a group in the vertical axis direction is called an item column.)
[0059]
From the date of the line 603 to the place name, the name of the metadata item corresponding to each item column is displayed. Thumb-A 602 is displayed at this position because it is located at the first frame of the moving image and is delimited by the change point of the date item. When the head of the section is divided by a plurality of metadata items, a plurality of thumbnails are arranged on the section line. For example, since the first frame of a moving image is also divided by color items, Thumb-A is also displayed in color items. The same applies to Thumb-J.
[0060]
A blank area such as 604 indicates that the metadata of the item corresponding to the divided section exists but is not the division criterion. The hatched portion such as 605 indicates that the metadata of the corresponding item is not added in the corresponding divided section. At the position where the thumbnails are arranged in this manner, the time position of the divided section and the metadata item that is the basis of the division can be grasped, and the section at the time of the division after the item can also be grasped. It will be easier.
[0061]
Items that can be easily displayed (text data and the like) such as dates and place names are displayed as overlays on the thumbnails like 606. As a result, the division criterion can be grasped in more detail. Reference numeral 607 indicates that this column has a division length. Reference numeral 608 indicates the length of the division of the corresponding time line in time. This can be any type as long as the length of the division length can be grasped.For example, even for the time code of the start point and the end point, an icon corresponding to the length is created and displayed. It does not matter. By performing the display as described above, the user can easily grasp the range in the moving image to be divided and the metadata items necessary for determining the division.
[0062]
The buttons 609 arranged in the vertical direction are section selection buttons for instructing selection of each section row. This indicates that the section of the button that is inverted to black is selected. Buttons 610 arranged in the horizontal direction are item selection buttons for instructing selection of each item column. In the item selection button, the metadata item serving as the division reference in the section selected by the section selection button 609 is displayed in black and inverted. Conversely, the section divided by the item selected by the item selection button 610 may be indicated by the reverse display of the button 609. For example, the display state of the other button can be changed according to which of the section selection button 609 and the item selection button 610 is pressed first.
[0063]
The delete button 612 is a button for instructing deletion of the section row or the item column selected by the section selection button 609 and the item selection button 610. As a result, it is possible to delete a section line that is not a division range or delete an item line of a metadata item that is a source of unnecessary division. The OK button 612 is a button for instructing to determine the section selected by the section selection button 609 and the item selection button 610 as the final division. The reset button 613 is a button for returning the state in which deletion or the like has been performed to the initial state when it is desired to start over. The pointer 614 is a pointer for performing a GUI operation, and can normally give various instructions to the apparatus by operating the mouse 205.
[0064]
In addition, a thumbnail such as 602 is indicated by the pointer 614, and by performing an operation such as double-clicking the mouse 205, the corresponding section can be reproduced and confirmed. At this time, a reproduction window illustrated in FIG. 7 is separately opened, and the reproduction is performed on this screen. In FIG. 7, reference numeral 700 denotes a reproduction window. Title bar 701 informs the user that this window is a “playback window”. The moving image reproduction area 702 is an area where the reproduced moving image is displayed.
[0065]
By operating the buttons and the like in this manner, the user can determine the division while performing trial and error.
The display of the GUI, the detection of the user operation, and various processes according to the operation are performed by the division information determination unit 301. In practice, this is realized by the CPU 201 performing appropriate processing while detecting operations of the keyboard 204 and the mouse 205.
[0066]
Next, moving to FIG. 4B, in step S411, it is checked whether or not there is a metadata extraction designation. The metadata extraction designation designates a metadata item for performing more detailed division for a specific section, in addition to the designation in the available metadata selection window described with reference to FIG.
[0067]
For the metadata extraction designation, the range is designated by the section selection button 609 in FIG. 6, and, for example, by clicking the right button of the mouse 205 on the highlighted section selection button 609 (black display), the extractable metadata can be extracted. This is made possible by displaying a selection window and selecting one of them.
[0068]
Here, an example of a user interface when selecting extractable metadata will be described with reference to FIG. Reference numeral 800 denotes an extractable metadata selection window. Title bar 801 informs the user that it is an “extractable metadata selection window”. An area 802 displays a list of metadata items that can be extracted in a specified section. Specifically, the metadata item existing in the designated section is acquired by referring to the metadata file, and the items not displayed in the division determination window are displayed. Among them, the underlined items are the selected metadata. An OK button 803 completes selection of a metadata item that can be extracted by pressing it. The pointer 803 can give various instructions by operating this.
[0069]
By operating the extractable metadata selection window 800, as a result of confirming the division result, it is possible to further divide the data into other metadata items. Further, when the metadata is only partially provided, the metadata can be extracted and divided for a section (such as 605) to which the metadata is not provided.
[0070]
If there is no metadata extraction instruction in step S411, the process proceeds to step S412, and it is checked whether division conditions have been set. The designation of the division condition can be executed when it is desired to perform the division again by changing the condition used in step S407 for a metadata item to be divided by a predetermined threshold, such as division by color. For example, when the user clicks the right button of the mouse 205 with the pointer 614 indicating the area 603 in which the name of the metadata item whose condition is desired to be specified is displayed, a split condition setting window is displayed, and the setting in this window is performed. , The condition can be set. It should be noted that the section to be divided may be designated by the section selection button 609.
[0071]
Here, an example of a user interface for setting the division condition will be described with reference to FIG. Reference numeral 900 denotes a division condition setting window. The title bar 901 informs the user that it is a “division condition setting window”. An area 902 displays the names and the like of metadata items for which conditions are set. In the setting 903 of the threshold value, conditions such as a threshold value are set by operating the keyboard 204 included in the user operation unit 302. Pressing the OK button 904 completes the condition setting. The pointer 905 can give various instructions by operating this. Conditions that can be set include conditions for extracting metadata, conditions for converting frame-based metadata to section-based metadata, and conditions for employing a divided section based on section-based metadata.
[0072]
By operating the division condition setting window 900, as a result of checking the division result, if the division result of the specific metadata item is not as expected, it is possible to change the condition relating to division and perform division again.
[0073]
Next, if there is a designation in step S411 or if conditions are set in step S412, the process proceeds to step S413 to check the applicable range. This is performed by detecting the section selected by the section selection button 609 and reading the corresponding section from the division information.
[0074]
Next, in step S414, within the applicable range confirmed in step S413, metadata is extracted for the metadata item selected in step S411 or the metadata item set as a condition in step S412. Of course, if the condition set in step S412 relates to metadata extraction, metadata extraction is performed based on the set condition. For example, when only the section “IN” is set as a condition in the Zoom item, metadata that meets the condition is extracted. Next, in step S415, generation of division information is performed in the same manner as in step S407 based on the extracted metadata. Of course, if the condition set in step S412 is related to generation of division information, generation of division information is performed based on the set conditions in the same manner as in step S407.
[0075]
Next, in step S417, a thumbnail of the generated division information is created in the same manner as in step S408. Next, in step S417, the generated division information is added, and a two-dimensional array display is performed as in step S409. Next, in step S418, the division information to which the generated information has been added is stored in the storage unit 300. The reason is the same as in step S410.
[0076]
Next, when the condition has not been set in step S412, or when the processing in step S418 has been completed, the process proceeds to step S419, and it is checked whether or not a deletion instruction has been issued with the item selection button 610. If there is no instruction, steps S420 and S421 are skipped. If there is an instruction, the process proceeds to step S420, where the deletion range is confirmed.
[0077]
Here, if only the item selection button 610 is selected, the entire selected item line is set as a deletion range. If only the section selection button 609 has been selected, the entire selected item line is set as a deletion range. For example, when both buttons are highlighted as shown in FIG. 6, it is possible to cancel the highlighted display by clicking the highlighted button in the row or column not to be deleted. is there.
[0078]
Next, the process proceeds to step S421 to delete the deletion determination window 600 for the deletion range. Then, it is deleted from the division information on the RAM 203. Next, the process proceeds to step S422, where the division length changed by the deletion is recalculated and displayed.
[0079]
Next, in step S423, it is checked whether a thumbnail such as 602 has been double-clicked and a reproduction instruction has been given. If the reproduction instruction has not been issued, the next step S424 is skipped. If a reproduction instruction has been given, the process proceeds to step S424, and the section corresponding to the specified thumbnail is reproduced in the reproduction window 700. Thereby, the contents of the section can be confirmed.
[0080]
Next, in step 425, it is determined whether or not a reset instruction has been issued by the reset button 613. If so, in step 427, the division information is read from the storage unit 300 and redisplayed. Thereby, the operation of the division determination can be redone from the beginning.
[0081]
Next, in step 423, it is determined whether the OK button 612 is pressed and the section selected by the section selection button 609 or the item selection button 610 is determined as the last division. If the OK button 612 has not been pressed, the process returns to step S411 because the division determination has not been completed. If it has been pressed, the flow advances to the next step S428, where the reproduction order is designated.
As described above, through steps S411 to S427, the user can perform trial and error until a desired division point is obtained.
[0082]
Here, an example of the user interface displayed in step S428 when specifying the reproduction order will be described with reference to FIG. Reference numeral 1000 denotes a reproduction order designation window. Title bar 1001 informs the user that this window is a “playback order designation window”. 1002 indicates that this column is in the reproduction order. A reproduction order is displayed as in 1003. Numeral 1004 indicates that this column is a thumbnail display area of the first frame of the determined division point. 1004 is the same as 5002 in FIG. 15, 1005 is the same as 5003, 1006 is the same as 5004, and 1007 is the same as 5005. A bold frame of 1008 indicates that the second row is selected. By dragging the selected frame up and down, the order of the reproduction order can be set. An OK button 1009 is a button for instructing the end of the order change by pressing the button. The pointer 1010 can give various instructions by operating this.
[0083]
Next, in step S428, it is determined whether or not the order has been designated in the playback order designation window of FIG. This can be determined based on whether the pointer 1010 has been selected and dragged. If there is a change instruction, in step S429, the order is changed according to the position after dragging, and the display content is updated.
[0084]
Next, in step S430, it is determined whether the reproduction order has been determined. This can be determined based on whether the OK button 1009 has been pressed. If not, the process returns to step S428 and repeats the processing up to step S429. If it is determined, the process moves to step S431 to generate a play list. The play list can be generated by copying the target moving image and the section in the division information in a specified order. Then, in step S432, the moving image reproduction list is stored in the storage unit 300. This is to make it available to other playback devices. Then, in step S433, the moving image is reproduced in the reproduction window shown in FIG. 7 in the specified reproduction order.
[0085]
Here, the moving image playlist will be described.
In the present embodiment, it is assumed that the moving image reproduction list is converted into one file and stored in the storage unit 300.
[0086]
FIG. 14 is a diagram illustrating a description example of a moving image playlist. In the present embodiment, the moving image reproduction list is described in the XML format, similarly to the metadata in FIG. Since the basic description of the data structure is the same as that of FIG. 11, the description is omitted here and each element is described. The “reproduction order” is described as a pair of “target moving image” and “section” in the order of reproduction. The “target moving image” and the “section” are copies of those in the “split information”. The reproduction unit 306 can reproduce a moving image having a content desired by the user by reproducing the target moving image section in the order.
[0087]
As described above, by appropriately selecting the metadata item used for the automatic division and setting the conditions, it is possible to obtain a desired division section and create a moving image reproduction list by the user.
[0088]
[Other embodiments]
In the above-described embodiment, the case where the generated division information is used for generating a moving image play list has been described as an example, but the division information can be used for various other purposes. For example, one moving image file is divided based on the division information to generate a plurality of moving image files. When the section specified in the division information is a part of the moving image file, only the specified section is extracted and one new moving image file is generated. Only a part (for example, the first two seconds) of the section specified in the division information is reproduced, and the digest reproduction is performed. The above applications are conceivable.
[0089]
In the present embodiment, the moving image and a part of the metadata are captured and added by another device, but may be configured as a single device.
[0090]
In the present embodiment, the association between the division information and the section in the moving image is described in the division information. However, only the association information may be put together and created as a management file, for example, in an XML format. Absent.
[0091]
In the present embodiment, the metadata and the division information are described in the XML format (text data). However, this is merely an example, and may be expressed in a binary format.
[0092]
In the present embodiment, the color distribution is used as metadata that can be added in the moving image processing apparatus. However, the present invention is not limited to this. For example, it is represented by luminance, co-occurrence matrix, contrast, entropy, Gabor transform, and the like. Texture features, edges, shape features such as Fourier descriptors, face recognition, and the like.
[0093]
In the present embodiment, only the information that can display the content of the metadata is copied as the division information. However, if there is no limit such as the capacity, all the information is copied and displayed. Only possible items may be displayed.
[0094]
In the present embodiment, in the two-dimensional array display shown in FIG. 6, the section is set to the row and the metadata item is set to the column. However, the metadata item may be set to the row and the metadata item is set to the column. Absent.
[0095]
An object of the present invention is to supply a software program for realizing the functions of the above-described embodiments to a system or an apparatus having a computer capable of executing the program directly from a recording medium or using wired / wireless communication. The present invention also includes a case where equivalent functions are achieved by a computer of the system or the apparatus executing the supplied program.
[0096]
Therefore, the program code itself supplied and installed in the computer to implement the functional processing of the present invention by the computer also implements the present invention. That is, the present invention includes the computer program itself for implementing the functional processing of the present invention.
[0097]
In this case, any form of the program, such as an object code, a program executed by an interpreter, and script data supplied to the OS, is applicable as long as the program has the function of the program.
[0098]
As a recording medium for supplying the program, for example, a magnetic recording medium such as a flexible disk, a hard disk, a magnetic tape, an MO, a CD-ROM, a CD-R, a CD-RW, a DVD-ROM, a DVD-R, a DVD- There are optical / magneto-optical storage media such as RW, and nonvolatile semiconductor memories.
[0099]
As a method of supplying the program using wired / wireless communication, a computer program itself forming the present invention on a server on a computer network, or a computer forming the present invention on a client computer, such as a compressed file including an automatic installation function, is used. A method of storing a data file (program data file) that can be a program and downloading the program data file to a connected client computer may be used. In this case, the program data file can be divided into a plurality of segment files, and the segment files can be arranged on different servers.
[0100]
That is, the present invention also includes a server device that allows a plurality of users to download a program data file for implementing the functional processing of the present invention on a computer.
[0101]
Further, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM and distributed to users, and key information for decrypting the encryption for a user who satisfies predetermined conditions is transmitted to, for example, a homepage via the Internet. It is also possible to realize the program by supplying it by downloading it from, and using the key information to execute an encrypted program and install it on a computer.
[0102]
The functions of the above-described embodiments are implemented when the computer executes the read program, and an OS or the like running on the computer executes a part of the actual processing based on the instructions of the program. Alternatively, all the operations are performed, and the functions of the above-described embodiments can be realized by the processing.
[0103]
Further, after the program read from the recording medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion board or the The CPU or the like provided in the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments can also be realized by the processing.
[0104]
【The invention's effect】
As described above, according to the present invention, a user can set a metadata item and its condition according to the content of a moving image and the intention of editing a moving image. In addition, since the division information is described in a general-purpose format, it has a small device dependency and can be widely used.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating a concept of automatic division of a moving image by metadata.
FIG. 2 is a block diagram illustrating a control configuration of the moving image processing device according to the embodiment of the present invention.
FIG. 3 is a block diagram illustrating a functional configuration of a moving image processing device according to an embodiment of the present invention.
FIG. 4A
FIG. 4B
FIG. 4C is a flowchart illustrating a processing procedure of the moving image processing apparatus according to the embodiment of the present invention.
FIG. 5 is a diagram illustrating an example of a user interface when selecting a metadata item that can be used for division in the embodiment.
FIG. 6 is a diagram illustrating an example of a user interface when a division point is determined in the embodiment.
FIG. 7 is a diagram showing an example of a playback window in the embodiment.
FIG. 8 is a diagram illustrating an example of a user interface when selecting extractable metadata in the present embodiment.
FIG. 9 is a diagram illustrating an example of a user interface when setting a division condition in the present embodiment.
FIG. 10 is a diagram illustrating an example of a user interface when a reproduction order is specified in the embodiment.
FIG. 11 is a diagram illustrating an example of description of metadata in the present embodiment.
FIG. 12 is a diagram illustrating an example of description of division information according to the present embodiment.
FIG. 13 is a flowchart illustrating details of division information generation processing of the moving image processing apparatus according to the present embodiment.
FIG. 14 is a diagram illustrating an example of description of division information in the present embodiment.
FIG. 15 is a diagram illustrating an example of display of a division result according to the related art.
FIG. 16 is a diagram illustrating an example of block division.

Claims (12)

Supplementary information collecting means for acquiring supplementary information on a moving image;
Display means for presenting a user interface capable of selecting the incidental information,
Division information generating means for generating division information for dividing the moving image into a plurality of sections based on selected additional information selected via the user interface among the additional information. Moving image processing device. Presenting a user interface in which the display means displays a result that would be obtained when the moving image is divided based on the division information in a two-dimensional array with time and a type of the selected incidental information as axes; The moving image processing apparatus according to claim 1, wherein: The two-dimensional array display includes a thumbnail image representing each section of the moving image divided based on the division information, a time section corresponding to each section, and the supplementary information used for dividing each section. 3. The moving image processing apparatus according to claim 2, wherein the moving image processing apparatus is arranged at a position where a section corresponding to the type intersects. The display means displays a user interface for selecting additional information other than the selected additional information from the additional information in response to an instruction via the user interface displaying the two-dimensional array. The moving image processing device according to claim 2 or 3. The division information generating means updates the division information according to selection of additional information other than the selected additional information,
The moving image processing apparatus according to claim 4, wherein the display unit updates the user interface displaying the two-dimensional array based on the updated division information. When a deletion instruction is issued through the user interface displaying the two-dimensional array, the division information generating unit performs the division so that the section of the moving image selected in the user interface displaying the two-dimensional array is deleted. Update information,
The moving image processing apparatus according to claim 4, wherein the display unit updates the user interface displaying the two-dimensional array based on the updated division information. When there is a division condition setting for any of the supplementary information items displayed in the two-dimensional array via the user interface in which the two-dimensional array is displayed, the division information generating unit generates the division information in accordance with the division condition setting. Updated,
The moving image processing apparatus according to claim 4, wherein the display unit updates the two-dimensionally arranged user interface based on the updated division information. A storage unit that stores division information used when the user interface displayed in a two-dimensional array is first presented;
When a return to the initial state is instructed via the user interface displaying the two-dimensional array, the division information generation unit updates the division information with the division information stored in the storage unit,
8. The moving image processing apparatus according to claim 4, wherein the display unit updates the two-dimensionally displayed user interface based on the updated division information. The supplementary information acquired from the moving image by the supplementary information collecting unit includes supplementary information previously given to the moving image and supplementary information acquired by analyzing the moving image. The moving image processing device according to claim 1. An additional information collecting step of obtaining additional information on a moving image;
A display step of presenting a user interface capable of selecting the supplementary information; and a division for dividing the moving image into a plurality of sections based on the selected supplementary information selected via the user interface among the supplementary information. A divided information generating step of generating information. A computer program that causes a computer device to function as the moving image processing device according to claim 1. A computer-readable recording medium storing the computer program according to claim 11.

Images