JPH05204990A - Key word imparting method for animation image information - Google Patents

Abstract

PURPOSE:To reduce the working burden of a registering operator and to obtain an easy-to-use key word by adding a common key word to a specific partial animation image section. CONSTITUTION:The video signals outputted from an animation image reproducer are successively converted into the digital image data by an A/D converter 22 and sent to a computer. Then the digital image is synthesized in the partial area 28 of an animation image display window 27 of a display screen. The serial numbers are imparted to the frames of the displayed animation image at and after the head of the image. Then a character input window 31 inputs a key word and calls the start and end frame numbers of the partial animation image section corresponding to the key word out of a relative information storage 24. Then the window 31 sends each frame number to the animation image reproducer via a control line 26 and reproduces an animation image screen. At the same time, the disks and the tapes are replaced with each other as necessary.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method of assigning a keyword to an object in a moving image, for example, a keyword suitable for searching a desired partial moving image section in a moving image database. Regarding the method of granting.

[0002]

2. Description of the Related Art A moving image database that can selectively call out information about an arbitrary scene (moving image section that should be treated as a block in terms of content or video) from a large amount of moving image information has a new moving image. When registering an image in the database, the registrant gives a keyword to each scene in the moving image in advance.

The keywords include, for example, "rain" and "snow".
There are things such as depictions of scenes such as scenes, and objects related to objects appearing in scenes such as "cars" and "girl". Keywords are registered in the database, and depending on whether the user inputs the keywords with a keyboard or selects from a keyword list, that is, an index,
The scene to which the keyword is added is called.

For each scene, one image is used as a keyword, such as selecting one of the frames that make up the moving image in that section and using the list of such frames as an index to find the corresponding desired scene. There is also a method (for example, "Automatic Video Indexing and Full-Video
Search for Object Appearances "IFIP WG 2.6 6ndW
orking Conference on Visual Database Systems (199
1). Reference 1) below.

[0005]

The conventional method has the following problems.

In order to add a keyword, the registrant must look at the moving image to be registered in order from the beginning, understand the scene for each scene, and give an appropriate description. Although this is a very troublesome task for humans, it is necessary to rely on humans at present because of the need for advanced scene understanding ability.

In the case of adding a keyword to an object in a moving image, it is necessary to extract a partial section in which the object appears from the moving image. If it is a method of searching for a frame in which an object appears, a known object search method of Document 1 (for an object to be searched, by specifying one reference image, a moving image including the object is displayed). This can be done automatically by a search method that finds all the frames in, but this method cannot specify a series of sections from the appearance of an object to its departure.

In the method of using the representative frame screen for each scene as a keyword, it is sufficient to simply sample the frame as a keyword, and the process without human labor is easy, but the keyword itself is input like a character. Therefore, the database user must always search for a keyword in the list, and as the number of registered keywords increases, the search becomes very difficult.

As described above, the conventional method has a problem in that it is inconvenient for the user if labor is not required at the time of keyword registration, and labor is required at the time of registration if importance is attached to convenience during retrieval.

An object of the present invention is to reduce the work burden on a registrant when assigning a keyword to a moving image,
In addition, it is to realize a mechanism that can add keywords that are easy for database users to use.

[0011]

According to the present invention, when a keyword is given to an arbitrary object in a moving image, the object searching method is applied, and the object is detected in all the frames of the moving image. Check whether or not it appears, extract the section where the frames that are determined to appear are continuous, and for all the extracted sections, common keywords and information related to the target etc. at once It is to be given. Further, for a scene of a moving image to which a keyword is to be added, if an object to which the keyword has already been added exists in the scene, the keyword of the corresponding object can be used as the keyword of the scene, Further, the keywords and related information of the target object are provided to the registrants.

[0012]

Operation: The registrant only needs to specify one image of the target object, and the machine automatically finds the partial section in which the target object appears in the moving image, and most of the partial section in which the target object appears. Since the common keyword and the related information are allotted to all of the above, it is not necessary to search the existing sections one by one in the moving image for the same object one by one and add the keyword one by one. In addition, when assigning a keyword to a scene, the keyword of the object in the scene is automatically added, and information about the object is presented, so
Even if the registrant does not remember the name of the object or related information,
When considering a new keyword, it is easy to cite the object information.

[0013]

EXAMPLE An example of the present invention will be described in detail below.

FIG. 1 is a flow chart showing an embodiment of the processing procedure when the present invention is applied to the processing for adding a keyword to moving image information, and FIG. 2 is a system configuration for realizing the present invention. 2 is a schematic block diagram of FIG. Also,
FIG. 3 shows a specific display example on the display of FIG.

In FIGS. 2 and 3, reference numeral 20 denotes a display such as a CRT, which displays an output screen of the computer 23. The moving image reproducing device 21 is an optical disk, a video deck, or the like. The video signal output from the moving image reproducing apparatus is sequentially converted into digital image data by 22 A / D converters and sent to a computer. The sent image is combined with the partial area 28 of the moving image display window 27 in the display screen. A number (frame number) is sequentially assigned to each frame of the displayed moving image from the beginning of the moving image. Reference numeral 31 is a character input window. By inputting a keyword, the start frame number and the end frame number of the partial moving image section corresponding to the keyword are called from the 24 related information storage devices, and these frame numbers are controlled by the control line. By sending the moving image to the moving image reproducing apparatus by 26, the moving image of the scene is reproduced. At this time, the disks and tapes are replaced as needed. Reference numeral 32 is a window for displaying a list of keywords, that is, an index. By selecting a keyword from the list, a moving image of the corresponding scene is reproduced. 29
Is a button area, and by moving the pointing cursor to the area and pointing, the control (playback / rewinding, etc.) corresponding to each area of the button area is performed on the moving image playback device.

The procedure for adding a keyword to moving image information will be described below with reference to the flowchart of FIG.

In process 100, the registrant reproduces the moving image using the moving image reproducing apparatus and waits for the target object to which the keyword is added to appear in the moving image display area 28.
When it appears, the moving image reproducing apparatus is brought into a still image display state.
When the section in which the object appears has passed, rewinding control is performed so that a still image is displayed in the frame in which the object appears. Then, an area surrounding the image area of the object in the frame displayed as a still image is designated by using an instruction cursor or the like, and the image of the area is cut out.

In step 102, all the frames in which the object appears are selected from the moving image. Specifically, the region image obtained in the process 100 is used as a reference image, and the process of applying the target object search method of the above-mentioned document 1 to a moving image to be registered is performed. FIG. 4 shows an example of the frame sequence of the moving image immediately after the object search. A shaded frame such as 40 indicates a frame in which an object exists, and a frame without a shade such as 41 is a frame in which the object is not searched. The frame sequence in this example consists of two cuts 42 and 43. Reference numeral 44 denotes a cut, which is a connecting portion between the cuts. Cut changes can be detected without human intervention by using the method used for cut division in Reference 1. The array 45 is a one-dimensional array that temporarily stores the result of the object search, and each of the elements of the array is sequentially associated with all the frames of the moving image. Each element at this time is 0 or 1
If any one of these values is taken and it is 1, it indicates that the object is present in the corresponding frame. A value of 0 indicates that the object does not exist.

In process 104, the partial sections in which the object appears continuously (the partial sections 46 to 49 in which 1 in the array 45 is continuous) are extracted. For example, in the array 45, when the elements are examined in order from the beginning and 1 appears for the first time,
The number of the element (corresponding to the frame number) is stored as the starting point information, the next element is searched one by one next, and the number of the element immediately before the first occurrence of 0 ( When the end element of the array is reached without 0 appearing, the end element number) is stored as the end point information. An ID is given to the partial section thus obtained, and the start point information and the end point information are registered together in a list (hereinafter, section list). If the end point at this time is not the end of the moving image, the element next to the end point is searched again in order, and the same processing is repeated.

Next, among the partial sections registered in the section list, those adjacent to each other at very short section intervals are integrated. For example, by sequentially examining the partial sections registered in the section list, the difference between the end point of the partial section and the start point of the subsequent partial section (that is, the number of frames between both sections)
Is smaller than the threshold set by the registrant (1
06) If smaller, the end point of the former partial section is replaced with the value of the end point of the latter partial section, and the latter partial section is removed from the section list (108). This process is performed between all adjacent partial sections registered in the section list (110). In the example of the figure, when it is determined that the interval between the subsections 46 and 47 is short, the two sections are integrated and a section from a to f is newly set as one subsection.

For each partial section registered in the updated section list, if there is a cut change in the partial section, the corresponding partial section is divided at the cut change position. For example, first, each partial section of the section list is sequentially checked to see if there is a cut change in the partial section (112). If there is only one cut change, create a partial section in which the frame number immediately after the cut change is the starting point and the end point is the same as the relevant partial section, add an ID, and add and register it in the section list. Replace the end point of the partial section with the frame number immediately before the cut change (11
4). When there are two or more cut changes, the same idea is used for division. In the illustrated example, since the cut change 44 exists in the partial section 48, the partial section 48 has i to j and k to
It is divided into two subsections of l. This series of processing is performed for all the partial sections registered in the section list (1
16).

Among the partial sections registered in the updated section list, the very short partial section is removed from the list. For example, each partial section of the section list is sequentially examined to see if the partial section consists of a number of frames smaller than the threshold set by the registrant (118).
If not, the partial section is removed from the section list (120). This series of processing is performed for all the partial sections registered in the section list (122).

For each partial section of the updated section list, whether the starting point and the ending point of the section are correctly defined or the section in which the object appears correctly is checked while being successively reproduced by the moving picture reproducing apparatus. If there is a mistake, the mistake is corrected (124).
For example, when the start point and the end point frame of the partial section are different, the change is made by searching for and marking the correct frame while reproducing the moving image, or in some cases, the entire section is removed from the section list. Further, when a unique keyword and related information are added to the partial section, this is done here.

Finally, a keyword and related description information are added to the object, and the keyword is registered in the related information storage device together with the section list. Also, the keyword list display window is updated (126).

As described above, the series of processing is performed on all the objects to which the keyword is added (128). When an unregistered section for an object that has already been keyword-registered is found while processing a certain object, the section is additionally registered in the section list of the corresponding keyword information in the database at a suitable time.

In process 130, the start point and the end point of the section are designated for an arbitrary scene to which a keyword is to be added. For example, the section is defined by searching for and marking the start point frame and end point frame desired by the registrant while reproducing the moving image.

In the defined scene section, if an object already registered as a keyword appears in the scene section (if there is a section where the existing section and the scene section in the moving image of the object overlap), For all relevant objects, the registered keywords and related description information, and the range in which the object appears in the scene section, etc. are registered in the related information storage device as information related to the scene, and displayed on the display. (134). The registrant, while referring to this information or using it as it is, adds appropriate keywords and related description information, and combines it with the start / end point information of the scene section and the pointer to the object information appearing in the scene. Register in the related information storage device (13
6). An explanatory view of this case is shown in FIG.

FIG. 6 shows an example of a data structure generated and registered by the keyword assigning process.
These are stored in the related information storage device 24. Reference numeral 51 is a total list of registered target object information, and each target object information has a data structure including a keyword character string, target object related information, section list, and the like, as indicated by 52. The section list is a list of each piece of section information as shown at 53, and each piece of section information is composed of start point information, end point information, etc. as shown at 54. 55 is
This is a list of registered scene information, and each scene information is represented by 56, which is a keyword character string, scene-related description information, a pointer to object information of an object existing in the scene, a starting point / It consists of end point information.

The keyword list window 32 displays the keyword character strings in the data structure 52 for all the object information registered in the object information list 51. When searching the partial moving image section in which the target object appears, each target in the target object information list is searched for the keyword character string selected by the database user from the keyword list or input from the character input window 31. It collates with the keyword character string of product information. With respect to the target object with which the keyword matches, the start point and the end point of each partial section registered in the section list are sent to the moving image reproducing apparatus, and the section is reproduced.

In the scene search, a keyword is added to the scene in the same manner as described above, or a keyword relating to an object appearing in the scene is input, and an object matching the keyword is searched for. Find a scene that has a pointer to the corresponding object. When searching for a scene in which multiple objects appear, if keywords are given for each of the multiple objects, the search is performed using the temporal existence relationship between the objects in the scene as a constraint condition. Good. For example, when the order in which the objects appear is a constraint condition, as shown in FIG. 7, for each object appearing in one scene, the existing sections in the scene are called to compare the starting points and The scene is searched when the order specified by the condition and the order of the start points match. In the same manner, the leaving order of objects and the fact that they appear at the same time are also set as constraint conditions.

In the above-described embodiment, the integration processing (106 to 110) and the removal processing (118 to 122) of the partial sections are performed.
The reason is that, as with many other image processing application techniques, reliable search results are not always obtained with the object search method. Due to the shape and color of the object that slightly changes frame by frame, noise, or the like, the frame in which the object exists cannot be found, or conversely, an irrelevant frame is erroneously searched for. Therefore, when the interval is very short in the adjacent partial sections (for example, when the normal playback time is less than 1 second, it is difficult to find the discontinuity with human eyes). Is
It can be considered that the section between them is a portion in which the search has failed despite the existence of the object, so that it is newly registered as a continuous partial section including the portion. Further, when the registered partial section is very short, it may be considered that the object is erroneously detected even though there is no object, so the registration of that part is deleted. At this time, it is needless to say that the criterion (threshold value) for determining that it is extremely short can be freely changed according to the nature of the moving image to be registered and the intention of the registrant (especially, when the threshold value is 0 frame, No processing or wiping is done).

In the processing 112 to 116 for dividing the sub-section at the position where the cut changes, even if the object appears continuously, the scenes before and after the change of the cut are different (for example, the background and the shooting position are different). Etc.), which should be treated separately.

In this embodiment, an example of a keyword using a character string is shown, but it is not denied that various information formats such as images and sounds are used as keywords. Of course, in the information format other than the character string, the keyword may not be directly input, but it can be used by a method such as selecting from the index.

The present invention can be implemented in the form in which various modifications are further made. For example, it can be applied to a task of defining a button on an object in a moving image hypermedia system. The button referred to here is a mechanism for presenting relevant information by pointing a partial area defined to surround the target object in the display image with an instruction cursor, etc., for the target object being displayed on the display. It means a data structure that combines the coordinates of the area and the processing procedure when pointed. To define a button, it is necessary to search for a partial section in which an object exists, and this embodiment (particularly, processing 10
0-124) is very effective.

Further, it is possible to easily give a hyperlink for a certain object by using buttons to mutually refer to the same object existing in different scenes. For example, after extracting each partial section in which the object appears, pointer information to another partial section is added to the button information set in each partial section.

[0036]

As described above, according to the present invention,
If the registrant specifies only one image of the target object to which the keyword is given, the machine will automatically find the partial section where the target object appears in the moving image, so for each target object With one-time keyword addition, keywords can be added simultaneously to most of the partial sections where the object appears. Also, when assigning keywords to the scene,
Keywords for objects are automatically assigned,
When adding a keyword, a part of necessary information is automatically presented, which makes it easy to think of the keyword.

[Brief description of drawings]

FIG. 1 is a flowchart showing an embodiment of a processing procedure of the present invention.

2 is a schematic system block diagram for implementing the embodiment of FIG. 1. FIG.

FIG. 3 is a specific example of a display screen on the display of FIG.

FIG. 4 is a conceptual diagram representing a moving image as a frame sequence.

FIG. 5 is a diagram for explaining keyword assignment when the same target object appears in a plurality of scene sections.

FIG. 6 is an example of a data structure generated by the embodiment of FIG.

FIG. 7 is an explanatory diagram for searching a case where a plurality of objects appear.

[Explanation of symbols]

27 ... Moving image display window, 28 ... Moving image display area,
29 ... Button area, 30 ... Instruction cursor, 31 ... Character input window, 32 ... Keyword list display window,
40 ... Frame in which the object appears, 41 ... Frame in which the object does not appear, 42, 43 ... Cut, 44 ... Cut change, 45 ... Object presence / absence sequence, 46 to 49 ... Object appears continuously Partial section, 50 ... Section list, 5
1 ... Object information list, 52 ... Object information, 53 ... Section list, 54 ... Section information, 55 ... Scene information list, 56
… Scene information.

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Mitsuru Ikusawa 1-280 Higashi Koikekubo, Kokubunji City, Tokyo Metropolitan Research Center, Hitachi, Ltd.

Claims (10)

[Claims] 1. A keyword assigning method of moving image information, wherein a keyword is assigned to an arbitrary object appearing in a moving image, and it is determined whether or not a specific object appears in each frame of the moving image. Then, a moving image characterized by searching for partial moving image sections in which frames in which it is determined that a specific target object appears continuously and assigning a common keyword to those partial moving image sections Image information keyword assigning method. 2. The moving image information keyword assigning method according to claim 1, wherein information relating to the object including a common keyword is attached. 3. The object-related information according to claim 2, wherein at least a keyword character string, object-related description information, and a section list of each partial section having start point information and end point information are stored. A method for assigning keywords to moving image information. 4. The partial moving image according to claim 1, wherein when the plurality of searched partial moving image sections are adjacent to each other at an interval shorter than a predetermined section interval, the plurality of partial moving image intervals are integrated. A method for assigning keywords to moving image information, characterized in that it is an image section. 5. The moving image information according to claim 1, wherein when there is a cut change in the searched out partial moving image section, the partial moving image section is divided at the position of the cut change. Keyword assignment method. 6. The keyword of moving picture information according to claim 1, wherein when the searched partial moving picture section is smaller than a predetermined number of frames, no keyword is given to the partial moving picture section. Granting method. 7. A keyword assigning method for moving image information, wherein a keyword is assigned to one scene in a moving image, by determining whether or not a specific object appears in each frame of the moving image, The partial moving image sections in which the frames determined to appear are continuously searched for, the common keywords are given to those partial moving image sections, and within the section of the one scene, A keyword assigning method for moving image information, comprising checking whether or not an object to which a keyword is assigned is present, and calling the keyword of the relevant object to use as a keyword of the one scene. 8. The method of assigning a keyword to moving image information according to claim 7, wherein the keyword of the corresponding object is presented to the registrant. 9. The method for assigning keywords to moving image information according to claim 8, wherein information relating to the object other than the keyword of the object is presented. 10. The object-related information according to claim 8, wherein at least a keyword character string, object-related description information, and a section list of each partial section having start point information and end point information are stored. As related information,
At least a keyword character string, scene-related description information, a pointer to object-related information of an object existing in a section of a scene, and a keyword assigning method of moving image information characterized by storing start point information and end point information ..