JP2004127311A - Associative retrieval device of image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an associative retrieval device of an image and its method allowing a user to freely and easily execute associative retrieval of the image. <P>SOLUTION: This associative retrieval device is provided with: an image display means having an image display region for viewing the image and an index display region for displaying index information; a point detection means for detecting point positions for the display regions; an object control means for previously registering matter or sound in a moving image appearing in the image; and a control means for defining the state of the image to be subsequently reproduced from the point information obtained from the point detection means and logical structure description of a certain image separately stored. Even if a desired scene can not be identified by the index information, multilateral image retrieval associatively using the respective displays can be carried out such that the scene is finally obtained by finding a scene where a clue as to the scene appears and by associatively retrieving the desired scene based on it. <P>COPYRIGHT: (C)2004,JPO

Description

The present invention relates to an apparatus and a method for associatively searching for an image and finding an arbitrary image.

In recent years, with the speeding up and large capacity of computers, the construction of databases for video information such as movies and videos, which could not be handled conventionally, has become active. Along with this, practical use of a search technique for efficiently selecting a desired scene from a large amount of accumulated videos has been promoted. As a method for such a search, a method in which a user designates a feature or a keyword of a desired scene, and a computer finds a scene to which a keyword matching the keyword is attached is generally used in the field of video databases. I have. However, it is very difficult for a user who is unfamiliar with the search, as well as a skilled person, to accurately specify the characteristics of the scene, and often the desired search result cannot be obtained.

A table of contents and an index are prepared as supplementary information for search in a book that is classic information. The table of contents is information in which keywords symbolizing a group of the text are listed in the order of progress in the text. The index is information in which important keywords in the text are arranged and listed in an easy-to-find order such as the Japanese syllabary. The most common feature of both is that such keywords are listed. And it is decided that it is always at the beginning or end of the book, so there is no need to search for it. Readers can use the table of contents and index to find passages in the text without having to consider keywords themselves. The table of contents also gives a quick overview of the text and whether the book is worth reading.

(4) In the search using the table of contents or the index, there is a problem that if there are too many displayed keywords, it is difficult to find an appropriate part, and if there are few displayed keywords, there is no appropriate keyword in the first place. However, these problems can be solved in combination with hypertext and full-text search. That is, first, the number of items in the table of contents / index is limited to some extent and presented to the user. The user may refer to the text using a suboptimal keyword that is likely to be related to the target part, and search for a keyword in the text that is likely to be directly related to the desired part. If found, the goal can be achieved by using the hypertext mechanism to refer to the desired location. This is a technique that is routinely used for searching online manuals. It is necessary that a keyword is registered in the hypertext in advance, but the same can be performed with a keyword that has not been registered by using a full-text search. In this way, the associative tracing mechanism of keywords expands the usable range of the table of contents and index, and in many cases, the target site can be searched by simply selecting keywords that appear in front of the eyes (hereinafter, referred to as (Called associative search).

Such a mechanism is also considered to be effective in video search. In an image, various things such as people and things appearing in the image can be used as corresponding to the above keywords. As an elemental technology for realizing the associative search using this, for example, as a method of referring to a related scene or information from an object on a video display screen, Japanese Patent Application Laid-Open No. 3-52070, "Moving Image Related Information Reference Method" There is. According to this, by providing a means for storing an image section and a position where each thing in a video appears and a means for associating with the related information, one point on the screen on which each thing is displayed can be a mouse or the like. By pointing, the user can easily jump to a related scene or call up related information. Japanese Patent Application Laid-Open No. Hei 5-204990 discloses an example of a means for saving the correspondence between each object and its related information by using image processing.

JP-A-5-204990

The prior art mentioned above is a means for exclusively associating each thing with its related information, and although the structure of the entire search system and the usability of the user have been sufficiently studied. Absent. In addition, there is a problem that it is possible to associatively trace only an object that has been associated with the related information in advance.

An object of the present invention is to allow a user to find a desired scene while associatively tracing a memory by simply selecting the limited information presented by a computer when searching for a video. To provide an interface.

The second object of the present invention is to provide a means for associatively tracing an item that has not been previously associated.

According to the present invention, an image display area for displaying an arbitrary image on a screen, an operation panel area for controlling a reproduction state of an image, and an area for displaying index information corresponding to a table of contents or an index of the image are provided. It has means for detecting which of the display areas is pointed, and means for determining the state of the video to be reproduced next from the point information and the descriptive information of the video separately stored. Further, there are provided a means for grasping the displayed thing and its position, superimposing and displaying related information of the thing, and a related information registration changing means. In addition, means for registering and managing information necessary for these processes is provided.

A means for designating a specific thing appearing in the scene being displayed; a means for extracting a feature amount of the thing; a means for searching for another video scene having a feature amount matching the feature amount; Means for immediately jumping to the found video scene.

According to the present invention, when searching for a desired scene, even if there is no information on an object directly related to the desired scene in the index information of the video, the object is searched for one after another while retrieving some objects reminiscent of the desired scene. Scene can be reached. In this way, by combining the index display and the associative search organically, the range of use of the index is greatly expanded, and in many cases, the target scene is simply processed by simply selecting the information presented by the computer. Be able to search. For this reason, it is not necessary to consider or indicate an appropriate keyword or image feature amount that uniquely determines a target scene, and a search can be performed even with a sloppy memory, so that even a beginner can easily understand. In addition, the related information superimposing means superimposes, on the position of the object in the reproduced video, the selected part or all of the relevant information of the object appearing in the video being reproduced, or the relevant information and the related information. Since the information is displayed in a form that clearly indicates that it corresponds to the information, it is possible to immediately and accurately know the information about the thing that appeared during the associative search without confusing the information about which thing. . Further, by providing the related information registration changing means, a part or all of the related information of the thing appearing in the video being reproduced can be registered or changed immediately on the spot where the thing appears.

Further, even when the information currently displayed on the screen has not yet been provided with information for jumping to a related scene, a feature amount of the object is extracted from the display screen and the feature amount is compared. Thus, another scene in which the object appears can be searched and projected on the spot.

According to the present invention, when a desired scene is searched for, even if the scene cannot be completely identified from the index information, as long as a scene in which some clue related to the scene appears, the scene in which the clue appears can be searched associatively. However, a desired scene can be finally obtained, so that a multi-faceted video search using the respective displays can be performed. In addition, it is possible to immediately and accurately know information on an object in a video being reproduced without confusing which information the object is. In addition, part or all of the information related to the object appearing in the video being reproduced can be changed immediately at the moment the object appears. Further, according to the monitor window of the present invention, it is possible to constantly monitor the position of the scene being reproduced in the entire video, and even if the scene jumps in the associative search, it is combined with special effects such as wipe. Therefore, it is clearly understood that it is not confused with a normal scene change. In addition, even if the user points at the display area of the related information superimposed and displayed, the same effect as that obtained by pointing the object can be obtained. Therefore, a convenient point method can be selected for each scene, and the operability is improved. Also, by listing the related information to be displayed, the trouble of directly inputting the key can be omitted, and even if the key is forgotten, the menu can be recalled by recalling. As described above, according to the present invention, a user-friendly associative search can be realized.

Hereinafter, one embodiment of the present invention will be described in detail.

FIG. 2 is a schematic block diagram of a system configuration example for realizing the present invention. A display device 1 such as a CRT displays an output screen of the computer 4. Reference numeral 12 denotes a speaker for reproducing sound. An instruction to the computer 4 can be given using an indirect pointing device 5 such as a mouse, a direct pointing device 13 such as a touch panel, or the keyboard 11. The video reproducing device 10 is a device for reproducing video from an optical disk, a video deck, or the like. The video signal output from the video playback device 10 is sequentially converted into a format that can be handled by the computer 4 by three video input devices, and sent to the computer 4. Inside the computer, the video data enters the memory 9 via the interface 8 and is processed by the CPU 7 according to the program stored in the memory 9. Each frame of the video handled by 10 is numbered sequentially from the top of the video, for example, a frame number. The video of the scene corresponding to the frame number is reproduced by transmitting the frame number from the computer 4 to the video reproducing device 10 through the control line 2. Video data and various information can also be stored in the external information storage device 6. The memory 9 stores various data created by the processing described below in addition to the program, and is referred to as needed.

Below, the outline of the associative search system is explained first, and then the detailed execution procedure of each technology is explained.

FIG. 1 shows an example of a screen of a system for realizing an associative search for video. 1 is a display device, 12 is a speaker for outputting sound or BGM, 5 is an indirect pointing device such as a mouse or a joystick, 11 is a keyboard, and 13 is a direct pointing device such as a touch panel. .

A monitor window 1100 in the display device 1 is a monitor screen, and has an operation panel 1102 of the same type as the VCR, so that a video can be freely reproduced and viewed. The image displayed on the monitor screen corresponds to "text" in "book", and the operation of the panel (button) corresponds to "page turning". The lower right window 1108 displays a scene list of representative images of each scene of the target video, and the lower right window 1112 displays a list of subjects appearing in the video. These list displays are collectively called "indexes". In the scene list display of the window 1108, a typical frame image is selected from each scene in the video, reduced, and arranged in a list as icons 1110 in chronological order. These images can be considered to be equivalent to the “headlines” of the scenes.
Table of Contents ". On the other hand, the subject is one of the important components of the scene, and in that sense, corresponds to a “keyword” in the text. Therefore, the display of the list of subjects in the window 1112 corresponds to “index”. When the mouse is clicked on icon 1110 in the scene list display, the image on the monitor screen is switched, and the scene indicated by the icon is reproduced. The subject list display includes an icon 1114 indicating what the subject is, and a time axis display section (bar graph) 1116 on the right side thereof. In the time axis display section (bar graph), the left end is the time axis indicating the beginning of the video and the right end is the end, and the portion displayed as a bar indicates the time section in which the subject appears. Clicking on the bar displays the video of that section on the monitor screen. Note that a cursor 1104 moves in accordance with the movement of a pointing device such as a mouse, and a window 1106 is a general-purpose input / output window for displaying various related information of video.

Next, the basic concept of the associative search according to the present invention will be described with a simple example. Suppose that a user wants to find a specific scene in which a subject B appears in a series of videos. If the icon of the target scene (the scene in which the subject B appears) or the icon of the subject B exists luckily in the scene list display of the representative image displayed in the index or the list of the subjects, directly click the icon, The intended purpose is achieved by regeneration. However, usually, the video information is enormous, and it is often difficult to find the target scene easily (for example, if the time during which the subject B appears in the video is short, it is not easy to search for. Will be understood). Therefore, the associative search of the present invention has an important meaning. That is, even if the target scene (subject B) cannot be directly searched for, the user often has some knowledge about the target scene (subject B). In the present invention, a link named associative is used by using the knowledge. It is to apply. For example, if the user remembers that the subject B and the subject A appeared at the same time (there should have been a scene), or thinks that the prediction that the possibility that the subject B and the subject A appeared at the same time is high holds. First, an attempt is made to search for the subject A.

FIG. 3 shows an image of the video associative search function of the present invention. The three pictures (scenes 1 to 3) in the figure are illustrations each representing one scene in the video displayed on the monitor screen when the associative search is performed. For example, the user searches the index (subject icon 1114 in the window 1112) for one scene in which the subject A that can associate the target subject B is displayed, and displays the scene on the monitor screen. When the leftmost scene 1 is reproduced on the monitor screen of the monitor window 1100, when the subject A is clicked with the mouse cursor among the appearing subjects A and C, the center in the figure where the subject A appears is displayed. The screen switches to scene 2 of. If another subject B that appears together with this scene 2 is clicked, it is possible to reach the scene 3 on the right side in the figure where B appears. If this scene is the target scene, the associative search ends.

That is, when finding a specific scene in which the subject B appears, based on the association that the subject B appears at the same time as the subject A, the identification of the subject B as the target scene is associatively performed through the subject A registered in the index. Can be traced to the scene. At this time, a troublesome operation such as thinking of a keyword is not necessary, and only the information appearing on the screen should be looked at and a selection can be made.

Note that, as will be described later, it is possible to perform a search using not only an association between a plurality of subjects but also an association based on all multimedia information of a video, such as a scene itself, words, BGM, and subtitles.

By the way, information necessary for realizing such an associative search function is basically composed of (1) a video section (appearance section) where the subject appears, and (2) a position of the subject on the screen (appearance position). ), (3) Other video sections (link information) to be switched when clicked. These three pieces of information are handled as a set.

Which subject is clicked during video playback is determined from the appearance section / appearance position information of (1) and (2), and the video switching destination is determined from the link information of (3) stored in the same set. You. Here, the video is realized by continuously displaying still images called frames at a rate of 30 frames per second. If a serial number called a frame number is sequentially assigned to these frames from the beginning of the video, the appearance section of (1) can be represented by the first frame number and the last frame number of the section. The appearance position (2) is coordinate information indicating in which region of each frame in the section (1) the subject is displayed. As the link information of (3), a link is provided so that another scene in which the same subject appears can be cycled one after another. The same subject often appears many times in one video, but with this link, all scenes in which the subject appears can be easily called by just clicking.

(4) The associative search method with the above configuration can be used only for a subject for which link information has already been set. However, among the three types of information necessary for the associative search described above, the appearance section and the appearance position of the subject can be obtained by, for example, an automatic subject search technology such as Japanese Patent Application No. 4-261333 by the inventors. .

FIG. 4 shows an outline of the subject automatic search algorithm. Basically, a color combination unique to a subject to be searched is found in a frame. First, the user selects only one frame in which the subject appears from the video image as an example image, and extracts a characteristic color from the image. After that, the system divides every frame in the video into smaller blocks one by one, and searches for blocks containing characteristic colors. If the number of blocks including the characteristic color is more than a certain number for each color in one frame, it is determined that the subject exists in that frame. The appearance position of the subject in the frame can be easily obtained by examining where in the frame the block including the characteristic color of the subject is distributed in the above-described subject search processing.

However, this subject search method itself basically presents an example image to the system, and it is necessary to manually find at least one section in which the subject to be searched appears, which is often troublesome. However, in the case of the associative search as in the present invention, the subject on the monitor screen can be used immediately as an example image, so that the subject search method can be used very effectively.

Furthermore, if all frames in the video are divided into blocks in advance and a list of the types of colors included in each block is stored in the storage device, the above-described subject search eliminates the need for the block division processing for each frame. Become very fast. The speed is 100 times faster than real-time performance even with the performance of the current workstation, and all the object appearing sections can be found in about 30 seconds from a one-hour video. If only one nearest appearing section needs to be searched from the currently displayed video, it can be found in about several seconds on average. Of course, the same color list stored in the storage device can be used regardless of the subject to be searched.

In the following, an execution procedure of the system for realizing the present invention will be described with reference to a block diagram of a software module executed by the CPU 7 according to a program stored in the memory 9. Each module described here can also be realized by hardware itself.

FIG. 5 is an example of a processing block diagram for realizing a scene associative search according to the present invention. Information (at what time and in what position in the video an object such as a subject appears in the video as a clue for the associative search), as well as its related information and information on the scene to jump to (link) Information) is stored in advance in the memory 9 or the external information storage device 6 in FIG. 2 in the form of a data structure called an object described later.

Here, information on an object such as a subject in a video is managed in an object-oriented data structure created one by one for each appearance section. FIG. 6 is an explanatory diagram showing the concept. This is hereinafter referred to as a video object or simply an object. The video is divided into a moving image part and an audio part, and the whole moving image can be represented by a three-dimensional space composed of an xy plane forming a frame image and an axis of time t. It can be considered that subspace. A video object is defined as a data structure associated with this subspace on a one-to-one basis. (In other words, even if the subject is the same, in principle, it is defined as a video object for each appearance section, and a link is provided between the video objects (between the subjects).) Various information in the video such as subtitles and scenes can be associated. Similarly, for audio information such as words and BGM, a video object can be defined as a data structure that is associated one-to-one with any partial section of the audio information space having a time axis. Then, the link information is stored as a pointer that mutually references the video object. In this way, even if the corresponding media, such as between moving images and audio, is different, the link can be set freely between all pieces of information in the video by managing them using the common data structure framework.

Now, returning to FIG. 5, the processing block diagram will be described in detail. The object management unit 120 is a module for managing these objects. The object management unit 120 performs registration, change, and deletion processing of the objects, and, when requested by another module, extracts object information 122 that meets the indicated conditions. , Presented in that module. The video playback display unit 100 performs video playback and display processing on a monitor window 1100, which is the monitor screen of the display device 1 in FIG. 1, and also displays the playback position information 218 of the currently displayed video in a 102-point object. Send to identification unit. The point position detection unit 104 constantly monitors the indirect pointing device 5 such as a mouse or the direct pointing device 13 such as a touch panel in FIG. 1, and displays position information 112 on the display screen on which the user has performed a point operation. To the point object identification unit 102. In addition, the position information 112 is also sent to the index management unit 108 and the operation panel unit 110. The point object identification unit 102 sends the reproduction position information 218 received from the video reproduction display unit 100 to the object management unit 120, and obtains information on all the objects registered as appearing at the reproduction position as objects. If there is a corresponding object, position information of the object is further obtained from the object and collation with the position information 112 from the point position detection unit 104 is performed to identify which object is pointed. The point thing identification unit 102 sends information 114 about the identified thing to the video control unit 106. The video control unit 106 sends the control information 208 to the video reproduction / display unit 100 to perform processing such as jumping to another scene where the object appears based on the link information in the information 114 on the identified object. send. In addition, as described later, the control information 210 is sent to the video reproduction display unit 100 when displaying the related information of the thing. The index management unit 108 stores a representative frame image of the registered video as an icon 1110, and displays a list of those icons in a window 1108. The index management unit 108 also stores the frame number together with the icon 1110. When the point position detection unit 104 detects that a certain icon is pointed, the index management unit 108 reproduces a scene corresponding to the icon. The control information 116 is transmitted to the video control unit 106. In addition, information 124 indicating which object is pointed is received from the point object identification unit 102, and a display is made such that the index indicates that the object has been pointed. The index management unit 108 also manages a list display of subjects in the window 1112 in FIG. In other words, the control information 116 is displayed so that the icon 1114 indicating the subject is displayed, the time axis of the subject is displayed (bar graph display), and when the bar is clicked, the video of the section is reproduced. The image is sent to the video control unit 106. The operation panel unit 110 displays the operation panel 1102 of FIG. 1 showing various playback states such as playback, fast forward, and rewind. When the point position detection unit 104 detects that the operation panel is pointed, , And sends control information 118 to the video control unit 106 so as to set the playback state corresponding to the pointed operation panel.

FIG. 7 is an example of a processing block diagram showing the video playback / display unit 100 in more detail. The video reproducing unit 200 reproduces the video according to the control information 208 sent from the video control unit 106 and receives instructions such as what video is to be reproduced from where and how. The reproduction position information 212 of the currently displayed video is sequentially sent to the thing presence / absence determination unit 202. 202 checks whether or not a pre-registered thing has appeared in the video at the playback position, and if so, acquires position information 216 on the display screen of all the appearing things, The information is sent to the related information display unit 204. The position information 216 is the same as the position information acquired by the point object identification unit 102 described above, and can be sent to the point object identification unit 102 as object information 218 in order to avoid duplication of position information acquisition processing. The related information display unit 204 can display related information of each item being reproduced along with the screen. The control information 210 determines whether or not related information is to be displayed, and if so, which related information is to be displayed in what form. In particular, the position information 216 of the object can clearly indicate which position in the display corresponds to the information. This display method will be described later. Depending on the display method, the image 214 is superimposed and synthesized, and the image is displayed on the image display unit 206.

FIG. 8 is an example showing the data structure of a video object. 500 is a frame of a data structure. Reference numeral 502 denotes an object ID number, which is given a unique number for identifying the object from other objects. A classification code 504 indicates whether the object represents, for example, a person, a caption, or a sound. Reference numeral 506 denotes a pointer to a video in which the object appears. In this example, as described later, the video has a data structure divided into two layers of a physical video 600 and a logical video 900, and reference numeral 506 is a pointer to the physical video. 510 is the frame number of the start point of the section in which the thing in the video represented by the object appears, and 512 is the frame number of the end point. Reference numeral 508 denotes a frame number of a video representing the object, which is used as a picture of an icon representing the object under an interface that visually handles the object. Reference numeral 514 denotes a pointer to a structure 700 for indicating the position on the screen of the thing represented by the object.

FIG. 9 shows an example of the object position structure 700. This structure takes the form of a linked list that is created one by one for each section where there is no movement of the object or that is sufficiently small. Reference numeral 702 denotes a start frame number of the section having no motion, and reference numeral 704 denotes an end frame number. Reference numerals 706 to 712 denote the origin coordinates and the size of the rectangular area when the thing is surrounded by the rectangular area. Reference numeral 516 denotes a pointer to a higher-level object having a higher abstraction degree. Every object can have its own related information, but it may be more convenient to share the related information among several objects. For example, a subject such as a person or an object in a video often has the same subject appearing in a plurality of scenes. Of course, the appearance and behavior when they appear differ from scene to scene, so each scene has its own related information. However, it is better to share information with a high degree of abstraction, such as the name, gender, age, and occupation. The data amount is small, and the integrity is not broken when information is updated. In this sense, such information having a high degree of abstraction has a data structure having a pointer 516 to the higher-level object as related information of the higher-level object. Reference numeral 518 denotes a pointer for referencing a lower-level object from a higher-level object. This is because the upper object and the lower object use the same 500 data structures. Of course, information directly related to the video, such as the start and end frames and the position information, is not required for the upper-level object, and thus a simplified version of the structure without such information can be used.

Reference numeral 520 denotes a pointer to the dictionary 800 that stores information related to an object. As shown in FIG. 10, the dictionary includes a key 802 which is a pointer to a character string 804 serving as a key for calling related information, and a character string 808 of related information registered in association with the key character string. It is composed of links 810 having contents 806 as pointers and pointers to related objects, and is created in the number of items of related information to be registered, and takes a linked list form in which the items are linked in a daisy chain. Reading related information of an object is performed by specifying a key and returning the contents of the dictionary structure that matches the key. For example, if the key is "name" and the content is "taro", specifying the key "name" will give the related information "taro". In the related information display unit 204, the selection of which related information is to be displayed results in the process of displaying the content corresponding to which key. The link is a pointer to an object to jump to when performing an associative search, and the content 806 contains a character string or a symbol representing the meaning of the link such as “the same subject appearing in another scene”. The link destination 810 contains a pointer to the object of the subject. When jumping in the associative search, the video control unit 106 reads the video in which the subject appears and the top frame number from the structure of this object, and controls the video reproduction unit 200 to reproduce the video from the video position. .

FIG. 11 is a more detailed processing block diagram of the video reproducing unit 200. The video has a two-layer structure of a logical video and a physical video. A logical video has only structural information as a set of scenes, and a physical video has actual video data. The logical video calling unit 300 calls a matching logical video from the logical video library 304 based on the reproduction position setting information 310 sent from the video control unit.

FIG. 12 shows an example of the data structure 900 of the logical video. Reference numeral 902 denotes an ID number for uniquely specifying a logical video. Reference numeral 904 denotes a scene number representing a logical image. Reference numeral 906 denotes a linked list representing the constituent scenes, which are linked in the order in which the scenes 1000 should be reproduced. Reference numeral 908 denotes setting information of special effects such as dissolve and wipe between scenes. 910 contains various related information.

FIG. 13 shows an example of the scene structure 1000. 1002 is the representative frame number of the scene, 1004 is the start point, and 1006 is the end frame number. A pointer to the corresponding physical image enters 1008. In 1010, a data structure of all objects appearing in this scene, that is, pointers to objects are entered in a linked list format. Scenes can be grouped by the connection of their video contents, and can be managed hierarchically in a pyramid shape. The upper scene 1012 is a pointer to such a higher scene, and the lower scene 1014 is a pointer to a linked list of all scenes one level lower. Reference numeral 1016 denotes scene attribute information. The physical video calling unit 302 determines the physical video to be called from the physical video library 308 and the frame position to be reproduced based on the information 312 obtained by adding the scene information to the frame number 300.

FIG. 14 shows an example of the physical video structure 600. 602 is an ID number for uniquely specifying a physical video. Reference numeral 604 denotes a classification code for identifying whether the image is a laser disk image, a video tape image, or data stored in an external information storage device. 606 is a representative frame number, 608 is a start point, and 610 is an end point frame number. 616 contains attribute information. The other information is necessary when the video data is included in the physical video data structure. 612 is the screen width of the video, 614 is the same height, and 618 is a directory that stores the address of the physical video from which the frame image data corresponding to a certain frame number exists. A frame number 620, frame pixel data 622, and audio data 624 are repeated by the number of frames. If the physical image retrieving unit determines from the classification code that the image is a video using the video reproducing device 10 such as a laser disk, the physical video retrieving unit sends a control command to the video reproducing device to perform a process of calling the relevant video, and the physical video is included in the physical video. In that case, call the corresponding video.

One of the merits of using logical video is that from one physical video that tends to have a large amount of data, a wide variety of video works edited in various ways can be created with a small amount of data using the video. In particular, the advantage of using a logical image is greater for an image that frequently uses past material images such as news. Another advantage is that by storing objects appearing for each scene in advance, it is not necessary to check which objects appear during video reproduction for all objects, and rapid processing can be expected.

The execution procedure of the associative search interface will be described in detail with reference to the computer screen example of FIG. 1 briefly described above. An arbitrary image is displayed on the monitor window 1100 by the image reproduction display unit 100 described above. Along with the display, sound is also output from the speaker 12. A cursor 1104 moves on the screen to perform a point operation in accordance with an indirect operation of the pointing device 5 such as a mouse or a joystick. The same point operation can be performed by a direct pointing device 13 such as a touch panel. In this case, a cursor can be omitted. The above-mentioned point position detection unit 104 constantly monitors these pointing devices, and when the cursor 1104 is moved in accordance with the movement of the mouse, or when the mouse button is pressed, it is determined that a point operation has been performed, and The position information of the cursor on the screen is sent to each processing module that needs the position information. In the case of a touch panel, when a touch is made, the touched position is detected and the position information is sent. Reference numeral 1102 denotes an operation panel for controlling the playback state of the video. The operation panel unit 110 has buttons on which pictures and characters indicating the playback state, such as playback and fast forward, are drawn, and buttons for changing the mode. A display area or the like for displaying various information from the video reproduction display unit is displayed. When the point position detection unit 104 notifies that the display area of the operation panel has been pointed, which button is pointed is further detected from the position information, and the control code associated with the button is used for video playback. It is sent to the display unit 100. Reference numeral 1106 denotes a general-purpose input / output window through which various information can be exchanged with the computer using the keyboard 11 or the like. By inputting a file name, a video to be subjected to associative search can be specified from this window. The input file name is sent to the video reproduction display unit 100 as reproduction position setting information 310 together with the number of the first frame indicating the reproduction start position, and the logical video calling unit 300 in 100 responds from the information. The image is called, and the image is displayed on the monitor window 1100 via the physical image calling unit. Further, various related information of the video can be displayed in the general-purpose input / output window 1106.

When the point position detection unit detects that one of the icons 1110 displayed in the window 1108 has been pointed, the index management unit 108 uses the first frame number of the scene corresponding to the icon as the reproduction position setting information to reproduce the image. Tell the display unit 100. 100 displays an image of the scene on a monitor window 1100. The displayed video can be controlled by the operation panel 1102 such as reproduction and fast forward. When the reproduction of the video is started, the reproduction position information 314 output from the logical video calling unit 300 is transmitted to the index management unit 108. In the window 1108, for example, the icon of the scene being reproduced is displayed. Highlighting such as highlighting and blinking is performed so that the scene corresponding to the video currently reproduced in the monitor window 1100 can be understood at a glance.

The display of the scene in # 1108 can be performed hierarchically. First, two types of points, for example, a click and a double click, are prepared, and the click is used as a point means for calling the above-mentioned image, and the double click is used as a point means for hierarchical management of a scene described later. When the point position detection unit detects that one of the icons displayed in 1108 is pointed, the index management unit 108 checks whether or not the icon is double-clicked. If it is not a double click, the above-mentioned video call processing is performed. If it is a double click, a window similar to 1108 is newly created by referring to the lower scene 1014 in the scene structure 1000 corresponding to the pointed scene. Then, a list of icons of the lower scenes is displayed. The newly created window becomes a target of point detection similarly to 1108. When an icon on this window is pointed, the index management unit displays the corresponding scene on the monitor window or further lower level. If there are scenes, a new window for displaying a list of lower-level scenes is created. Such hierarchical management can also be used when selecting a video, and if one video is associated with one top-level scene that bundles all of the scenes, it falls within the scope of the above framework. Thus, a desired video can be selected from the registered video from the window, and a list of lower-level scenes can be displayed.

An icon 1114 is composed of an icon 1114 and a time axis display unit 1116. The icon 1114 collects and classifies several objects classified based on criteria such as appearing in different scenes but actually being the same subject. Is displayed, and beside it, the section in which those things appear in the entire video is displayed as a bar graph with the horizontal axis representing the time axis. Objects of the same classification are managed by the object structure 500, and the upper object 516 has a pointer to a common object structure. Conversely, the upper object has a pointer to the object structure of each thing in the lower object 518 in a linked list format. The index management unit 108 stores and manages upper-level objects. What is displayed as the icon is a reduced image of the representative frame stored in the structure of the upper-level object. The bar graph examines each of the lower-level objects, and calculates and draws a section occupying the entire image from the start and end frame numbers. When it is detected that a portion corresponding to an appearance section of an object in the bar graph is pointed, the index management unit 108 displays an image of the portion on the monitor window 1100. If an object is selected by pointing to an icon and related information is given or changed, the object is registered as related information of a higher-order object, that is, as information common to all objects of the same classification.

On the other hand, when it is detected that the monitor window 1100 is pointed, the point object identification unit 102 detects which object in the video is pointed from the information of the point position. In this processing, the reproduction position information 314 indicating which scene is currently being reproduced is received from the logical video calling unit 300, and for each of the objects stored in the corresponding object 1010 of the scene structure corresponding to the scene. The start point and the end point are checked, and compared with the reproduction position information 316 indicating the currently reproduced frame number, to determine whether or not the object represented by the object is currently appearing on the screen. For each of the objects determined to be appearing, the present area of the current object is obtained from the position of the object, that is, the object position 514 and the reproduction position information 316, and whether the pointed position is included in the area. Determine whether If more than one match, only one with higher priority is selected. The priority can be expressed, for example, in the order of registration of the linked list. With this method, there is no need to prepare a special data area for priority. If there is a thing determined to be pointed, the object attribute information 520 in the object structure of the thing is checked to find a dictionary structure 800 having a key meaning "jump destination of associative search", and a link. The start frame number of the object registered in 810 is read, and jumping to that frame is performed. If there is no key corresponding to the object attribute information 520, the scene jumps to a scene in which another thing having a common upper object appears. This refers to the linked list of lower objects registered in the object one rank higher than the pointed thing, reads out the starting frame number of the next object linked to the thing, and jumps to that frame.

As described above, it is possible to hierarchically search for a scene and hit it, then check the video on the monitor window, perform an associative search, and check on the index window. This is achieved by introducing video management means using a logical video composed of scenes.

FIG. 15 shows a detailed screen example of the monitor window 1100. Reference numeral 1200 denotes an area in which a video is actually displayed, and reference numeral 1202 denotes a frame number being reproduced from the video reproduction unit 200. The part displaying the frame number also serves as a numerical value input part. When the number is corrected by using a keyboard or the like, the corrected number is regarded as a new frame number, and the video from the scene corresponding to that number is displayed. Can be played. Reference numeral 1204 denotes an indicator panel for displaying which part of the entire video is currently being reproduced. The reproduction position is indicated by the position on the panel where the indicator rod 1206 is located. The position of the pointer is calculated from the frame number and the structure data of the logical video being reproduced. The vertical bar 1208 is a line indicating a transition between scenes, and it is possible to intuitively know which scene is being reproduced. With this panel, the jump by the associative search can be clearly known by the large movement of the pointing bar 1206, and confusion such as indistinguishable that the scene has just changed naturally in the video is eliminated. When the pointing bar 1206 is pointed by the pointing bar 120 and is forcibly moved by a drag operation, the operation panel unit 110 uses the position information determined after the movement obtained by the point position detecting unit 104 to change the position to that position. The corresponding scene and frame number are calculated, and this information can be transmitted to the video control unit 106 so that the video portion corresponding to that position is reproduced. A button 1210 is used to close the monitor window.

FIG. 16 is an example of a video display screen when there is an object mapped to audio. Since the voice is invisible information, it is visualized in the form of buttons 1400 and 1402. The determination as to whether or not a voice is made can be made by the object presence / absence determining unit 202 examining the object classification code 504. 202 checks which object appears using the information of the currently reproduced scene and frame, and displays a button if the classification code of the appearing object is audio. The display position of the button is registered in the position 514 of the object. Thus, the user can jump to a scene related to the sound by pointing this button without changing the processing of the point object identification unit. The buttons are displayed only for the type of object mapped to the currently reproduced sound, and are distinguished by the title on the button surface.

(A) to (c) of FIG. 17 are examples of display screens when jumping to another scene by associative search. When an object on the screen is pointed, the video playback / display unit 100 makes a change with a special effect added so that it can be easily distinguished from a normal scene change in the video. For example, from the center of gravity of the area of the pointed object, the scene is changed such that a reduced image of the scene at the jump destination becomes larger. As a result, it is immediately apparent which object was pointed.

{Circle around (2)} 1212 is a button for determining whether or not to display the related information of the thing. When this button is pointed, a menu such as 1300 shown in FIG. 18 appears. In this menu, the type of related information that can be displayed at present is displayed in addition to OFF for not displaying the related information. The user can select the type of related information to be viewed from this menu. This information is transmitted as a control signal 210 to the related information display unit 204 of the video reproduction display unit 100 via the video control unit 106, and it is determined whether the relevant information is to be displayed and, if so, which key corresponds to the key. Is done. This menu is created for each video, and all the dictionaries of the dictionary of the object attribute information 520 in all the object structures 500 registered for the video are checked, and all types are listed in the menu. Reference numeral 1214 denotes a button for changing a mode, which can switch between an associative search mode and a mode for changing related information. As a result, the internal state of the point thing identification unit 102 is changed, and the corresponding processing when a point is transmitted from the point position detection unit is made to correspond to each internal state.

FIG. 19 shows an example of a screen for displaying related information. The related information is displayed so as to be superimposed on the thing so that the relationship between the thing 1500 and the related information 1502 in the video can be seen at a glance. When the object presence / absence determination unit 202 determines the current appearing object in the above-described procedure, the position 514 of the object is read out for those objects, the center of gravity is obtained from the position information, and it is necessary to display the related information. Then, the display position of the related information is determined such that the center of gravity of the region is obtained. However, when a plurality of objects are in close contact with each other, they are offset from each other so that the display of 1502 does not overlap. The related information 1502 is not limited to text as shown in the figure, and may be an image such as an icon. Also, at the time of the associative search, the point object identification unit 102 can identify that the corresponding object has been pointed by pointing to the display area of the related information 1502, and can jump to another scene. This is performed by giving two pieces of position information to one object and making an OR determination. In addition, as shown in FIG. 20, by connecting the related information 1502 and the thing 1500 with a connection line 1504, it is possible to display an easy-to-understand association. In particular, by fixing the display position of the related information 1502 and changing only the connection line in accordance with the movement of the object, even if the movement of the object is so severe that it is difficult to point to the object, the fixed 1502 is fixed. By pointing to, an associative search can be easily performed.

When the internal state of the system is in the related information change mode, as shown in FIG. 21, when the user points to the displayed related information text 1502, a character correction cursor 1506 appears, and the character is immediately changed on the spot using a keyboard or the like. be able to. If the displayed information is related information stored in a higher-order object, this change causes the related information to be simultaneously updated for all objects sharing the same higher-order object. When the related information other than the displayed information is changed, a related information change window 1600 as shown in FIG. 22 appears. Reference numeral 1602 denotes a list of keys of related information. In this list, in addition to the relevant information of the object, there is also the relevant information of the higher-level object. When the user points at the button 1604, a character input window appears. When a new key is entered, the character is registered and registered in the list 1602. Keys displayed in the list of 1602 can be selected by points, and when selected, they are highlighted. In this state, when something is input in the character input area 1608, it is registered as related information corresponding to the selected key. Reference numeral 1606 denotes a button for deleting a key. When the user points at 1606 while the key is selected, registration of the related information corresponding to the key is deleted. Reference numeral 1610 denotes a button to point when the change made in this way is accepted and completed, and reference numeral 1612 denotes a button to point to cancel and cancel all the changes.

と き に は Also, when the internal state of the system is in the object copy mode, it is possible to copy the object that appears in the video being played back and paste it into another video between the moving image and the audio. Copying is performed by duplicating the entire structure of the object of the pointed object. The copied object shares the higher-level object and is added as a lower-level object of the higher-level object. As for the pasting, since the thing in the video is associated with the subspace of the video information, it can be performed by replacing the subspace with the same shape of the pasted video information. In this copying and pasting, since the copying and pasting can be performed together with the related information, there is almost no work regarding the related information.

In the above embodiment, an example has been described in which a search is performed using a workstation-level computer, but the search may be implemented as a function such as a VTR or TV.

It is a structural example of the screen of the system which realizes the associative search of the video. FIG. 1 is a block diagram of an apparatus configuration of a video associative search system according to an embodiment of the present invention. It is explanatory drawing of the associative search function of an image. FIG. 4 is a diagram illustrating a subject search method. It is a processing block diagram for realizing associative search of a video. FIG. 3 is a schematic diagram of an object-oriented data structure. It is a detailed processing block diagram of a video reproduction display part. FIG. 4 is a diagram illustrating a structure that stores a video object. FIG. 3 is a diagram illustrating a structure that stores a position of an object. It is a figure showing the structure which stores a dictionary. It is a detailed processing block diagram of a video reproduction part. FIG. 3 is a diagram illustrating a structure that stores a logical image. FIG. 3 is a diagram illustrating a structure that stores a scene. FIG. 3 is a diagram illustrating a structure that stores a physical image. It is an example of a screen showing a monitor window. It is an example of a display screen of a monitor window. It is an example of a display screen of a monitor window. It is an example of a menu display. It is an example of a display screen of a monitor window. It is an example of a display screen of a monitor window. It is an example of a display screen of a monitor window. It is a figure showing the window for changing related information.

Explanation of reference numerals

Reference Signs List 1 display, 2 control signal line, 3 video input device, 4 computer, 5 pointing device, 6 external information storage device, 7 CPU, 8 connection interface, 9 memory, 10 video playback device , 11 ... keyboard, 12 ... speaker, 13 ... touch panel.

Claims (10)

Video display means having a video display area for viewing video, an index display area for displaying index information of the video, point detection means for detecting which of the display areas is pointed, and video The state of the video to be reproduced next from the object management means for pre-registering things or sounds in the moving image appearing in the video, and the point information obtained by the point detection means and the logical structure description of the video separately stored And an associative search device for a video, wherein the video associative search device comprises: 2. The video associative search device according to claim 1, further comprising an attribute information superimposing means, wherein at least a part of selected attribute information of the object appearing in the video being reproduced is converted into the information of the object in the reproduced video. A video associative search device characterized in that it is superimposed on a position or displayed in a form that clearly indicates that the object and its attribute information correspond to each other. 2. The video associative search device according to claim 1, further comprising an attribute information changing unit, wherein at least a part of the attribute information of the object appearing in the video being reproduced is changed immediately when the object appears. A video associative search device, characterized in that: 2. The video associative search device according to claim 1, wherein a video display area, an area for displaying a video playback position, and a video playback state are controlled as partial areas of the display screen in the video display means. Video associative search having an operation window having an area for displaying a button for displaying the attribute information and an area for displaying a button for determining whether or not to display attribute information and the type of display information apparatus. 2. The video associative search device according to claim 1, wherein when a scene changes due to an object point, the change is executed by adding a special video effect, so that the change is displayed separately from a normal scene change. Video associative search device. 2. The video associative search device according to claim 1, wherein when the attribute information of the object is displayed, it is determined that the object is pointed even when the display area of the attribute information of the object is pointed. Video associative search device. 2. The video associative search device according to claim 1, wherein the type of attribute information to be displayed is specified by displaying a list of types of attribute information in the video to be subjected to the associative search. Video associative search device. 7. The video associative search device according to claim 6, wherein the display position of the attribute information of the object is fixed, and the association with the object is a line that changes so as to always connect the position of the object and the display position of the attribute information. A video associative search device characterized by minute display. Video display means for displaying at least moving image and video indexes; audio output means for reproducing audio; input means for instructing a control state of video; video reproduction means; Video input means for converting the video into a data format that can be handled by a computer, a memory for storing the data obtained by the video input means, and a video input device based on the information input by the input means. An associative video search system having control means for controlling a display state. The state of the video to be reproduced next is determined by pre-registering the thing or sound in the moving image appearing in the video and detecting that the point on the moving image corresponding to the registered thing is pointed. A video associative search method, wherein a video is displayed on a video display means.

Images