JP3711993B2 - Video associative search device - Google Patents

Description

  The present invention relates to an apparatus and a method for associatively searching for an image and finding an arbitrary image.

  In recent years, with the increase in speed and capacity of computers, the construction of databases for video information such as movies and videos that could not be handled in the past has become active. Along with this, search technology is being put to practical use for efficiently selecting a desired scene from a large amount of stored video. As a method for such a search, a method in which a user designates a desired scene feature or keyword and a computer finds a scene to which a keyword matching the keyword is attached has become common in the field of video databases. Yes. However, it is very difficult not only for users who are unfamiliar with the search but also for the skilled person to specify the features of the scene accurately, and the desired search results are often not obtained.

The book, which is classical information, has a table of contents and an index as auxiliary information for searching. The table of contents is information in which keywords that symbolize a group of texts are listed in order of progress in the text. The index is information in which important keywords in the text are arranged and arranged in an easy-to-find order such as alphabetical order. The biggest feature common to both is that such keywords are listed. And it is always at the beginning or end of the book, so it doesn't take time to search. Readers can find passages in the text without thinking about keywords themselves by using the table of contents and index. Also, if you look at the table of contents, you can get an overview of the main text and see if the book is worth reading in a short time.

  Searching by table of contents or index has the problem that it is difficult to find the appropriate part if there are too many keywords displayed, and conversely, if there are too few keywords, there are no appropriate keywords in the first place. However, these problems can be solved by using hypertext and full text search together. That is, first, the number of items in the table of contents / index is limited to some extent and presented to the user. The user simply refers to the text using the second best keyword that seems to be related to the target part, and searches the text for a keyword that seems to be directly related to the intended part. Once found, the goal can be achieved by referring to the intended site using the hypertext mechanism. This is a technique that is routinely used for searching online manuals. Hypertext requires keywords to be registered in advance, but if you use full-text search, you can do the same with keywords that are not registered. In this way, the associative tracking of keywords expands the range of contents and indexes that can be used, and in many cases, the target site can be searched by simply selecting keywords that appear in front of you (see below). This is called associative search).

Such a mechanism is considered to be effective in video retrieval. In video, various things such as people and objects appearing in the video can be used as equivalent to the above keywords. As an elemental technology for realizing an associative search using this, for example, as a method of referring to related scenes and information from things in a video display screen, Japanese Patent Laid-Open No. 3-52070, “Related information reference method of moving image” There is. According to this, by providing means for storing the image section and position where each thing in the video appears and means for linking with the corresponding related information, a point on the screen on which each thing is displayed is provided by a mouse or the like. You can easily jump to related scenes by calling and call related information. Further, as means for saving the correspondence between each thing and its related information by using image processing, there is, for example, Japanese Patent Laid-Open No. 5-204990 by the inventors.

Japanese Patent Laid-Open No. 5-204990

  The prior art mentioned above is a means for associating each thing with its related information exclusively, and although the configuration of the entire search system and the usability on the user side have been sufficiently studied. Absent. In addition, there is a problem that it can only be traced associatively only for things that have been associated with related information in advance.

  An object of the present invention is to allow a user to find a desired scene while associatively tracing a memory by simply selecting and selecting from limited information presented by a computer when searching for a video. To provide an interface.

  A second object of the present invention is to provide means that can associatively trace things that have not been associated in advance.

  The present invention includes a video display area for displaying an arbitrary video on the screen, an operation panel area for controlling the playback state of the video, and an area for displaying index information corresponding to the video table of contents and index. And a means for detecting which of the display areas is pointed, and a means for determining the state of the video to be reproduced next from the point information and the video description information stored separately. Also, a means for grasping the displayed object and its position and displaying the related information of the object in a superimposed manner and a related information registration changing means are provided. Also, a means for registering and managing information necessary for these processes is provided.

  Furthermore, means for designating a specific thing appearing in the scene being displayed, means for extracting the feature amount of the thing, means for searching for another video scene having a feature amount that matches the feature amount, , Means for immediately jumping to the found video scene.

  According to the present invention, when searching for a desired scene, even if there is no information of an object directly related to the desired scene in the index information of the video, the object is traced one after another for associating the desired scene. Can reach the scene. In this way, organically combining index display and associative search greatly expands the range of indexes that can be used, and in many cases, the target scene can be obtained by simply selecting the information presented by the computer. It becomes possible to search. Therefore, it is not necessary to think or instruct an appropriate keyword or image feature that uniquely determines the target scene, and it is possible to search even with ambiguous memory, which is easy for beginners to understand. In addition, the related information superimposing means superimposes a part or all of the selected information related to the thing appearing in the video being played back at the position of the thing in the playback video, or the thing and its relation. Because it is displayed in a form that clearly shows that it corresponds to information, it is possible to immediately and accurately know information about an object that appears during an associative search without confusing what information it is. . In addition, by providing a related information registration changing means, it is possible to immediately register or change part or all of the related information of the thing appearing in the video being reproduced when the thing appears.

  In addition, even when an item currently displayed on the screen has not yet been given information for jumping to a related scene, a feature amount is extracted from the display screen and a feature amount is collated. By doing this, you can search and display another scene where the thing appears.

  According to the present invention, when searching for a desired scene, even if a scene where some clue related to the scene appears even if it cannot be specified from the index information, the scene where the clue appears is associatively searched. However, a multi-faceted video search using each display can be performed such that a desired scene is finally obtained. In addition, it is possible to know information about an object in the video being played back immediately and accurately without being confused as to what information it is. In addition, it is possible to immediately change a part or all of the related information of the thing appearing in the video being reproduced at the place where the thing appears. In addition, according to the monitor window of the present invention, it is possible to always monitor the position of the scene being played in all the images, and even if the scene jumps by associative search, it is combined with special effects such as wipe. This is clearly understood and is no longer confused with normal scene changes. Also, pointing the display area of related information displayed in a superimposed manner provides the same effect as pointing to an object, so that a convenient point method can be selected for each scene, improving operability. Also, by listing related information to be displayed, you can save time and effort to enter keys directly, and even if you forget a key, you can think of it by looking at the menu. As described above, according to the present invention, a user-friendly associative search can be realized.

  Hereinafter, an embodiment of the present invention will be described in detail.

  FIG. 2 is a schematic block diagram of a system configuration example for realizing the present invention. Reference numeral 1 denotes a display device such as a CRT, which displays an output screen of the computer 4. Reference numeral 12 denotes a speaker for reproducing sound. Commands to the computer 4 can be performed using an indirect pointing device 5 such as a mouse 5, a direct pointing device 13 such as a touch panel, or the keyboard 11. The video playback device 10 is a device for playing back video from an optical disk or a video deck. The video signal output from the video playback device 10 is sequentially converted into a format that can be handled by the computer 4 by the video input device 3 and sent to the computer 4. Inside the computer, the video data enters the memory 9 via the interface 8 and is processed by the CPU 7 in accordance with a program stored in the memory 9. Each frame of the video handled by 10 is numbered sequentially from the top of the video, for example, a frame number. By sending the frame number from the computer 4 to the video reproduction apparatus 10 through the control line 2, the video of the scene corresponding to the frame number is reproduced. Video data and various types of information can also be stored in the external information storage device 6. In addition to the program, the memory 9 stores various data created by the processing described below and is referred to as necessary.

  Below, the outline of the associative search system is explained first, and then the detailed execution procedure of each technology is explained.

  FIG. 1 shows a screen example of a system that realizes associative video search. 1 is a display device, 12 is a speaker for outputting sound, BGM, etc., 5 is an indirect pointing device such as a mouse or joystick, 11 is a keyboard, and 13 is a direct pointing device such as a touch panel. .

The monitor window 1100 in the display device 1 is a monitor screen, and has an operation panel 1102 of the same format as the VCR, so that the video can be freely reproduced and viewed. The video displayed on the monitor screen corresponds to “text” in “book”, and the panel (button) operation corresponds to “page turning”. The lower right window 1108 is a scene list display of representative images of each scene of the target video, and the right middle window 1112 is a list display of subjects appearing in the video. These list displays are collectively referred to as “index”. In the scene list display of the window 1108, typical frame images are selected from each scene in the video, reduced, and arranged in the list as icons 1110 in time order. These images can be thought of as equivalent to the “heading” of the scene.
It corresponds to "Contents". On the other hand, the subject is one of the important components of the scene, and in that sense, corresponds to a “keyword” in the text. Therefore, the list display of subjects in the window 1112 corresponds to “index”. When the icon 1110 in the scene list display is clicked with the mouse, the video on the monitor screen is switched and the scene indicated by the icon is reproduced. The subject list display includes an icon 1114 indicating what the subject is, and a time axis display portion (bar graph) 1116 on the right side thereof. In the time axis display part (bar graph), the left end is the time axis indicating the beginning of the video and the right end is the end, and the portion displayed as a bar indicates the time interval in which the subject appears. Clicking on the bar will display the video for that section on the monitor screen. Note that reference numeral 1104 denotes a cursor that moves in accordance with the movement of a pointing device such as a mouse, and reference numeral 1106 denotes a general-purpose input / output window that displays various types of related information of video.

  Next, the basic concept of associative search according to the present invention will be described with a simple example. Assume that a user wants to find a specific scene in which a subject B appears in a series of videos. If the icon of the target scene (the scene where subject B appears) or the subject B itself is luckily present in the scene list display or the subject list display of the representative image displayed in the index, click it directly, By replaying, the intended purpose is achieved. However, the video information is usually enormous, and the target scene is often not easily found (for example, if the time during which the subject B appears in the video is short, it is easy that the search cannot be easily performed). Will be understood). Therefore, the associative search of the present invention has an important meaning. That is, even if the target scene (subject B) cannot be searched directly, the user often has some knowledge about the target scene (subject B). It is to give. For example, if the user remembers that subject B and subject A appeared at the same time (there should be a scene), or thinks that the prediction that there is a high possibility that they will appear at the same time holds First, an attempt is made to search for the subject A.

  FIG. 3 shows an image of the video associative search function of the present invention. Three pictures (scenes 1 to 3) in the figure are illustrations each representing one scene in the video displayed on the monitor screen when an associative search is performed. For example, the user searches the index screen (subject icon 1114 of the window 1112) for one scene that includes the subject A that can be associated with the target subject B and displays it on the monitor screen. When the leftmost scene 1 is reproduced on the monitor screen of the monitor window 1100, when the subject A of the appearing subjects A and C is clicked with the mouse cursor, the subject A appears in the center in the figure. The screen switches to scene 2. If another subject B appearing together in this scene 2 is clicked, the scene 3 on the right side in the figure where B appears can be reached. If this scene is the target scene, the associative search ends.

  That is, when finding a specific scene in which subject B appears, based on the association that subject B appears at the same time as subject A, identification of subject B that is associatively the target scene through subject A registered in the index Can be traced to. At this time, troublesome operations such as thinking about keywords are unnecessary, and only the information appearing on the screen is viewed and selected.

  As will be described later, it is possible to perform a search using associations based on all multimedia information such as scenes, words, BGM, and subtitles, as well as associations between a plurality of subjects.

  The information necessary to realize such an associative search function basically includes (1) a video section where an object appears (appearance section), and (2) a position on the screen of the subject (appearance position). ), (3) Other video sections (link information) to be switched when clicked. These three pieces of information are handled as a set.

  Which subject is clicked during video playback is determined from the appearance section / appearance position information of (1) and (2), and the video switching destination is determined from the link information of (3) stored in the same set. The Here, the video is realized by continuously displaying still images called frames at a rate of 30 frames per second. If consecutive numbers called frame numbers are assigned to these frames in order from the top of the video, the appearance section of (1) can be expressed by the top frame number and the end frame number of that section. The appearance position in (2) is coordinate information indicating in which area of each frame in the section (1) the subject is shown. As the link information of (3), a link is provided so that different scenes in which the same subject appears can be visited one after another. In many cases, the same subject appears many times in one video, but this link makes it easy to call all scenes where the subject appears by simply clicking.

  The associative search method having the above-described configuration can be used only for a subject for which link information has already been set. However, among the three types of information necessary for the associative search mentioned above, the appearance section and the appearance position of the subject can be obtained by subject search technology such as Japanese Patent Application No. 4-261103 by the inventors. .

  An outline of the automatic subject search algorithm is shown in FIG. Basically, a combination of colors unique to the subject to be found is found in the frame. First, the user selects only one frame in which the subject appears from the video image as an example image, and extracts a characteristic color from the image. After that, the system divides all the frames in the video into small blocks one by one and searches for blocks containing the feature color. If there are more than a certain number of blocks including characteristic colors in one frame, it is determined that the subject is in that frame. The appearance position of the subject in the frame can be easily obtained by examining in which position in the frame the blocks including the characteristic color of the subject are distributed in the subject search process described above.

  However, this subject search method itself is based on the fact that an example image is presented to the system, and at least one section where the subject to be searched for needs to be found manually, which is often troublesome. However, in the case of the associative search as in the present invention, the subject on the monitor screen can be immediately used as an example image, so this subject search method can be used very effectively.

  Furthermore, if all the frames in the video are divided into blocks in advance, and a list of color types included in each block is stored in the storage device, the block division processing for each frame is unnecessary from the subject search. Become very fast. The speed is 100 times faster in real time than the performance of current workstations, and all subject appearance sections can be found in about 30 seconds from one hour video. If it is only necessary to find the nearest appearance section from the currently displayed video, it can be found in about a few seconds on average. Of course, the same list of colors stored in the storage device can be used regardless of the subject to be searched.

  Below, the execution procedure of the system which implement | achieves this invention is demonstrated using the block diagram of the software module performed by CPU7 according to the program stored in the memory 9. FIG. Each module described here can also be realized by hardware itself.

  FIG. 5 is an example of a processing block diagram for realizing scene associative search according to the present invention. Information on what position in the video the subject, etc. in the video, which is a clue for the associative search, appears at which time in the video (appearance section, appearance location), and related information and scene information that is the destination (link) Information) is stored in advance in the memory 9 or the external information storage device 6 in FIG. 2 in the form of a data structure called an object described later.

Here, information on things such as subjects in the video is managed in an object-oriented data structure created one by one for each appearance section. FIG. 6 is an explanatory diagram showing the concept. This is hereinafter referred to as a video object or simply an object. The video is divided into a moving image portion and an audio portion. The entire moving image can be expressed in a three-dimensional space consisting of an xy plane that forms a frame image and an axis of time t. It can be considered that subspace. A video object is defined as a data structure associated with this partial space on a one-to-one basis. (In other words, even for the same subject, as a rule, each video object is defined for each appearance section, and a link is provided between those video objects (between subjects).) Various information in video such as subtitles and scenes can be associated. Similarly, for audio information such as words and BGM, a video object can be defined as a data structure associated with an arbitrary partial section of an audio information space having a time axis. The link information is stored as a pointer that refers to the video objects. In this way, even if the corresponding media is different, such as between video and audio, by managing with a common data structure framework, it is possible to freely set a link between all information in the video.

  Now, returning to FIG. 5, the processing block diagram will be described in detail. The object management unit 120 is a module for managing these objects. The object management unit 120 performs processing for registering, changing, and deleting objects, and if there is a request from another module, takes out information 122 of the object that matches the indicated condition. , Present in that module. The video playback / display unit 100 performs video playback and display processing on the monitor window 1100 which is the monitor screen of the display device 1 in FIG. 1, and sets the playback position information 218 of the currently displayed video to 102 points. Send to the identification unit. The point position detection unit 104 constantly monitors the indirect pointing device 5 such as a mouse and the direct pointing device 13 such as a touch panel in FIG. 1, and position information 112 on a display screen on which the user performs a point operation. Is sent to the point object identification unit 102. In addition, the position information 112 is also sent to the index management unit 108 and the operation panel unit 110. The point thing identification unit 102 sends the reproduction position information 218 received from the video reproduction display unit 100 to the object management unit 120, and obtains information on all things registered as appearing at the reproduction position as objects. If there is a corresponding object, the position information of the thing is further acquired from them and collated with the position information 112 from the point position detecting unit 104 to identify which thing is pointed. The point thing identifying unit 102 sends information 114 about the identified thing to the video control unit 106. Based on the link information in the information 114 relating to the identified thing, the video control unit 106 performs processing such as jumping to another scene in which the thing appears, so that the control information 208 is sent to the video reproduction display unit 100. send. Further, as will be described later, control information 210 is sent to the video reproduction display unit 100 when related information of an object is displayed. The index management unit 108 stores representative frame images of registered videos as icons 1110 and displays them as a list of these icons in the window 1108. The index management unit 108 also stores the frame number together with the icon 1110. When the point position detection unit 104 detects that an icon is pointed, the scene corresponding to the icon is reproduced. The control information 116 is transmitted to the video control unit 106. In addition, information 124 indicating which thing has been pointed is obtained from the pointed thing identification unit 102, and a display is made so that the fact that the thing has been pointed can also be understood from the index. The index management unit 108 also manages the list display of subjects in the window 1112 of FIG. That is, the control information 116 is displayed so that the icon 1114 indicating the subject is displayed, the time axis display (bar graph display) of the subject is performed, and the video of the section is reproduced when the bar portion is clicked. The image is sent to the video controller 106. The operation panel unit 110 displays the operation panel 1102 in FIG. 1 representing various playback states such as playback, fast forward, and rewind. When the point position detection unit 104 detects that the operation panel is pointed, , Control information 118 is sent to the video control unit 106 so as to set the playback state corresponding to the pointed operation panel.

  FIG. 7 is an example of a processing block diagram showing the video reproduction / display unit 100 in more detail. The video reproduction unit 200 reproduces the video in response to an instruction such as which video is reproduced from where by the control information 208 transmitted from the video control unit 106. The reproduction position information 212 of the currently displayed video is sequentially sent to the thing presence / absence determination unit 202. 202 checks whether or not a pre-registered thing has appeared in the video of the playback position, and if there is, acquires position information 216 on the display screen of all the appearing things, The information is sent to the related information display unit 204. This position information 216 is the same as the position information acquired by the above-described point thing identifying unit 102, and can be sent to the point thing identifying unit 102 as the matter information 218 in order to avoid duplication of position information acquisition processing. The related information display unit 204 can display the related information of each item being reproduced on the screen. The control information 210 determines whether or not related information is displayed, and if it is displayed, which related information is displayed in what form. In particular, the position information 216 of an object can clearly indicate which position is being displayed. This display method will be described later. Depending on the display method, the image 214 is superimposed and combined, and the image is displayed on the image display unit 206.

  FIG. 8 shows an example of the data structure of the video object. Reference numeral 500 denotes a large frame of the data structure. Reference numeral 502 denotes an object ID number which is given a unique number for identification from other objects. Reference numeral 504 denotes a classification code indicating whether the object represents, for example, a person, a caption, or a sound. Reference numeral 506 denotes a pointer to a video in which the object appears. In this example, as will be described later, the video has a data structure divided into two layers of a physical video 600 and a logical video 900, and 506 is a pointer to the physical video. 510 is the frame number of the start point of the section in which the thing in the video represented by the object appears, and 512 is the frame number of the end point. Reference numeral 508 denotes a frame number of a video representing the thing, and is used as a picture of an icon representing the thing under an interface for visually handling the object. Reference numeral 514 denotes a pointer to the structure 700 for indicating the position on the screen of the thing represented by the object.

  FIG. 9 shows an example of the object position structure 700. This structure takes the form of a concatenated list that is created one by one for each sufficiently small interval where there is no movement of things or is small. Reference numeral 702 denotes the start frame number of the section without movement, and reference numeral 704 denotes the end frame number. Reference numerals 706 to 712 denote the origin coordinates and size of the rectangular area when the object is surrounded by the rectangular area. Reference numeral 516 denotes a pointer to a higher-order object having a higher abstraction level. Every object can have its own related information, but it may be convenient to share related information among several objects. For example, subjects such as people and objects in an image often appear in a plurality of scenes. Of course, the appearance and behavior when it appears differ from scene to scene, so there is related information unique to each scene, but it is better to share information with a high degree of abstraction such as name, gender, age, and occupation. The amount of data is small, and when information is updated, consistency is not broken. In this sense, such information with a high degree of abstraction is given to related information of a higher-level object, and has a data structure having a pointer 516 to the object. Reference numeral 518 denotes a pointer for referring to a lower object from an upper object. This is because the upper 500 object and the lower object use the same 500 data structures. Of course, since the information directly related to the image such as the start point / end point frame and the position information is not necessary for the upper object, a simplified structure without them can be used.

Reference numeral 520 denotes a pointer to the dictionary 800 for storing related information of things. As shown in FIG. 10, the dictionary includes a key 802 that is a pointer to a character string 804 that is a key for calling up related information, and a related information character string 808 that is registered in association with the key character string. It consists of a link 810 having contents 806 as pointers and pointers to related objects, and is formed in the number of items of related information to be registered, and takes a linked list form in which they are connected in a daisy chain. Reading related information of an object is performed by specifying the key and returning the contents of the dictionary structure that matches the key. For example, when the key is “name” and the content is “Taro”, the related information “Taro” can be obtained by specifying the key “name”. In the related information display unit 204, selection of which related information is displayed results in a process of displaying the content corresponding to which key. The link is a pointer to a jump destination object when performing an associative search, and the content 806 includes a character string or a symbol representing the meaning of the link such as “the same subject appearing in another scene”, for example. In the link destination 810, a pointer to the subject object is entered. When jumping by associative search, the video control unit 106 reads the video in which the subject appears and the first frame number from the structure of the object, and controls the video playback unit 200 to play back from the video position. .

  FIG. 11 is a more detailed processing block diagram of the video playback unit 200. The video has a two-layer structure of logical video and physical video. A logical video has only structural information as a collection of scenes, and a physical video has actual video data. The logical video calling unit 300 calls the matching logical video from the logical video library 304 from the reproduction position setting information 310 sent from the video control unit.

  FIG. 12 shows an example of a logical video data structure 900. Reference numeral 902 denotes an ID number that uniquely identifies a logical video. Reference numeral 904 denotes a scene number representing a logical image. Reference numeral 906 denotes a concatenation list representing the constituent scenes, which are consecutive in the order in which the scenes 1000 should be reproduced. Reference numeral 908 denotes setting information for special effects such as dissolve and wipe between scenes. In 910, various related information is entered.

  FIG. 13 shows an example of the scene structure 1000. Reference numeral 1002 denotes a scene representative frame number, reference numeral 1004 denotes a start point, and reference numeral 1006 denotes an end point frame number. A pointer to the corresponding physical image enters 1008. In 1010, data structures of all things appearing in this scene, that is, pointers to objects are entered in a linked list format. Scenes can be organized in units of video content connections, and can be managed in a pyramid-like hierarchy. The upper scene of 1012 is a pointer to such an upper scene, and the lower scene of 1014 is a pointer to a linked list of all the scenes one level lower. Reference numeral 1016 denotes scene attribute information. The physical video calling unit 302 determines the physical video to be called from the physical video library 308 and the frame position to be reproduced based on the information 312 in which the scene information is added to the frame number 300.

  FIG. 14 is an example of a physical video structure 600. Reference numeral 602 denotes an ID number that uniquely identifies a physical video. Reference numeral 604 denotes a classification code for identifying whether the image is a laser disk image, a video tape, or data stored in an external information storage device. 606 is a representative frame number, 608 is a start point, and 610 is an end point frame number. 616 contains attribute information. The other information is necessary when the video data is contained in a physical video data structure. Reference numeral 612 denotes a video screen width, reference numeral 614 denotes the same height, and reference numeral 618 denotes a directory that stores from which address of the physical video frame image data corresponding to a certain frame number exists. The format 620 is a frame number, 622 is a frame pixel data, and 624 is audio data. If the physical video calling unit knows that the video using the video playback device 10 such as a laser disk is obtained from the classification code, the physical video calling unit sends a control command to the video playback device to call the corresponding video and is in the physical video. If so, call the appropriate video.

  One of the merits of using logical video is that, from one physical video that tends to have a large amount of data, a variety of edited video works using that video can be created with a small amount of data. In particular, the advantage of using logical images is greater for images that frequently use past material images such as news. Another merit is that by storing objects that appear in each scene in advance, it is not necessary to check all objects to see what is appearing during video playback, and rapid processing can be expected.

The execution procedure of the associative search interface portion will be described in detail using the example of the computer screen of FIG. 1 briefly described above. An arbitrary video is displayed on the monitor window 1100 by the video reproduction display unit 100 described above. Along with the display, sound is also output from the speaker 12. Reference numeral 1104 denotes a cursor, which moves on the screen in accordance with the operation of an indirect pointing device 5 such as a mouse or a joystick, and performs a point operation. A similar point operation can also be performed by a direct pointing device 13 such as a touch panel, in which case the cursor can be made unnecessary. The above-described point position detection unit 104 constantly monitors these pointing devices, and when the cursor 1104 is moved in accordance with the movement of the mouse or when the mouse button is pressed, it is assumed that there is a point operation. The position information of the cursor on the screen is sent to each processing module that needs the position information. In the case of a touch panel, when a touch is made, the touched position is detected and the position information is sent. Reference numeral 1102 denotes an operation panel for controlling the playback state of the video. A button on which a picture or character indicating the playback state such as playback / fast-forward is drawn by the operation panel unit 110, a button for changing the mode, A display area for displaying various kinds of information from the video reproduction display unit is displayed. When the point position detection unit 104 informs that the display area of the operation panel has been pointed, it detects which button is pointed further from the position information, and the control code associated with the button reproduces the video. It is sent to the display unit 100. A general-purpose input / output window 1106 can exchange various information with the computer using the keyboard 11 or the like. By inputting the file name, the video for associative search can be specified from this window. The input file name is sent to the video playback / display unit 100 as playback position setting information 310 together with the number of the first frame indicating the playback start position, and the logical video calling unit 300 in 100 responds from the information. The video is called and the video is displayed on the monitor window 1100 via the physical video calling unit. Various related information of the video can be displayed on the general-purpose input / output window 1106.

  When the point position detection unit detects that one of the icons 1110 displayed in the window 1108 has been pointed, the index management unit 108 reproduces the video by using the first frame number of the scene corresponding to the icon as reproduction position setting information. Tell the display unit 100. 100 displays the video of the scene on the monitor window 1100. The displayed video can be controlled such as reproduction and fast-forward by the operation panel 1102. Thus, when the video playback is started, the playback position information 314 output from the logical video calling unit 300 is transmitted to the index management unit 108. In the window 1108, for example, the icon of the scene being played back is displayed. Highlighting such as highlighting or blinking is performed so that the scene corresponding to the video currently reproduced in the monitor window 1100 can be seen at a glance.

  The scene display in 1108 can be performed hierarchically. First, two types of points are prepared, for example, click and double click, and the click is used as a point means for calling the above-mentioned video, and double click is used as a point means for managing the hierarchy of a scene to be described later. When the point position detection unit detects that one of the icons displayed in 1108 has been pointed, the index management unit 108 checks whether it is a double click. If it is not double-clicked, the above-mentioned video call processing is performed. If it is double-clicked, a lower window 1014 in the scene structure 1000 corresponding to the pointed scene is referred to, and a window similar to 1108 is newly created. Then, the icons of the lower scenes are displayed in a list. The newly created window becomes a target for detecting points in the same manner as 1108. When the icon on this window is pointed, the index management unit displays the corresponding scene on the monitor window or lower level. If there is a scene, a new window for displaying a list of the subordinate scenes is created. Such hierarchical management can also be used for video selection, and if one top-level scene that bundles all the scenes is associated with each video, the category of the above-mentioned framework. Thus, it is possible to select a desired video from the registered video from the window and display a list of lower scenes.

  Reference numeral 1112 includes an icon 1114 and a time axis display unit 1116. For example, a single icon 1114 that summarizes several things classified by criteria such as appearing in different scenes but actually being the same subject. This is an index that displays a bar graph with the horizontal axis as the time axis, and the section where those things appear in the entire video. Things of the same classification are managed by the object structure 500, and the upper object 516 has a pointer to the common object structure. Conversely, the upper object has a pointer to the object structure of each thing in the lower object 518 in a linked list format. The index management unit 108 stores and manages higher-order objects. What is displayed as an icon is a reduced image of the representative frame stored in the structure of the upper object. The bar graph examines each of the subordinate objects, and calculates and draws the section that occupies the entire video from the start point and end point frame numbers. When it is detected that the part corresponding to the appearance section of the object in the bar graph is pointed, the index management unit 108 displays the video of that part on the monitor window 1100. If an object is selected by pointing to an icon and related information is given / changed, it is registered as related information of a higher-level object, that is, information common to all things of the same classification.

  On the other hand, when it is detected that the monitor window 1100 has been pointed, the point thing identifying unit 102 detects which thing in the video has been pointed from the information on the point position. This processing receives reproduction position information 314 indicating which scene is currently being reproduced from the logical video calling unit 300, and for each object stored in the corresponding object 1010 of the scene structure corresponding to the scene. The start and end points are checked and compared with the reproduction position information 316 indicating the frame number currently being reproduced, and it is determined whether or not the thing represented by the object is currently appearing on the screen. For each of the things that are determined to appear, the current area of the current thing is obtained from the position of the thing, that is, the object position 514 and the reproduction position information 316, and whether the pointed position is included in it. Determine if. If there are multiple matches, only one with the highest priority is selected. The priority can be expressed, for example, in the order of registration in the linked list. With this method, there is no need to prepare a special data area for priority. When there is an object determined to be pointed, the object attribute information 520 in the object structure of the object is checked, and a dictionary structure 800 having a key meaning “jump destination of associative search” is searched for and linked. The start point frame number of the object registered in 810 is read and jumped to that frame. When there is no corresponding key in the object attribute information 520, a jump is made to a scene in which another thing having a common upper object appears. This refers to the connection list of the lower objects registered in the object one rank higher than the pointed thing, reads the start frame number of the next object connected to the thing, and jumps to that frame.

  As described above, it is possible to search for scenes hierarchically, hit the screen, check the video in the monitor window, perform associative search, and check in the index window. This is achieved by introducing video management means using logical video composed of scenes.

  FIG. 15 shows a detailed screen example of the monitor window 1100. Reference numeral 1200 denotes an area where an image is actually displayed. Reference numeral 1202 denotes a frame number being reproduced sent from the image reproducing unit 200. The part displaying the frame number also serves as a numerical value input part. When the number is corrected using a keyboard, the corrected number is regarded as a new frame number, and the video from the scene corresponding to that number is displayed. Can be played. Reference numeral 1204 denotes an indicator panel for displaying which part is currently being reproduced in the entire video. The playback position is indicated by the position on the panel where the pointing bar 1206 is located. The position of the pointer is calculated from the frame number and the structure data of the logical image being reproduced. The vertical bar 1208 is a line that represents the transition of the scene, which makes it possible to intuitively know which scene is being reproduced. By this panel, it is possible to clearly know that the jump has been made by the associative search by the large movement of the pointer 1206, and the confusion that it is impossible to distinguish whether the scene has naturally changed in the video is eliminated. When the point position detection unit is pointed to the pointing rod 1206 and is forcibly moved by a drag operation, the operation panel unit 110 uses the position information determined by the point position detection unit 104 and determined after the movement to move to the position. The corresponding scene and frame number are calculated, and this information can be communicated to the video control unit 106 so as to reproduce from the video portion corresponding to the position. Reference numeral 1210 denotes a button for closing the monitor window.

  FIG. 16 is an example of a video display screen when there is an object mapped to audio. Since the voice is invisible information, it is visualized in the form of buttons 1400 and 1402. Whether or not it is a sound can be determined by the object presence / absence determination unit 202 by examining the object classification code 504. When the information of the currently reproduced scene and frame is used to check which object appears, 202 displays a button if the classification code of the appearing object is audio. The display position of the button is registered at the object position 514. As a result, it is possible to jump to a scene related to the sound by pointing to this button without changing the processing of the point object identifying unit. Buttons are displayed only for the type of object mapped to the currently playing audio, and are distinguished by the title of the button face.

  17A to 17C show display screen examples when jumping to another scene by associative search. When an item on the screen is pointed, the video playback / display unit 100 makes a change with a special effect so that it can be easily distinguished from the transition of a normal scene in the video. For example, the scene is changed so that the reduced image of the destination scene becomes larger from the center of gravity of the area of the pointed object. This makes it easy to see what was pointed to.

  By the way, 1212 in FIG. 15 is a button for deciding whether or not to display related information of things. When this button is pointed, a menu such as 1300 shown in FIG. 18 appears. In this menu, in addition to turning off the display of related information, the type of related information that can be displayed is displayed. The user can select the type of related information he wants to see from this menu. This information is transmitted to the related information display unit 204 of the video reproduction display unit 100 as a control signal 210 through the video control unit 106, and it is determined whether the related information is displayed or, if so, which key corresponds to the information. Is done. This menu is created for each video, and all the dictionary keys of the object attribute information 520 in all object structures 500 registered for the video are checked, and all types are listed on the menu. Reference numeral 1214 denotes a button for changing the mode, and can switch an associative search mode, a mode for changing related information, and the like. As a result, the internal state of the point thing identification unit 102 is changed, and the corresponding processing when a point is transmitted from the point position detection unit is made in accordance with each internal state.

  FIG. 19 is an example of a screen that displays related information. The related information is displayed so as to be superimposed on the thing so that the relationship between the thing 1500 in the video and the related information 1502 can be understood at a glance. When the thing presence / absence determination unit 202 determines the things currently appearing in the above-described procedure, the object position 514 is read for those things, the center of gravity is obtained from the position information, and necessary for displaying related information. The centroid of the area to be obtained is determined, and the display position of the related information is determined so that the centroids coincide. However, when a plurality of things are in close contact with each other, an offset is applied to each other so that the display of 1502 does not overlap. The related information 1502 is not limited to text as shown in the figure, and may be an image such as an icon. Further, at the time of associative search, the pointed thing identification unit 102 can identify that the corresponding thing is pointed by pointing to the display area of the related information 1502, and can jump to another scene. This is done by giving two pieces of positional information for one thing and judging by the OR. In addition, as shown in FIG. 20, it is possible to display the association easily by connecting the related information 1502 and the thing 1500 with a connecting line 1504. In particular, by fixing the display position of the related information 1502 and changing only the connecting line in accordance with the movement of the object, even if the movement of the object is intense and it is difficult to point to the object, the fixed information 1502 is fixed. It is possible to easily perform an associative search by pointing to.

  When the internal state of the system is in the related information change mode, as shown in FIG. 21, when the related information text 1502 displayed is pointed, a character correction cursor 1506 appears, and it is immediately changed on the spot using a keyboard or the like. be able to. If the displayed information is related information stored in a higher-level object, the related information is updated for all things sharing the same higher-level object by this change. When related information other than the displayed information is changed, a related information change window 1600 as shown in FIG. 22 appears. Reference numeral 1602 denotes a list of related information keys. In this list, there is related information of the superordinate object in addition to related information of the thing. When the button 1604 is pointed, a character input window appears, and when a new key is input there, it is registered and registered in the list 1602. Keys displayed in the list 1602 can be selected by points and are highlighted when selected. In this state, if something is input to the character input area 1608, it is registered as related information corresponding to the selected key. Reference numeral 1606 denotes a button for deleting a key. When 1606 is pointed with a key selected, the associated information corresponding to the key is deleted. Reference numeral 1610 denotes a button which is pointed when the change made in this way is accepted and completed, and reference numeral 1612 denotes a button which is pointed out when all the changes are canceled and canceled.

  In addition, when the internal state of the system is the thing copy mode, it is possible to copy the thing that appeared in the video being played and paste it on another video between the moving images and between the sounds. Duplication is done by duplicating the object structure of the pointed object. The copied object shares the upper object and is added as a lower object of the upper object. As for pasting, since the thing in the video is associated with the partial space of the video information, it can be replaced with a partial space of the same shape of the pasted video information. Since this copying / pasting can be performed by copying and pasting related information together, there is almost no amount of work related to the related information.

  In the above-described embodiments, the example in which a search is performed using a workstation-level computer has been described. However, the search can be realized as one function such as a VTR or a TV.

It is a structural example of the screen of the system which implement | achieves the associative search of an image | video. 1 is a block diagram of an apparatus configuration of a video associative search system according to an embodiment of the present invention. It is explanatory drawing of the video associative search function. It is a figure explaining a to-be-photographed object search method. It is a process block diagram for implement | achieving the associative search of an image | video. It is a schematic diagram of an object-oriented data structure. It is a detailed processing block diagram of a video reproduction display unit. It is a figure which shows the structure which memorize | stores a video object. It is a figure which shows the structure which memorize | stores the position of an object. It is a figure which shows the structure which memorize | stores a dictionary. It is a detailed processing block diagram of a video reproduction unit. It is a figure which shows the structure which memorize | stores a logic image | video. It is a figure which shows the structure which memorize | stores a scene. It is a figure which shows the structure which memorize | stores a physical image | video. It is an example of a screen which shows a monitor window. It is an example of a display screen of a monitor window. It is an example of a display screen of a monitor window. It is an example of a menu display. It is an example of a display screen of a monitor window. It is an example of a display screen of a monitor window. It is an example of a display screen of a monitor window. It is a figure which shows the window for changing related information.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Display, 2 ... Control signal line, 3 ... Video input device, 4 ... Computer, 5 ... Pointing device, 6 ... External information storage device, 7 ... CPU, 8 ... Connection interface, 9 ... Memory, 10 ... Video reproducing device 11 ... Keyboard, 12 ... Speaker, 13 ... Touch panel.

Claims (7)

A video display unit having a video display area for viewing the video recorded in the recording means and an index display area for displaying video index information;
A storage unit that preliminarily divides all frames constituting the video and stores a first feature amount for each block; and a point indicating an object in the video frame displayed on the video display unit A point position detection unit for detecting
A point detector for detecting the object pointed to by the point position detector;
An extraction unit for extracting a second feature amount of the detected object ;
By comparing the first and second feature amounts, the video section searched from the recorded video is searched for the video section closest to the video displayed on the video display unit and displayed on the video display unit. A control unit that displays the searched video section instead of the video
A video associative search device, wherein the video displayed on the display unit is switched every time the point is detected. 2. The associative search device according to claim 1, wherein the storage unit stores a list of types of colors included for each of the blocks. A storage unit that stores object information including characteristics of an object included in a shot frame of a video section and link information indicating a position of a next video section when the object appears;
When the object information of the detected object is recorded, the control unit determines a video section to be reproduced next from the object information and the object pointed to by the point detection unit, and the video The associative search device according to claim 1 or 2, wherein the video section is displayed instead of the video displayed on the display unit.   The control unit further includes information superimposing means, and at least a part of the selected object information of the object appearing in the video being reproduced is superimposed on the position of the object in the reproduced video, or 4. The video associative search apparatus according to claim 3, wherein an object and its attribute information are displayed in a clearly specified form.   5. The control unit according to claim 3 or 4, further comprising information changing means for changing at least a part of the object information appearing in the video being reproduced when the object appears. The video associative search device described in 1. When the scene of the video displayed on the display unit changes,
By executing the change by adding special video effects, associative search device of a video according to claim 1乃optimum 5, characterized in that the displayed differently from normal changes in a scene.   6. When the object information is displayed on the display unit, it is determined that the object is pointed even when the object information display area of the object is pointed. The video associative search device described in 1.

Images