JPH07175816A - Video associative retrieving device and method - Google Patents

Abstract

PURPOSE:To acquire a desired scene while tracing associatively a memory by deciding the state of an image to be reproduced next based on the video description information which is stored separately from the point information. CONSTITUTION:When a feat that one of icons displayed in a window 1108 is pointed is detected by a point position detecting part, an index control part checks the presence or absence of a double click state. If the absence of the double click state is decided, the video call processing is carried out. If the presence of the double click state is confirmed, a new window similar to the window 1108 is produced and the icons of lower orders are shown in a list by referring to a lower order scene 1014 that is included in a scene structure 1000 corresponding to the pointed scene. Furthermore the index control part displays the corresponding scene in a monitor window or produces a new window to show scenes of lower orders, if detected, in a list when an icon is pointed in the window.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus and method for associatively retrieving an image and finding an arbitrary image.

[0002]

2. Description of the Related Art In recent years, against the backdrop of high-speed and large-capacity computers, construction of a database for image information such as movies and videos that cannot be handled in the past has become active. Along with this, a search technology for efficiently selecting a desired scene from a large amount of accumulated video is being put into practical use. As a method for such a search, the user specifies a desired scene feature or keyword,
The method by which a computer finds a scene with a keyword that matches it is common in the field of video databases. However, it is very difficult not only for a user who is unfamiliar with searching but also for an expert to accurately specify the characteristics of the scene, and the desired search result is often not obtained.

Books, which are classical information, have a table of contents and an index as auxiliary information for retrieval. The table of contents is information that lists keywords that symbolize a group of texts in the order of progression in the text. The index is information in which important keywords in the text are arranged in an order that is easy to find, such as alphabetical order. The greatest feature common to both is that such keywords are displayed in a list. And it is always decided that it is at the beginning or end of the book, and it does not take time to search. Readers can find a passage in the text by using the table of contents or index without having to think about keywords themselves. The table of contents also gives you an overview of the text, and you can quickly see if the book is worth reading.

For the search by the table of contents or the index, if too many keywords are displayed, it is difficult to find an appropriate part.
On the contrary, if there are few, there are no suitable keywords in the first place,
There is also a problem. However, these problems can be solved by using hypertext and full-text search together. That is, first, the number of items in the table of contents / index is limited to some extent and presented to the user. The user may refer to the text using the suboptimal keyword that is likely to be related to the target site, and search for a keyword that is likely to be directly related to the desired site in the text. If found
The purpose can be achieved by referring to the desired part using the hypertext mechanism. This is a technique that is routinely used for searching online manuals.
It is necessary for keywords to be registered in the hypertext in advance, but if full-text search is used, the same can be done for keywords that have not been registered. In this way, the mechanism for associatively tracking keywords expands the usable range of the table of contents and index, and in many cases, the target site can be searched by simply selecting the keywords that appear in front of the eyes (hereinafter, Called associative search).

It is considered that such a mechanism is also effective in image retrieval. In the video, various things such as people and things appearing in the video can be used as those corresponding to the above keywords. As an elemental technology for realizing an associative search using this, for example, as a method of referring to a related scene or information from an object on a video display screen, Japanese Patent Laid-Open No. Hei 3-52070, “Related information reference method for moving images” is disclosed. There is. According to this, by providing means for storing the image section and position where each object appears in the video and means for associating with the corresponding related information, a point on the screen where each object is displayed can be moved to a mouse or the like. You can easily jump to related scenes by pointing by
Relevant information can be called. Further, as a means for saving the correspondence between each thing and its related information by using image processing, there is, for example, Japanese Patent Laid-Open No. 5-204990 by the inventors.

[0006]

The above-mentioned prior art is a means exclusively for associating each object with its related information, and is related to the configuration of the retrieval system as a whole and the usability for the user. Is not well considered. In addition, there is a problem in that it is possible to associatively follow only things that have already been associated with related information.

An object of the present invention is to search a video image.
The purpose is to provide an interface that allows a user to find a desired scene while associatively tracing memory by simply selecting and selecting from the limited information presented by a computer.

A second object of the present invention is to provide means for associatively tracing things that are not associated in advance.

[0009]

The present invention corresponds to a video display area for displaying an arbitrary video on the screen, an operation panel area for controlling the playback state of the video, and a table of contents or index of the video. Area for displaying index information, and means for detecting which of these display areas has been pointed out, and this point information and video description information separately stored should be reproduced next. It has means for determining the state of the image. Further, a means for grasping the displayed object and its position, superimposing and displaying the related information of the object, and a related information registration changing means are provided. Also, a means for registering and managing the information necessary for these processes will be provided.

Further, means for designating a specific object appearing in the scene being displayed, means for extracting the feature amount of the object, and another video scene having a feature amount matching the feature amount are displayed. A means for finding and a means for immediately jumping to the found video scene are provided.

[0011]

According to the present invention, when a desired scene is searched for, even if there is no information of an object directly related to the desired scene in the index information of the video, some objects that are associated with the desired scene are successively displayed. You can reach the target scene while tracing. In this way, by organically combining index display and associative search, the usable range of the index is greatly expanded, and in many cases, the target scene can be obtained by simply selecting the information presented by the computer. You will be able to search. Therefore, it is not necessary to consider or instruct an appropriate keyword or image feature amount that uniquely determines the target scene, and it is possible to perform a search even in uncertain memory, which is easy for beginners to understand. Further, the related information superimposing means superimposes a part or all of the information selected from the related information of the thing appearing in the image being reproduced on the position of the object in the reproduced image, or the object and its relation. Since it is displayed in a form that clearly indicates that the information and the information correspond, it is possible to immediately and accurately know the information about the thing that appeared during the associative search without confusing which thing the information was. . Further, by providing the related information registration changing means, it is possible to immediately register or change a part or all of the related information of the thing appearing in the video being reproduced, on the spot where the thing appears.

In addition, the thing currently displayed on the screen is
Even when the information for jumping to the related scene is not added yet, the feature quantity of the thing is extracted from the display screen, and the means for collating the feature quantity is used to change another scene in which the thing appears. You can search and display on the spot.

[0013]

EXAMPLE An example of the present invention will be described in detail below.

FIG. 2 is a schematic block diagram of a system configuration example for realizing the present invention. A display device 1 such as a CRT displays an output screen of the computer 4. Reference numeral 12 is a speaker for reproducing voice. Instructions to the computer 4 can be performed using an indirect pointing device 5 such as a mouse 5, a direct pointing device 13 such as a touch panel, or the keyboard 11. The video reproduction device 10 is a device for reproducing a video on an optical disc, a video deck, or the like. The video signal output from the video playback device 10 is sequentially converted into a format format that can be handled by the computer 4 by the video input device 3 and sent to the computer 4. Inside the computer, the video data enters the memory 9 via the interface 8 and is processed by the CPU 7 according to the program stored in the memory 9. 1
A number, for example, a frame number, is sequentially assigned to each frame of the video handled by 0 from the beginning of the video. By sending the frame number from the computer 4 to the video playback device 10 via the control line 2, the video of the scene corresponding to the frame number is played back. The video data and various types of information can also be stored in the external information storage device 6. In addition to the program, the memory 9 stores various data created by the processing described below and is referred to as necessary.

In the following, an outline of the associative search system will be described first, and then detailed execution procedures of each technique will be described.

FIG. 1 shows an example of a screen of a system that realizes associative search of video. 1 is a display device, 12 is a speaker for outputting voice, BGM, etc., 5 is an indirect pointing device such as a mouse or joystick, 1
Reference numeral 1 is a keyboard, and 13 is a direct pointing device such as a touch panel.

A monitor window 1100 in the display device 1 is a monitor screen, and has an operation panel 1102 of the same format as a VCR so that a video can be freely reproduced and viewed. The image displayed on the monitor screen corresponds to "text" in "book", and panel (button) operation corresponds to "page turning". The lower right window 1108 displays a scene list of representative images of each scene of the target video,
The window 1112 in the middle right is a list display of subjects appearing in the video. Collectively, these list displays are
It is called "index". In the scene list display of the window 1108, typical frame images are selected from each scene in the video, reduced, and arranged in the list as icons 1110 in chronological order. These images can be thought of as being equivalent to the "heading" of the scene, and the scene list that arranges them in time series corresponds to the "table of contents" of the "book". On the other hand, the subject is one of the important constituent elements of the scene, and in that sense corresponds to the “keyword” in the text. Therefore, the list display of the subjects in the window 1112 corresponds to the “index”. When the icon 1110 in the scene list display is clicked with the mouse, the image on the monitor screen is switched and the scene indicated by the icon is reproduced. The subject list display includes an icon 1114 indicating what the subject is and a time axis display unit (bar graph) 1116 on the right side thereof. The time axis display (bar graph) is
The left end is the time axis indicating the beginning and the right end is the end, and the portion displayed as a bar indicates the time interval in which the subject appears. If you click on the bar,
The video of the section is displayed on the monitor screen. Incidentally, 1104
Is a cursor that moves in accordance with the movement of a pointing device such as a mouse, and the window 1106 is a general-purpose input / output window that displays various related information of an image.

Next, a basic example of the associative search according to the present invention will be described with a simple example. Suppose a user wants to find a specific scene in which a subject B appears in a series of videos. If the target scene (the scene where the subject B appears) or the icon of the subject B itself is lucky in the scene list display of the representative images displayed in the index or the subject list display, click it directly, The intended purpose is achieved by reproducing. However, the amount of video information is usually enormous, and it is often the case that the target scene cannot be easily found (for example, if the time during which the subject B appears in the video is short, it is easy to perform the search easily). As you can see). Therefore, the associative search of the present invention has an important meaning. That is, the user does not need to directly search for the target scene (subject B), but the user does not want to search for the target scene (subject B).
In many cases, the user has some knowledge of the above, and in the present invention, the knowledge is used to make a link named associative. For example, if the user remembers that subject B and subject A were appearing at the same time (there should have been a scene), or thinks that there is a high probability that they are likely to appear at the same time. , First, try to search the subject A.

FIG. 3 shows an image of the video associative search function of the present invention. The three pictures (scenes 1 to 3) in the figure are illustrations showing one scene in the video displayed on the monitor screen when the associative search is performed. For example, the user searches the index (subject icon 1114 of the window 1112) for one scene in which the subject A that can associate with the target subject B is found and displays it on the monitor screen. When the leftmost scene 1 is reproduced on the monitor screen of the monitor window 1100 and the subject A of the appearing subjects A and C is clicked with the mouse cursor, the subject A appears. The screen switches to scene 2 of. By clicking another subject B appearing together in this scene 2, it is possible to reach the scene 3 on the right side in the figure where B appears. If this scene is the target scene, the associative search ends.

That is, in the case of finding a specific scene in which the subject B appears, the subject B which is a target scene is associated with the subject A registered in the index based on the association that the subject B appears at the same time as the subject A. The specific scene of B can be traced. At this time, there is no need for troublesome operations such as thinking of keywords, and it suffices to select only by looking at the information that appears on the screen.

It should be noted that, as will be described later, the retrieval can be performed using not only associations between a plurality of subjects but associations based on all multimedia information of video such as scenes, words, BGM, and subtitles. is there.

The information required to realize such an associative search function is basically (1) a video section (appearing section) in which the subject appears, and (2) the position of the subject on the screen. (Appearance position) and (3) other video sections (link information) to be switched when clicked.
These three pieces of information are handled as a set.

Which subject was clicked during video reproduction is determined from the appearance section / appearance position information of (1) and (2), and the video switching destination is determined from the link information of (3) stored in the same set. Is determined. Here, the video is realized by continuously displaying still images called frames at a rate of 30 frames per second. If consecutive numbers called frame numbers are sequentially assigned to these frames from the beginning of the video, the appearance section of (1) can be expressed by the beginning frame number and the ending frame number of the section. The appearance position of (2) is coordinate information indicating in which area of each frame in the section (1) the subject is imaged. As the link information of (3), a link is provided so that different scenes in which the same subject appears can be visited one after another. The same subject often appears in a single image many times, but this link allows you to easily call up all the scenes in which that subject appears by simply clicking.

The associative search method with the above configuration can be used only for a subject for which link information has already been set. However, of the three types of information required for the associative search mentioned above, the appearance section and the appearance position of the subject are, for example,
It can be obtained by the subject automatic retrieval technique such as Japanese Patent Application No. 4-261033 by the inventors.

An outline of the subject automatic retrieval algorithm is shown in FIG.
Shown in. Basically, it is necessary to find out from the frame a color combination unique to the subject to be searched. First, the user selects only one frame in which the subject appears in the video image as an exemplary image, and extracts a characteristic color from the image. After that, the system divides every frame in the video into small blocks one by one, and searches for blocks containing characteristic colors. If there are more than a certain number of blocks containing characteristic colors in one frame,
It is determined that the subject exists in the frame. The appearance position of the subject in the frame can be easily obtained by investigating to which position in the frame the block including the characteristic color of the subject is distributed in the above-described subject search processing.

However, the subject retrieval method itself is
As a general rule, an example image is presented to the system, and it is necessary to manually find at least one section in which the subject to be searched appears, which is often troublesome. However, in the case of the associative search as in the present invention, since the subject on the monitor screen can be immediately used as an example image, this subject retrieval method can be used very effectively.

Furthermore, for all frames in the video,
If the block division is performed in advance and a list of color types included in each block is stored in the storage device, the block division processing for each frame is unnecessary from the above-described subject search, which is extremely fast. The speed is 100 times faster in real time even with the performance of current workstations, and all the subject appearance sections can be found in about 30 seconds from the image of 1 hour. If only one closest appearance section from the currently displayed image needs to be searched, it can be found in a few seconds on average. Of course, the same color list stored in the storage device can be used regardless of the subject to be searched.

In the following, the execution procedure of the system for implementing the present invention will be described in C according to the program stored in the memory 9.
This will be described with reference to the block diagram of the software module executed by the PU 7. Each module described here can also be realized by hardware itself.

FIG. 5 is an example of a processing block diagram for realizing a scene associative search according to the present invention. Information (appearance section, appearance position) at which position in the video, such as a subject in the video, which is a clue for the associative search, appears in the video, as well as related information and information about the scene to which the jump is made (link (Information) is stored in advance in the memory 9 or the external information storage device 6 in FIG. 2 in the form of a data structure called an object described later.

Here, information about objects such as a subject in a video is managed in an object-oriented data structure created one by one for each appearance section. Figure 6
Is an explanatory diagram showing the concept. Hereinafter, this is called a video object, or simply an object. The video is divided into a moving image part and an audio part. The entire moving image can be expressed in a three-dimensional space consisting of the xy plane forming the frame image and the axis of time t, and the appearance section and the appearance position of the subject are It can be considered to be the subspace. A video object is defined as a data structure that is in one-to-one correspondence with this subspace. (That is,
Even for the same subject, as a general rule, each appearance section is defined as each video object, and a link is provided between those video objects (subjects). )
The video object can be associated with various information in the video such as the subject, subtitles, and scenes. Similarly, for audio information such as words and BGM, a video object can be defined as a data structure that is associated with an arbitrary subsection of an audio information space having a time axis in a one-to-one correspondence. Then, the link information is stored as a pointer that mutually refers to the video objects. In this way, even if the corresponding media such as moving images and audio are different, by managing with a common data structure framework, it is possible to freely set links between all the information in the video.

Now, returning to FIG. 5, the processing block diagram will be described in detail. The object management unit 120 is a module that manages these objects, performs registration / change / deletion processing of the objects, and retrieves information 122 of the objects that match the indicated conditions if requested by another module. ， Present to that module. The video reproduction display unit 100 is a monitor window 1100 which is a monitor screen of the display device 1 of FIG.
First, the video is reproduced and displayed, and the reproduction position information 218 of the currently displayed video is sent to the point thing identifying unit 102. The point position detector 104 is shown in FIG.
Indirect pointing device 5 such as a mouse or a direct pointing device 13 such as a touch panel
Is constantly monitored, and the position information 112 on the display screen where the user has performed the point operation is sent to the point thing identifying unit 102. In addition, the position information 112 is also sent to the index management unit 108 and the operation panel unit 110. The point object identification unit 102 sends the reproduction position information 218 received from the video reproduction display unit 100 to the object management unit 120, and obtains information of all the objects registered as appearing at the reproduction position as objects. If there is a corresponding object, the position information of the thing is further acquired from them, and it is collated with the position information 112 from the point position detection unit 104 to identify which thing is pointed. The point thing identifying unit 102 sends information 114 regarding the identified thing to the video control unit 106. Since the video control unit 106 performs processing such as jumping to another scene in which the object appears, based on the link information in the information 114 regarding the identified object, the video reproduction display unit 100 displays the control information 208. send. Further, as will be described later, when displaying the related information of the thing, the control information 210 is sent to the video reproduction display unit 100. The index management unit 108 stores a representative frame image of the registered video as an icon 1110 and displays a list of those icons in the window 1108. The index management unit 108 stores the frame number together with the icon 1110, and when the point position detection unit 104 detects that the icon is pointed to, it reproduces the scene corresponding to the icon. The control information 116 is transmitted to the video control unit 106.
In addition, the information 124 indicating which thing is pointed is received from the point thing identifying unit 102, and a display is made such that the index shows that the thing is pointed. The index management unit 108 also manages the list display of subjects in the window 1112 of FIG. In other words, the icon 1114 indicating what the subject is is displayed, and the time axis display (bar graph display) of the subject is performed. It is sent to the video control unit 106.
The operation panel unit 110 displays the operation panel 1102 of FIG. 1 showing various reproduction states such as reproduction, fast forward, and rewind, and when the point position detection unit 104 detects that the operation panel is pointed. , Control information 11 to set the playback state corresponding to the pointed operation panel
8 is sent to the video control unit 106.

FIG. 7 is an example of a processing block diagram showing the video reproduction display unit 100 in more detail. The video reproduction unit 200 includes control information 208 sent from the video control unit 106.
The video is played according to instructions such as which video is played from where and how. The reproduction position information 212 of the currently displayed video is sequentially detected by the presence / absence determination unit 202.
Sent to. The 202 checks whether or not the pre-registered thing appears in the video of the reproduction position, and if there is, acquires the position information 216 on the display screen of all the appearing things, Related information display section 204
Send to. This position information 216 is the same as the position information acquired by the point thing identifying unit 102 described above, and can be sent to the point thing identifying unit 102 as thing information 218 in order to avoid duplication of the position information obtaining process. The related information display unit 204 can display the related information of each thing being reproduced on the screen. The control information 210 determines whether or not the related information is displayed, and if so, which related information is displayed in what form. Particularly, the position information 216 of the object can clearly indicate which position in the display corresponds to the information. This display method will be described later. Depending on the display method, the image 214 is subjected to the superimposing synthesis process, and the image is displayed on the image display unit 206.

FIG. 8 is an example showing the data structure of a video object. 500 is the outline of the data structure.
An object ID number 502 is given a unique number for distinguishing it from other objects. 504 is
It is a classification code indicating whether the object represents, for example, a person, a subtitle, or a voice. Fifty
Reference numeral 6 is a pointer to the video in which the object appears. In this example, the video is a physical video 6 as will be described later.
00 and logical video 900 have a data structure divided into two layers, and 506 is a pointer to a physical video. 5
10 is the frame number of the start point of the section in which the object in the video represented by the object appears, and 512 is the frame number of the end point. Reference numeral 508 denotes a frame number of a video representing the object, which is used as a pattern of an icon representing the object under the interface for visually handling the object. Reference numeral 514 is a pointer to the structure 700 for indicating the position of the object represented by the object on the screen.

FIG. 9 shows an example of the object position structure 700. This structure is in the form of a concatenated list in which one is created for each interval where there is no movement of objects or is small enough, and they are linked together. 702 is
The starting point frame number of the section in which there is no movement and 704 are the ending point frame numbers. Reference numerals 706 to 712 denote the origin coordinates and the size of the rectangular area when the object is enclosed by the rectangular area. Reference numeral 516 is a pointer to an upper object having a higher degree of abstraction. Every object can have its own associated information, but it may be convenient for some objects to share the associated information. For example,
As for subjects such as people and objects in the image, the same subject often appears in multiple scenes. Of course, since the appearance and behavior when it appears is different for each scene, there is related information unique to each scene, but it is a name, sex, age,
Sharing information with a high degree of abstraction, such as occupation, requires a smaller amount of data, and does not break the integrity even when the information is updated. In that sense, such information having a high degree of abstraction is given to related information of a higher-order object, and has a data structure having a pointer 516 to the object. Reference numeral 518 is a pointer for referencing a lower object from a higher object. This is because both the upper and lower objects use the same 500 data structures.
Of course, the upper-level objects do not require information directly related to the video such as the start / end frames and position information, so a simplified version of the structure without them can be used.

Reference numeral 520 is a pointer to the dictionary 800 that stores the related information of the thing. As shown in FIG. 10, the dictionary is a key 80 that is a pointer to a character string 804 that is a key for calling related information.
2 and a content 806 that is a pointer to a character string 808 of related information to be registered in association with the key character string, and a link 810 that has a pointer to a related object, and has only the number of items of related information to be registered. It takes the form of a linked list that is created and linked together. Object related information is read by specifying a key and returning the contents of the dictionary structure that matches the key. For example, when the key is “name” and the content is “Taro”, the related information “Taro” can be obtained by specifying the key “name”. In the related information display unit 204, the selection of which related information is to be displayed results in the process of displaying the content corresponding to which key. The link is a pointer to an object to jump to when performing an associative search, and the content 806 contains, for example, a character string or a symbol representing the meaning of the link, such as “the same subject appearing in another scene”. A pointer to the object of the subject is entered in the link destination 810. When jumping by associative search, the video control unit 106 reads the video in which the subject appears and the start frame number from the structure of this object, and controls the video playback unit 200 to play back from that video position. .

FIG. 11 is a more detailed processing block diagram of the video reproducing unit 200. The video has a two-layer structure of a logical video and a physical video. A logical video has only structural information as a set of scenes, and a physical video has actual data of the video. The logical video calling unit 300 calls the matching logical video from the logical video library 304 based on the reproduction position setting information 310 sent from the video control unit.

FIG. 12 shows an example of a logical video data structure 900. 902 is an ID that uniquely identifies the logical video
It is a number. Reference numeral 904 is the number of the scene that represents the logical image. 906 is a concatenation list representing constituent scenes.
The scenes 1000 are in the order in which they should be played.
Reference numeral 908 is setting information of special effects such as dissolve and wipe between scenes. Various related information is entered in 910.

FIG. 13 shows an example of the scene structure 1000. 1002 is the representative frame number of the scene, and 100
4 is a start point and 1006 is an end point frame number. A pointer to the corresponding physical image enters 1008. 1010
A data structure of all things appearing in this scene, that is, pointers to objects, is entered in the linked list format. For scenes, the connection of the video content can be collected in units, and hierarchical management in a pyramid shape can be performed. The upper scene of 1012 is a pointer to such an upper scene, and the lower scene of 1014 is a pointer to a concatenation list of all the scenes that are one step lower. Reference numeral 1016 is scene attribute information. The physical image calling unit 302 sets the frame number to 30
The physical image called from the physical image library 308 and the frame position to be reproduced are determined by the information 312 to which the scene information is added at 0.

FIG. 14 shows an example of the physical image structure 600. An ID number 602 uniquely identifies the physical image. Reference numeral 604 is a classification code for identifying whether the image is a laser disk image, a video tape image, or data stored in an external information storage device.
Reference numeral 606 is a representative frame number, 608 is a start point, and 610 is an end point frame number. Attribute information is entered in 616. The other is information required when the video data is contained in the physical video data structure. 612 is a screen width of the video, 614 is the same height, and 618 is a directory that stores from which address of the physical video the frame image data corresponding to a certain frame number exists.
A frame number 620, frame pixel data 622, and audio data 624 are repeated for the number of frames. If the physical video calling unit finds from the classification code that the video uses the video playback device 10 such as a laser disk, it sends a control command to the video playback device to call the corresponding video, and the physical video calling unit is in the physical video. In that case, call the corresponding video.

One of the merits of using a logical image is that one physical image, which tends to have a large amount of data, can be used to create a wide variety of edited video works with a small amount of data. is there. In particular, the more frequently the past material video such as news is reused, the more the advantage of using the logical video is. Another advantage is that by storing the objects that appear in each scene in advance, it is not necessary to check for all the objects what is appearing during video playback, and rapid processing can be expected.

The execution procedure of the interface portion of the associative search will be described in detail with reference to the computer screen example of FIG. 1 briefly described above. Monitor window 1100
, An arbitrary image is displayed by the image reproduction display unit 100 described above. Along with the display, voice is also output from the speaker 12. A cursor 1104 moves on the screen according to an indirect operation of the pointing device 5 such as a mouse or a joystick to perform a point operation. The same point operation can be performed by the direct pointing device 13 such as a touch panel, and in that case, the cursor can be omitted. The point position detection unit 104 described above
Constantly monitors these pointing devices, moves the cursor 1104 according to the movement of the mouse,
When the mouse button is pressed, it is assumed that there is a point operation, and the position information of the cursor on the screen at that time is sent to each processing module that requires the position information. In case of touch panel, when touched,
The touched position is detected and the position information is sent.
Reference numeral 1102 denotes an operation panel for controlling the reproduction state of the image. The operation panel unit 110 uses a button on which a picture or a character indicating the reproduction state such as reproduction / fast forward is drawn, and a button for changing the mode. A display area or the like for displaying various information from the video reproduction display unit is displayed. When the display area of the operation panel is pointed,
When transmitted from the point position detection unit 104, which button is pointed is further detected from the position information, and the control code associated with the button is sent to the video reproduction display unit 100. A general-purpose input / output window 1106 allows various kinds of information to be exchanged with a computer using the keyboard 11 or the like. By entering the file name, you can specify the video for associative search from this window. The input file name is sent to the video reproduction display unit 100 as reproduction position setting information 310 together with the number of the first frame indicating the reproduction start position, and the logical video calling unit 300 in 100 responds from the information. The video is called, and the video is displayed on the monitor window 1100 via the physical video calling unit. Also, various related information of the video can be displayed on the general-purpose input / output window 1106.

Icon 1 displayed in window 1108
When the point position detection unit detects that one of the 110 is pointed, the index management unit 108
The top frame number of the scene corresponding to the icon is transmitted to the video reproduction display unit 100 as reproduction position setting information.
100 displays the video of the scene on the monitor window 1100. The displayed image can be controlled by the operation panel 1102 such as reproduction and fast forward. When the video reproduction is started by this, the reproduction position information 314 output by the logical video calling unit 300 is changed to the index management unit 10
8 is displayed, and in the window 1108, for example, the icon of the scene being reproduced is highlighted or blinked, and the scene corresponding to the video currently being reproduced in the monitor window 1100 is displayed at a glance. To understand.

The scene display at 1108 can be performed hierarchically. First, two types of points are prepared, for example, click and double-click, and the click is used as the point means for calling the above-mentioned video, and the double-click is used as the point means for hierarchical management of the scene described later. When the point position detection unit detects that one of the icons displayed in 1108 is pointed, the index management unit 108 checks whether it is a double click. If it is not a double click, the above-mentioned video calling process is performed. If it is a double click, the lower scene 1014 in the scene structure 1000 corresponding to the pointed scene is referred to, and 1
A window similar to that of 108 is newly created and the icons of the subordinate scenes are displayed in a list. The window newly created in this way becomes a target for detecting a point similarly to 1108, and when the icon on this window is pointed, the index management unit displays the corresponding scene in the monitor window or further down. If there are scenes, create a new window that displays a list of those subordinate scenes. Such hierarchical management can also be used when selecting an image, and if one top-level scene that bundles all the scenes is associated with each image, the above-mentioned framework is included. Then, a desired image can be selected from the registered images from the window, and a list of subordinate scenes can be displayed.

Reference numeral 1112 is composed of an icon 1114 and a time axis display portion 1116, and is a representative of several things classified according to a criterion such as appearing in different scenes but actually the same subject. One icon 1114 is displayed, and next to it, a section in which those things appear in the entire image is displayed as a bar graph with the horizontal axis as the time axis. Objects of the same classification are managed by the object structure 500, and a pointer to a common object structure is used as the upper object 51.
Have 6 On the contrary, the upper object has a pointer to the object structure of each thing in the lower object 518 in the linked list format. The index management unit 108 stores and manages upper objects. What is displayed as an icon is a reduced image of the representative frame stored in the structure of the upper object. The bar graph examines each of the subordinate objects, calculates the section that occupies the entire image from the start and end frame numbers, and draws the section. When it is detected that the portion corresponding to the appearance section of the thing in this bar graph is pointed, the index management unit 108 causes the monitor window 1100 to display the image of the portion. If you point to the icon and select an object to add / change related information, it will be registered as related information for the higher-level object, that is, as information common to all things of the same classification.

On the other hand, when it is detected that the monitor window 1100 is pointed, the point thing identification section 102 detects which thing in the image is pointed to from the information of the point position. This processing is performed by the reproduction position information 3 indicating which scene is currently being reproduced.
14 is received from the logical video calling unit 300, the start point and the end point of each of the objects stored in the corresponding object 1010 of the scene structure corresponding to the scene are checked, and the reproduction position indicating the frame number currently being reproduced is displayed. It is compared with the information 316 to determine whether the object represented by the object is currently appearing on the screen.
For each object determined to appear, the current object existing area is obtained from the object position, that is, the object position 514 and the reproduction position information 316, and whether the pointed position is included in it. Determine whether When there are multiple matches, only the one with the highest priority is selected. The priority can be expressed in the order of registration of the connection list, for example. With this method, it is not necessary to prepare a special data area for priority. If there is an object determined to be pointed, the object attribute information 520 in the object structure of the object is checked,
A dictionary structure 800 having a key that means “jump destination of associative search” is searched for, the starting point frame number of the object registered in the link 810 is read, and the frame is jumped to. Object attribute information 520
If there is no key corresponding to, jump to a scene in which another object with a common upper object appears. This refers to the connection list of the lower objects registered in the object one rank higher than the pointed object, reads the starting point frame number of the next object connected to the object, and jumps to that frame.

As described above, it is possible to search for a scene hierarchically and hit it, then check the video in the monitor window, perform an associative search, and check it in the index window. This is achieved by introducing an image management means using a logical image composed of scenes.

FIG. 15 shows a detailed screen example of the monitor window 1100. Reference numeral 1200 denotes an area in which a video is actually displayed, and 1202 displays a frame number being reproduced which is sent from the video reproducing unit 200. The part displaying the frame number also serves as a numerical value input part, and when the number is corrected with a keyboard or the like, the corrected number is regarded as a new frame number, and the image from the scene corresponding to that number is displayed. Can be played. Reference numeral 1204 is an indicator panel for displaying which part of the entire image is currently being reproduced. The reproduction position is indicated by the position on the panel where the indicator rod 1206 is located. The position of the indicator rod is calculated from the frame number described above and the structure data of the logical image being reproduced. The vertical bar of 1208 is
It is a line that represents a scene transition, and this makes it possible to intuitively know which scene is being reproduced. With this panel, it is possible to clearly know that the jump has been made by the associative search by the large movement of the indicator rod 1206, and there is no confusion that it is impossible to distinguish whether the scene has changed naturally in the image. When the pointer 1206 is pointed at the point position detection unit and is forcibly moved by the drag operation, the operation panel unit 110 uses the position information determined by the point position detection unit 104, which has been determined after the movement, to move to that position. The corresponding scene and frame number are calculated and this information can be communicated to the video controller 106 so that it will be played back from the video portion corresponding to that position. 1210 is a button for closing this monitor window.

FIG. 16 is an example of a video display screen when there is an object mapped to voice. Since the voice is invisible information, the buttons 1400 and 140
It is visualized in the form of 2. Whether or not it is a voice can be determined by the thing presence / absence determining unit 202 by checking the object classification code 504. When checking which object is appearing by using the information of the scene and frame currently being reproduced, 202 displays a button if the classification code of the appearing object is voice. The display position of the button is registered in the position 514 of the object. As a result, by pointing to this button without changing the processing of the point thing identification part,
You can jump to the scene associated with that voice. Buttons are displayed only by the type of object mapped to the sound currently being played, and are distinguished by the title on the button surface.

FIGS. 17A to 17C are examples of display screens when jumping to another scene in the associative search. When an object on the screen is pointed, the video playback display unit 100
Changes with a special effect so that it can be easily distinguished from the transition of the normal scene in the video. For example, from the center of gravity of the area of the pointed object, the scene is changed so that the reduced image of the scene at the jump destination becomes large. This makes it easy to see which item was pointed.

By the way, 1212 in FIG. 15 is a button for deciding whether or not to display the related information of the thing. If you point this button, for example,
A menu like 1300 shown in appears. In this menu, the type of related information that can be displayed at present is displayed in addition to OFF for not displaying the related information. The user
From this menu, you can select the type of related information you want to see. This information is transmitted as a control signal 210 to the related information display unit 204 of the video reproduction display unit 100 through the video control unit 106, and determines whether the related information is displayed, and if so, which key corresponds to the information. To be done.
This menu is created for each video, and all object structures 500 registered for that video are displayed.
All the keys of the dictionary of the object attribute information 520 in are examined and all kinds are listed in the menu. 1
Reference numeral 214 denotes a button for changing the mode, which can switch the associative search mode, the mode for changing related information, and the like. As a result, the point thing identification unit 10
The internal state of 2 is changed so that the corresponding processing when the point is transmitted from the point position detection section corresponds to each internal state.

FIG. 19 is an example of a screen displaying related information. Things 1500 in the video and related information 1502
The related information is displayed so as to be superimposed on the thing so that the relationship with and can be seen at a glance. When the thing presence / absence determination unit 202 determines the presently appearing thing by the above-mentioned procedure, it reads out the position 514 of the object for these things, obtains the center of gravity from the position information, and is necessary for displaying related information. The center of gravity of the area is calculated, and the display position of the related information is determined so that the centers of gravity match. However, when a plurality of objects are in close contact with each other, offsets are applied to each other so that the displays of 1502 do not overlap. Related information 1
502 is not limited to the text as shown in the figure, and may be an image such as an icon. Further, at the time of associative search, the point thing identifying unit 102 can identify that the corresponding thing is pointed by pointing to the display area of the related information 1502, and jump to another scene. This is done by having two position information for one thing and judging by its OR. Further, as shown in FIG. 20, by connecting the related information 1502 and the thing 1500 with a connecting line 1504, an easy-to-understand display of the correspondence can be performed. In particular, by fixing the display position of the related information 1502 and changing only the connecting line according to the movement of the object, even if the movement of the object is difficult and it is difficult to point the object 1502. An associative search can be easily performed by pointing to.

When the internal state of the system is in the related information changing mode, as shown in FIG. 21, when the user points the displayed related information text 1502, the character correction cursor 1506 appears, and the character correction cursor 1506 appears on the spot. Can be changed immediately. If the displayed information is the related information stored in the higher-level object, this change causes the related information to be updated all at once for all things sharing the same higher-level object. If you want to change the related information other than the one shown,
A related information change window 1600 as shown in 2 appears. Reference numeral 1602 is a list of keys of related information. In this list, in addition to the related information of the thing, the related information of its superordinate objects is also included. When the user presses the button 1604, a character input window appears, and when a new key is input there, it is registered and registered in the list 1602. The keys displayed in the list 1602 can be selected by points and are highlighted when selected.
In this state, if something is input in the character input area of 1608, it will be registered as the relevant information corresponding to the selected key. Reference numeral 1606 denotes a button for deleting a key, and when the key is selected and the user points to 1606, the related information corresponding to the key is deleted.
Reference numeral 1610 is a button to point when accepting and completing the changes made in this way, and 1612 is a button to point when canceling all the changes and canceling them.

When the internal state of the system is the object copy mode, the object appearing in the image being reproduced is copied,
It is also possible to attach it to other images between video and audio. Copying is performed by duplicating the entire structure of the object of the pointed thing. The copied object shares the superordinate object and is added as a subordinate object of the superordinate object. Regarding pasting, since the objects in the video are associated with the subspace of the video information,
This can be done by replacing the subspace of the same shape of the pasted video information. Since this copying / pasting can be performed by copying / pasting the related information as well, there is almost no work for the related information.

In the above-described embodiments, an example in which a computer of the workstation level is used for searching has been described, but it is also possible to realize it as a function of a VTR or TV.

[0055]

According to the present invention, when a desired scene is searched for, even if it is not possible to specify it from the index information, as long as it is possible to find a scene in which some clue related to the scene appears, the scene in which the clue appears can be found. Multi-faceted video search using fusion of each display is possible, such that desired scene is finally obtained while associatively searching. In addition, it is possible to immediately and accurately know the information about the thing in the video being reproduced without confusing which thing the information is. In addition, some or all of the information related to the thing appearing in the video being played,
You can change it immediately when the thing appears. Further, according to the monitor window of the present invention, the position of the scene being reproduced in all the images can be constantly monitored, and even if the scene jumps in the associative search, it is combined with a special effect such as a wipe. That's why you can see it explicitly and don't get confused with the usual scene changes. Further, even if the user points the display area of the related information displayed in a superimposed manner, the same effect as that of pointing the object can be obtained, so that a convenient point method can be selected for each scene and the operability is improved. In addition, by displaying a list of related information to be displayed, it is possible to save the trouble of directly entering a key, and even when the key is forgotten, the menu can be recalled and reminded. As described above, according to the present invention, a user-friendly associative search can be realized.

[Brief description of drawings]

FIG. 1 is a configuration example of a screen of a system that realizes an associative search of video.

FIG. 2 is a block diagram of a device configuration of a video associative search system according to an embodiment of the present invention.

FIG. 3 is an explanatory diagram of a video associative search function.

FIG. 4 is a diagram illustrating a subject search method.

FIG. 5 is a processing block diagram for realizing a video associative search.

FIG. 6 is a schematic diagram of an object-oriented data structure.

FIG. 7 is a detailed processing block diagram of a video reproduction display unit.

FIG. 8 is a diagram showing a structure for storing a video object.

FIG. 9 is a diagram showing a structure for storing the position of an object.

FIG. 10 is a diagram showing a structure that stores a dictionary.

FIG. 11 is a detailed processing block diagram of a video reproducing unit.

FIG. 12 is a diagram showing a structure for storing a logical image.

FIG. 13 is a diagram showing a structure storing a scene.

FIG. 14 is a diagram showing a structure for storing a physical image.

FIG. 15 is an example of a screen showing a monitor window.

FIG. 16 is a display screen example of a monitor window.

FIG. 17 is a display screen example of a monitor window.

FIG. 18 is an example of a menu display.

FIG. 19 is a display screen example of a monitor window.

FIG. 20 is a display screen example of a monitor window.

FIG. 21 is a display screen example of a monitor window.

FIG. 22 is a diagram showing a window for changing related information.

[Explanation of symbols]

1 ... Display, 2 ... Control signal line, 3 ... Video input device, 4 ... Computer, 5 ... Pointing device,
6 ... External information storage device, 7 ... CPU, 8 ... Connection interface, 9 ... Memory, 10 ... Video reproducing device, 11 ... Keyboard, 12 ... Speaker, 13 ... Touch panel.

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Kazuaki Tanaka 5030 Totsuka-cho, Totsuka-ku, Yokohama City, Kanagawa Prefecture Hitachi, Ltd. Software Development Division

Claims (10)

[Claims] 1. A video display means having a video display area for viewing a video and an index display area for displaying index information of the video, and a point for detecting which one of the display areas is pointed. Detection means,
Based on the object management means for pre-registering the thing or sound in the moving image appearing in the video and the point information obtained by the point detection means and the logical structure description of the video stored separately, A video associative retrieval device, characterized in that it has a control means for determining a state and changes the displayed video. 2. The video associative retrieval apparatus according to claim 1, further comprising attribute information superimposing means, wherein at least a part of the selected attribute information of the objects appearing in the video being reproduced is reproduced. A video associative retrieval device characterized by being displayed in a superimposed manner on the position of the object in the inside, or by displaying in a form clearly showing that the object and its attribute information correspond. 3. The video associative search apparatus according to claim 1, further comprising attribute information changing means for displaying at least a part of attribute information of an object appearing in a video being reproduced,
An associative search device for video, characterized in that the object is changed immediately when it appears. 4. The video associative retrieval apparatus according to claim 1, wherein a video display area, a video display position display area, and a video display area are provided as partial areas of the display screen of the video display means. It is characterized by having an operation window having an area for displaying a button for controlling a reproduction state and an area for displaying a button for determining presence / absence of display of attribute information and a type of display information. Video associative search device. 5. The video associative retrieval apparatus according to claim 1, wherein when a scene changes depending on the point of an object, the change is executed by adding a special video effect to display the change separately from a normal scene change. A video associative search device characterized in that 6. The video associative retrieval apparatus according to claim 1, wherein when the attribute information of the object is displayed, it is determined that the object is also pointed when the display area of the attribute information of the object is pointed. A video associative search device characterized by: 7. The video associative search apparatus according to claim 1, wherein the type of attribute information to be displayed is specified by displaying a list of the types of attribute information in the video that is the target of the associative search. A video associative search device characterized by: 8. The video associative search device according to claim 6, wherein the display position of the attribute information of the object is fixed, and the correspondence with the object is always linked to the position of the object and the display position of the attribute information. A video associative search device characterized by clearly displaying by changing line segments. 9. A video display means for displaying at least a moving picture and a video index, an audio output means for reproducing audio, an input means for instructing a control state of the video, a video reproducing means, and the video. A video input means for converting the video obtained by the reproducing means into a data format that can be handled by a computer;
A video associative search system having a memory for storing data obtained by the video input means, and a control means for controlling a display state of the video based on information input by the input means. 10. An object or voice in a moving image appearing in a video is registered in advance, and by detecting that an area on the moving image corresponding to the registered object is pointed, the next reproduction should be performed. A video associative search method characterized in that a video state is determined and the video is displayed on a video display means.