JPH08235198A - Multimedia information management system - Google Patents

Abstract

PURPOSE: To record and retrieve multimedia information with high efficiency by enabling information management wherein a human's way of memorizing and a human's way of recalling are simulated. CONSTITUTION: This system is provided with a dividing means (episode extraction processing part 8) which groups multimedia information on the basis of the community of physical features of the multimedia information and a summarizing means (storage structure learning process 9) which groups the respective groups divided by the dividing means to high order on the basis of the relativity that the community of the respective groups has and further groups them repeatedly to higher order to put the groups in hierarchical structure, and makes a meaning network connecting languages on the basis of the meaning of language information correspond to the community information and relativity information of the respective groups in the hierarchical structure; and the community information and relativity information of the respective groups, and the language information are used as retrieval conditions of the multimedia information.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for recording and retrieving multimedia information composed of moving images, sounds, etc., and in particular, simulating a human memory method and a human memory tracing method, The present invention relates to a multimedia information management system suitable for efficiently recording and searching for.

[0002]

2. Description of the Related Art Most human activities are carried out based on the past experience and knowledge of the person or others. Documents are an excellent medium for transmitting such past experience and knowledge, but in recent years, audio and moving images have been recorded with the progress of technology in video cameras, VTRs (video tape recorders), and laser disk devices. It has become possible to record and transmit multimedia information including multimedia information. Furthermore, the speed of the electronic computer (computer) has increased, and it has become possible to have a large storage capacity. As a result, an environment has been established in which not only the accumulated video is simply sequentially reproduced like a VTR, but also a desired video can be randomly accessed and used. And
As the amount of stored information becomes huge, it is important to have a function of quickly searching for a desired image.

[0003] Conventionally, many techniques have been adopted to add a title, a date and time, and a keyword to accumulated documents and videos to facilitate retrieval. However, it is difficult for the person himself to add a key word that is easy to remember later. As a conventional technique for solving such a problem, there is, for example, a document information storage / retrieval system disclosed in Japanese Patent Publication No. 5-84538. In this technology, since it is possible to retrieve from ambiguous memory, a network showing the relation between concepts is constructed as a knowledge base, and document retrieval is realized by inference based on this knowledge base. To do. For example, you can search for a document recorded based on the information in "Workstation articles manufactured by a company in Saitama Prefecture" using the ambiguous search statement: "Computer articles made by electronics companies in Kanto." Becomes

However, when a person wants to remember a document that he or she saw in the past and the storage location of the document, he or she wants to remember it by the association from the situation when the document is seen rather than the contents of the document as described above. I often do it. That is, by remembering when, where, and under what circumstances, the contents and storage location of the document can be recalled in more detail. The above-mentioned conventional technique cannot perform such an episodic search.
Further, in the above-mentioned conventional technique, a search key attached to a document is used.
However, since the sentence has a structure from words, the search flexibility is improved, but the sentence of the search key is a result summarized by the subjectivity of the registrant. Therefore, when the searcher is different from the registrant, the viewpoint at the time of registration is different from the viewpoint at the time of search, and an incorrect search key may be entered. Can not.

As a device that enables an episodic search, there is a moving image management device disclosed in Japanese Patent Laid-Open No. 5-282379. In this device, moving images are divided according to physical changes and meanings, and they are sequentially hierarchically grouped in the time direction, and the hierarchical relationship is represented by a tree structure. For each node of the tree structure, a representative still image and attribute search key
As. A random image search is performed using this tree structure and search key. This technique allows temporal episode memory / retrieval. However, it does not take into account time-dependent summaries of activities or expressions of spatial episodes in activities. Further, the search based on the content of each node is fixed. Therefore, it is not sufficient to perform a search based on information that is easy for a human to remember (easy to remember) or information that is easy to remember.

Further, Japanese Patent Laid-Open No. 52070/1993 discloses that
By pointing one thing on the screen with a mouse or the like, a technique of calling up related information previously associated with the pointed thing is further disclosed in Japanese Patent Application No. 6-260.
No. 013 describes a technique by which this technique is improved so that a user can find a desired scene while associatively tracing memory even for information that is not associated in advance. In the technique described in Japanese Patent Application No. 6-260013, for example, even when the subject B to be searched is not registered as an index, the user is based on the association that the subject B appears at the same time as the subject A. Through the subject A registered in, the specific scene in which the subject B appears can be associatively traced. Furthermore, this technique is not limited to merely associations between a plurality of subjects, and retrieval can be performed using associations based on multimedia information in a video such as a scene itself, words, BGM, and subtitles.

However, the technique described in Japanese Patent Application No. Hei 6-260013 uses the knowledge that the user has about the object to be searched for association, and the user himself / herself searches for the index used for the association. I have to do it. If the user's knowledge is incorrect, the association cannot be performed correctly and the desired search cannot be obtained. In the technique described in Japanese Patent Application No. 6-260013, no consideration is given to a support technique for making such a user's association highly reliable. That is, as mentioned above,
No consideration is given to information that is easy for humans to remember (easy to remember) or to perform a search based on information that is easy to remember.

[0008] Human memories include "penguins are a type of bird" and knowledge such as "apples fall to the ground due to gravity" that are learned at school, and "I went to the zoo yesterday. There is a memory of personal experiences such as “Tata” and “Taro fell off the ladder”. Psychologist Tulving named the former "meaning memory" and the latter "episode memory." In the study of mnemonics, it is known that memorable and unforgettable semantic memory and episodic memory are related to each other.

As described above, in human memory, words and facts associated with a large amount of information such as known words, facts, and concepts are less likely to be forgotten and more likely to remain in the memory. By realizing such a memory in the machine, it becomes possible to perform a quick search based on the information that the user remembers or the information that is easy to remember. For this purpose, in addition to the semantic content of the video, it is necessary to be able to integrally manage episodic information including spatial information such as time and place, which is close to the image. is there. In addition, manual labeling of image information and the like tends to be subjective. For this reason, labeling and structuring based on real images are necessary. However, as described above, in the related art, no consideration is given to such information that a person can easily memorize (remember) or a search based on information that is easy to remember.

[0010]

The problem to be solved by the prior art is that in the prior art, it is possible to record and retrieve information that is easy for humans to remember (easy to remember) and multimedia information based on information that is easy to remember. This is a point that cannot be done. The object of the present invention is to solve these problems of the prior art,
An object of the present invention is to provide a multimedia information management system capable of highly efficient recording and retrieval of multimedia information.

[0011]

In order to achieve the above object, the multimedia information management system of the present invention comprises:
(1) Dividing means (episode extraction processing unit 8) that divides multimedia information into groups based on the commonality of physical characteristics of the multimedia information, and each group divided by this dividing means, the commonality of each group Meaning of summarizing to a higher-level group based on the relevance of each, and repeating the summarization of this higher-level group to further higher-level groups to generate groups in a hierarchical structure, and connecting languages based on the meaning of linguistic information Summarizing means (memory structure learning processing 9) for associating the network with the commonality information and the relevance information of each group of the hierarchical structure is provided, and the commonality information, the relevance information, and the language information of each group of the hierarchical structure are provided. It is characterized by being used as a search condition for multimedia information.

(2) In the multimedia information management system described in (1) above, means for displaying commonality information, relevance information and linguistic information of each group in the hierarchical structure (central processing unit 20, interface unit) 6, a display 3) is provided. Also, (3)
In the multimedia information management system as described in either (1) or (2) above, the dividing means (episode extraction processing unit 8) uses the multimedia information based on the commonality of objects in the moving image of the multimedia information. The feature is that the media information is divided into groups.

(4) In the multimedia information management system described in (3) above, the commonality information used by the dividing means (episode extraction processing section 8) is determined based on the movement of the camera capturing the moving image. Is characterized by. (5) In the multimedia information management system according to any one of (3) or (4) above,
The positional relationship between the groups is obtained based on the movement of the camera that captured the moving image, and the positional relationship between the groups is used as relevance information used by the summarizing means (memory structure learning processing 9). .

(6) In the multimedia information management system according to any one of (1) to (5), the spatial positional relationship between a plurality of things in the multimedia information is determined by group commonality information. It is characterized by using as. (7) In the multimedia information management system according to any one of (1) to (6), commonality information and relevance information are formed by a pattern,
The feature is that the commonality and the relevance are discriminated by pattern recognition. Also, (8) above (1) to (7)
In the multimedia information management system described in any one of (1) to (5), the commonality information and the relevance information are formed by a language including symbols, and the commonality and the relevance are determined by semantic analysis.

(9) In the multimedia information management system according to any one of (1) to (8), the groups are associated with each other in a time-direction context, and the association information is searched for the multimedia information. It is characterized in that it is used as a condition. (10) In the multimedia information management system according to any one of (1) to (9), the groups are associated with each other in a spatial positional relationship, and the association information is used as a search condition for multimedia information. It is characterized by (11) In the multimedia information management system according to any one of (1) to (10), the hierarchical grouping is corrected based on an instruction input from an operator.

[0016]

In the present invention, as in the case of human memory, the information about the meaning is associated with the summary result of the multimedia information such as images and sounds which is the experience of the individual and used for the retrieval of the multimedia information. , It is possible to provide information that humans can easily remember as a search condition. The memory of experience in human episodic memory has information similar to temporal and spatial images, but it is not a strict image like photographs and records. In addition, for example, the captured moving image data has different sections that are semantically recognized as a unit depending on the viewpoint.

In the present invention, for example, the photographed moving image data is divided (grouped) based on the feature extraction result from the image data. This makes it independent of subjectivity,
It is possible to objectively detect a group of sections having the same meaning. Moving image data further divided into one unit (no
As a function), hierarchical summarization (combining) is performed based on the physical characteristics of moving image data and the semantic commonality (relevance) between the nodes. As a result, not only concrete objects such as people and things but also abstract concepts such as activities (for example, meetings and moving sections) can be grouped together under the same framework, and multiple viewpoints can be realized. A tailored summary is possible. Furthermore, in each node, as additional information,
By providing the objects in the image and the positional relationship in space between the objects, the node can be provided with information close to an image including spatial information. In addition,
Correspondence with semantic memory can be achieved by adding symbols or words that characterize the words. Further, by storing the pattern characterizing the node and the recognition procedure, respectively, the node can be identified even at the pattern level.

[0018]

Embodiments of the present invention will now be described in detail with reference to the drawings. FIG. 1 is a block diagram showing an embodiment of the configuration of the multimedia information management system of the present invention according to the present invention, and FIG. 2 is an example of the configuration of each device used in the operation of the multimedia information management system in FIG. It is a block diagram shown. Each device in FIG. 2 is provided for recording and searching video images of daily activities in an office, and is a central processing unit (indicated as CPU in the drawing).
20, a magnetic disk device (indicated as HDD in the figure) 10,
A memory device (indicated as memory in the figure) 30, a control unit (indicated as CONT in the figure) 40 that AD-converts moving images and sounds from the camera 1 and the microphone 2 and digitizes them, and then displays. (Indicated as CRT in the figure) 3, keyboard (indicated as KEY in the figure) 4, and interface unit 6 (indicated as IF in the figure) 6 for controlling input / output of information from the mouse 5.

The magnetic disk device 10 is a meaning storage device (described as a language in the drawing) for storing semantic information (language information).
101, an episode storage device (described as EPS in the drawing) 102 for storing episodic information, a camera 1 and a microphone 2
It is composed of a multimedia information storage device (indicated as DATA in the figure) 103 for storing moving images and sounds from the.
Here, the multimedia information storage device 103 may be an external storage device such as a laser disk device instead of the magnetic disk device. In this case, the moving image or sound from the camera 1 or the microphone 2 may be directly registered in the multimedia information storage device 103 without passing through the control unit 40.

In FIG. 1, reference numeral 7 is a media information management unit for storing in the multimedia information storage device 103 moving images and sounds sent from the camera 1 and the microphone 2, and 8 is a moving image sent from the camera 1. An episode extraction processing unit as dividing means according to the present invention for dividing, a reference numeral 9 denotes a storage structure learning processing unit as summarizing means according to the present invention for summarizing based on the commonality between cuts recorded in the episode storage device 102, Reference numeral 11 is a language analysis unit that performs semantic analysis of the language input from the keyboard 4, 12 is a search unit that performs information search based on the language, and 13 is a browser.

With such a configuration, in the multimedia information management system of this embodiment, the input multimedia information is divided and summarized (commonality information,
Relevance information) is managed in association with information (language information) about meaning. As a result, the form of information acquisition and storage in the machine can be matched to the characteristics of human memory generation and recall. The outline of the characteristic operation according to the present invention will be described below.

First, based on the extraction result (commonality information) of the features from the image data, the captured moving image data is divided, and the divided moving image data is used as one unit (node). Based on the physical features and semantic commonality (relevance information) of moving image data in, a hierarchical summary is given. The summary result is also summarized as an upper node. Then, add a pointer to the storage location of the corresponding multimedia information, the spatial relationship between the objects in the image and the space between the objects, symbols and words that characterize the nodes, patterns that characterize the nodes, and recognition procedures. Information is added to each node and the summary result is stored. Further, as a relationship between the nodes, a front-and-rear relationship in the time direction, a positional relationship in space, a hierarchical relationship between the nodes, and an aggregate relationship that is a part / whole relationship of the nodes are stored.

At the time of information retrieval, the stored summary result (commonality information, relevance information), the stored information about meaning (language information), and part or all of the correspondence between the two are displayed. Then, in response to an instruction from the operator to display the node, the associated image and audio information is reproduced. Furthermore, based on a search inquiry in the operator's language (language information), desired stored image and audio information is searched and displayed. In this way, the operator
It is possible to interactively operate the display content and easily search for the desired display content. Further, in the maintenance of information, the stored summary result and the information regarding the meaning are displayed, and the summary processing, the summary result, and the information regarding the meaning are changed based on the instruction input by the operator as to whether the content of the processing result is correct or not. As described above, in this embodiment, the recording and retrieval of the multimedia information can be performed by simulating the way of human memory and the way (remember) of human memory.

The operation of the present invention will be described below for each processing unit. Program and data of each processing unit,
The intermediate results of the processing are stored in the memory 30 of FIG.
The central processing unit 20 of FIG. In this embodiment, it is assumed that the camera 1 and the microphone 2 are attached to the head of the worker. The moving images and voices of activities such as offices recorded from the camera 1 and the microphone 2 are sent to the media information management unit 7 and the episode extraction processing unit 8. The media information management unit 7 stores the transmitted moving image or sound in the multimedia information storage device 103 as it is.

The episode extraction processing unit 8 divides the transmitted moving image. Hereinafter, this divided section will be referred to as a cut. Generally, moving images are input at 30 frames / sec. Thus, a cut is one or more consecutive frames. The episode extraction processing unit 8 records the correspondence (the start and end frame numbers, etc.) between the extracted cut and the frame of the corresponding image stored in the multimedia information storage device 103. Further, from the information such as the moving image and the sound, the person and the object appearing in the cut, and the background and the atmosphere are recognized and recorded in the episode storage device 102. The memory structure learning processing unit 9 summarizes based on the commonality between the cuts recorded in the episode memory device 102, and structures as shown in FIG. The media information management unit 7 organizes the multimedia information storage device 103 based on the contents of structuring and leaves only necessary information.

FIG. 3 is an explanatory diagram showing an example of a storage structure according to the present invention. FIG. 3 shows an example of expressing an episode of a situation in which a worker has moved to work after performing deskwork work at his desk. The input moving image and sound are stored in the multimedia information storage device 103, and the cuts 1020 to 1
It is divided into 023 or less, and the correspondence between both memories is recorded. Cuts 1020 to 1023 are scenes where two different places (for example, the display and the material) are repeatedly viewed alternately.

FIG. 4 is a block diagram showing a detailed configuration example of the episode extraction processing unit and the storage structure learning processing unit in FIG. FIG. 4 is a diagram particularly showing the flow of processing of the episode extraction processing unit 8 and the storage structure learning processing unit 9, and the feature extraction unit 81 extracts features such as brightness and edges from the image. The motion classification unit 82 calculates the motion vector of each point by, for example, obtaining the difference in brightness or the optical flow of the image. Then, the motion of the camera is obtained from the difference value of the brightness and the motion vector. From the characteristics of this movement, for example, the cut division processing unit 8
In 3, the moving image is divided into a cut having a large movement and a cut having a small movement. In FIG. 3, only cuts with little movement are displayed. The cut division processing unit 83 extracts the direction of movement of a cut having a large movement from an optical flow or the like. Each processing described above can be realized by the existing image processing technology.

The memory structure learning processing section 9 summarizes the cuts based on the physical characteristics and semantic commonality of moving images and sounds. The physical characteristic-based summarization processing unit 91 summarizes the physical characteristics based on, for example, the commonality of the positions between the cuts. The commonality of places can be detected, for example, by pattern matching between cut images. It can be determined that cuts that have a high degree of similarity that are well matched with each other are cuts at the same place. By this processing, the cuts 1020 and 1022 in FIG. 3 are combined into a node 1026, and the cuts 1021 and 1023 in FIG. 3 are combined into a node 1027.
The summarized node is hereafter called a scene. This scene 1
Regarding 026 and the scene 1027, as shown in FIG. 3, based on the information on the direction of movement between the cuts extracted by the episode extraction processing unit 8, as shown in FIG. ) And distance (not shown) are registered.
The physical characteristic-based summarization processing unit 91 further summarizes the activity scene 1028 around the desk and the scene 1029 moved by an errand, as shown in FIG. 3, according to the movement range of the space.

Here, the intermediate result of the summary is temporarily stored in the short-term work area 92, and for example, the scene 1028 in FIG.
When a scene change such as a scene 1029 is detected, it is registered in the episode storage device 102. Also,
The scene in which the temporal structure is retained is scene 10 in FIG.
The time relationship is registered as “time” shown between 28 and scene 1029. In FIG. 3, the time structure is retained between the cuts 1020 to 1023, but the illustration is omitted in the figure. The correspondence between the scene and the image stored in the multimedia information storage device 103 is recorded separately from the cut. Thereby, a moving image characterizing the scene can be displayed.

In this embodiment, the structuring result of FIG. 3 is displayed on the display 3 of FIGS. 1 and 2. Then, according to the evaluation of the division result and the summary result from the user,
The motion classification learning unit 93 and the place classification learning unit 94 change the classification standard. For example, the user gives an instruction of insufficient division or excessive division using the keyboard 4 shown in FIGS. Motion classification learning unit 93 and place classification learning unit 94
Changes the threshold value of the motion in cut division and the threshold value of the similarity degree of the place according to this instruction.

The semantic storage device 101 shown in FIGS. 1 and 2 constructs semantic information as shown in FIG. 3 using, for example, the semantic network of the concept described in Japanese Patent Publication No. 5-84538. Here, the relationship between words according to language is expressed. For example, in FIG. 3, the semantic information 1011 (meeting material creation) includes semantic information 1010 (deskwork) and activities such as meetings and surveys not shown.
The cuts and scenes are labeled with the language so as to be related to the meaning of the meaning storage device 101. For example, in FIG. 3, scene 1028 can be labeled with the word deskwork. It is also possible to use a network expression centered on sentences and verbs as shown in the semantic network configuration example of FIG. This makes it possible to establish a relationship between the language and the real world. Here, for example, the characters can be automatically labeled based on the recognition result in the episode extraction processing unit 8 without depending on the human hands.
In recognition, it is not necessary to specify the person exactly,
Vague labeling like "with a quick man" is also possible.

The cuts and scenes shown in FIG. 3 in the episode storage device 102 can be realized by the data structure shown in FIG. FIG. 6 is an explanatory diagram showing a data structure example of common information and related information according to the present invention. The data structure example shown in FIG. 6 is an example described by the technique adopted in the current object-oriented database. Cuts and scenes
It is expressed by a class named "ba" (hereinafter referred to as "ba class"). The class is a tabular framework composed of a plurality of item (attribute) and value pairs, like the frame proposed in the frame theory. Instances are those that copy a class and store concrete values in attributes.

In the object-oriented database, an actual object is first created in the memory 30 of FIG. 2 by copying a class and creating an instance. When an instance is created, the instance is given an identifier for identifying the created instance. For example, if you copy one place class, place "0"
Is instantiated, and further copying creates an instance of the identifier of field "1". Each attribute of the instance can store a plurality of symbols, numbers, or identifiers of the instance. In particular, by storing the identifiers of a plurality of instances, it is possible to express a one-to-many relationship between the instances.

In FIG. 6, a partial attribute with a diamond mark and a character / object attribute represented by a black circle and a line segment can be represented by this relationship. The former is a relation that represents an aggregation relation, and the instance marked with a diamond indicates the whole, and the instance connected by a line segment shows the portion. The relation between departments and divisions and divisions and departments in a company organization is this aggregate relation. The detailed attribute expresses a similar relationship, but expresses the structure of a temporally consecutive subordinate concept summary. The field class 201 has a time attribute whose attribute value is the identifier of the field instance. This expresses the time relationship between cuts and scenes.

FIG. 7 is an explanatory diagram showing a representation example of the episode memory structure according to the present invention based on the class definition in FIG. By storing the identifier of the field instance 2011 in the time attribute of the field instance 2010, it is indicated that the field instance 2010 temporally precedes the field instance 2011. Positional relationship instance 2020
Represents the positional relationship between scene 1026 and scene 1027 in FIG. The partial attribute is scene 1026 in FIG.
Represents the aggregation relation between the cut 1020 and the cut 1022. It corresponds to the field instance 2010, the field instance 2013, and the field instance 2014, respectively.
Further, in the detailed attribute, the identifier of the lower first place instance in which the time relation is stored is stored. In this way, the spatiotemporal relationship between fields can be expressed.

The image head frame and the image frame length attribute respectively store the head pointer and the frame length of the image of the multimedia information storage device 103 in FIGS. The character / object attribute stores the identifier of the character instance. The positional relationship between the character instances is also expressed by the positional relationship class 204 like the field instances 2010 and 2012. In FIG. 7, both are people and have a lower right / upper left positional relationship with each other. By storing the identifier of the concept instance corresponding to the attribute in the character instance, not only a person but also an object or a place can be stored. In this way, the persons and objects appearing in the image and their positional relationship can also be saved and stored. The language expression of FIG. 5 registers itself as an instance of the language expression class in the description attribute of FIG.

A feature of the object-oriented technique such as the object-oriented database is that not only data but also procedural programs can be linked and stored. In the present embodiment, there is an attribute that stores an identifier of an instance of the recognition pattern 205 of FIG. 6 that stores a pattern that characterizes a scene or a cut and a recognition procedure that matches an input voice or image with the pattern. As a result, even if the language cannot describe whether the input is the scene or cut, it is possible to respond to the inquiry at the pattern level. This function can also be used in the summary processing unit 91 based on the physical characteristics shown in FIG.

[0038] In information retrieval, the recognition and reproduction function in human recall is supported. Regarding the support of this recognition,
This will be described with reference to FIG. The browser 13 displays the summary result of FIG. 3 on the display 3. The scenes and cuts are displayed, for example, as labels, labels in language, still images characterizing the labels, or repeatedly reproduced images of corresponding moving images as icons. The user interactively searches using the keyboard 4 and the mouse 5. The summary result of FIG. 3 is a summary of the past experiences that have been made meaning, which preserves a lot of time and space information. Therefore, by searching this as an index, various units of time and meaning
Images can be searched quickly. In addition, this search can help you remember what you forgot.

With the reproduction support, a language-based search is possible. When a search sentence is input with a word or a sentence, the language analysis unit 11 analyzes the search sentence, for example, as shown in FIG. Then, the search unit 12 performs a search based on the knowledge of the meaning storage device 101. For example, in response to the question “when I met Mr. A”, if there is knowledge in the meaning storage device 101 that Mr. A is a man, Mr. A is 30 years old, and Mr. A is a quick talk, You can search for "with" scenes as candidates. Also, the instance of Mr. A's person is registered in the episode storage device 102, and the feature of the voice and the recognition program are stored as attributes. Then, the link is registered with Mr. A of the meaning storage device 101. Similarly, the voice feature is registered in the person instance in FIG. 7. At the time of search, the feature of the voice is input and the speaker is recognized as the feature of the voice of Mr. A, so that the person-based search can be performed without recognizing Mr. A at the time of registration. As a voice feature, for example, a long-time spectrum is registered.

As described above with reference to FIGS. 1 to 7, in the multimedia information management system of this embodiment,
Information obtained by dividing the input multimedia information and managing it by associating it with information related to meaning, and matching the information acquisition and memory forms in the machine with the characteristics of human memory generation and recall. Can be In this way, by recording and retrieving multimedia information by imitating the way human memory and how human memory follows (remember), it is edited and processed for easy later retrieval. Not only video, but also long-time moving images of daily activities, etc. can be automatically and efficiently organized and stored, and you can quickly search for the desired scene,
Moreover, it is possible to search accurately and quickly based on information that the user can easily remember or remember.

The present invention is not limited to the embodiments described with reference to FIGS. 1 to 7, and various modifications can be made without departing from the scope of the invention. For example, in the present embodiment, the system configuration is performed using the object-oriented technology, but a system configuration in which the operation according to the present invention is performed by a program using another programming language is also possible. In addition, each node (cut 1
020 to 1023, scenes 1026 to 1029, etc.), all may be displayed, or “scene 1028” + “scene 1026” + “scene 1027”
Combination of "Scene 1028" + "Cut 1020"
It is also possible to limit the display to combinations and the like.

[0042]

According to the present invention, the method of human memory and
It is possible to manage multimedia information that mimics how human memory traces (remembers), and to perform accurate and quick search based on information that is easy for the user to remember or remember. It is possible to improve efficiency.

[Brief description of drawings]

FIG. 1 is a block diagram showing an embodiment of a configuration according to the present invention of a multimedia information management system of the present invention.

FIG. 2 is a block diagram showing a configuration example of each device used in the operation of the multimedia information management system in FIG.

FIG. 3 is an explanatory diagram showing an example of a memory structure according to the present invention.

FIG. 4 is a block diagram showing a detailed configuration example of an episode extraction processing unit and a storage structure learning processing unit in FIG. 1.

5 is an explanatory diagram showing a configuration example of a semantic network in the meaning storage device of FIG. 1. FIG.

FIG. 6 is an explanatory diagram showing a data structure example of common information and related information according to the present invention.

7 is an explanatory diagram showing a representation example of an episode storage structure according to the present invention based on the class definition in FIG. 6. FIG.

[Explanation of symbols]

1: camera, 2: microphone, 3: display, 4: keyboard, 5: mouse, 6: interface unit, 7: media information management unit, 8: episode extraction processing unit, 9: storage structure learning processing unit, 10: magnetism Disk device, 11: language analysis unit, 12: search unit, 13: browser, 20: central processing unit, 30: memory, 40: control unit, 50: bus, 8
1: feature extraction unit, 82: motion classification unit, 83: cut division processing unit, 91: physical feature-based summarization processing unit, 9
2: short-term work area, 93: movement classification learning unit, 94: place classification learning unit, 101: meaning storage device, 102: episode storage device, 103: multimedia information storage device,
201: place class, 202: positional relationship class, 203:
Character class, 204: positional relationship class, 1010
-1013: Semantic information, 1020-1023: Cut, 1026-1029: Scene, 2010-201
4: field instance, 2020: positional relationship instance, 2030, 2031: character instance

Front page continuation (72) Koichi Kimura 1-280, Higashi Koikekubo, Kokubunji, Tokyo Inside Hitachi Central Research Laboratory (72) Inventor Hiromichi Fujisawa 1-280 Higashi Koikeku, Tokyo Kokubunji City Central Research Laboratory, Hitachi Ltd.

Claims (11)

[Claims] 1. An information management system for recording multimedia information and outputting multimedia information corresponding to a search condition based on an operator's input operation of the search condition,
Dividing means for dividing the multimedia information into groups based on the commonality of physical characteristics of the multimedia information, and each group divided by the dividing means, based on the relevance of the commonality of each group, Put it in a group,
In addition, the group is repeated into a further upper group to generate the group in a hierarchical structure, and a semantic network for connecting languages based on the meaning of the language information is used to define commonality information of each group in the hierarchical structure. Multimedia information characterized by using the commonality information and relevance information of each group in the hierarchical structure, and the language information as search conditions for the multimedia information. Management system. 2. The multimedia information management system according to claim 1, further comprising means for displaying commonality information, relevance information and language information of each group of the hierarchical structure. Management system. 3. The multimedia information management system according to claim 1 or 2,
The multimedia information management system, wherein the dividing means divides the multimedia information into groups based on the commonality of objects in the moving image of the multimedia information. 4. The multimedia information management system according to claim 3, wherein the commonality information used by the dividing means is determined based on the movement of the camera that photographed the moving image. . 5. The multimedia information management system according to claim 3 or 4,
A multimedia information management system, wherein the positional relationship between the groups is obtained based on the movement of the camera that captured the moving image, and the positional relationship between the groups is used as relevance information used by the summarizing means. 6. The multimedia information management system according to claim 1, wherein the spatial positional relationship between a plurality of things in the multimedia information is
A multimedia information management system characterized by being used as commonality information of the above group. 7. The multimedia information management system according to claim 1, wherein the commonality information and the relevance information are formed by a pattern, and the commonality and the relevance information are recognized by pattern recognition. A multimedia information management system characterized by making a distinction. 8. The multimedia information management system according to claim 1, wherein the commonality information and the relevance information are formed by a language including symbols, and the commonality information and the relevance information are analyzed by semantic analysis. A multimedia information management system characterized by determining relevance. 9. The multimedia information management system according to claim 1, wherein the groups are associated with each other in a time-direction context, and the association information is used as a search condition for the multimedia information. A multimedia information management system characterized by being used. 10. The multimedia information management system according to any one of claims 1 to 9, wherein the groups are associated with each other in a spatial positional relationship, and the association information is used as a search condition for the multimedia information. A multimedia information management system characterized by being used. 11. The multimedia information management system according to any one of claims 1 to 10, wherein the grouping of the hierarchical structure is corrected based on an instruction input from an operator. Information management system.