JP2003323439A - Multimedia information providing method, device and program, and recording medium for storing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To divert the defined link relation to the other in a multimedia information providing device for providing the contents as the relevant information to each scene or an object of an image, as the relevant information object. <P>SOLUTION: This multimedia information providing device comprises a database 402 for accumulating the link relation of the meta information of the object as a momentum of an event and the meta information of the relevant information object to the object, a database 406 for accumulating the meta information representing the characteristic of the object as the momentum of the event, a database 407 for accumulating the meta information representing the characteristic of the relevant information object, and a relevant information object retrieving part 403 for determining the relevant information object to be disclosed on the basis of the retrieval of the meta information, and the matching of the meta information. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video-related information presentation system for additionally presenting video content provided as main content and information related to the video content. In particular, for example, a user indicates a subject in a video. The present invention relates to a multimedia information providing method and apparatus for presenting related information of multimedia contents in combination with a video when triggered by such an event.

[0002]

2. Description of the Related Art Japanese Patent Application Laid-Open No. 9-34869 discloses a method of defining an action such as presenting related multimedia contents in response to an event such as a subject instruction by a user who is viewing video contents. ing. In this method, for example, when the user instructs a still image pasted on the display screen, another multimedia content related to the instructed still image can be newly presented.

Further, multimedia contents related to a part of an instructed image are presented to an event such as user interaction with a part of the image such as a subject in the still image or the video instead of the entire still image or the video. can do.

Further, contents can be presented by using an internal trigger as an event without an external input such as user interaction as a trigger for the event occurrence. An example of this would be to use an internal timer to periodically check the playback frames of the video, and when the video becomes a predefined scene, present another multimedia content related to that scene. What
There are scenarios such as synchronizing with video contents.

Hereinafter, such a conventional multimedia information providing method will be described with reference to an example of realizing an electronic aquarium service as shown in FIG. In the electronic aquarium service here, the image 101 of the aquarium aquarium is displayed on the screen 100, and when the fish appears as the subject 102 in the aquarium image 101, the user is the subject (fish). When an event of pointing 102 occurs, a text 103 that is a detailed description of the fish.
Is combined with the image 101 of the aquarium and presented.

[0006] FIG. 2 is a diagram for explaining the structure of a multimedia information providing apparatus for realizing the conventional multimedia information providing method and the data flow there.

The multimedia information providing apparatus includes an event detection unit 201 for detecting an event given as an external input 200, a database 202 for describing and storing a link relationship between an object that triggers an event and a related information object, and detection. A related information object search unit 203 that searches the database 202 for a related information object based on the retrieved event, and a related information presentation unit 205 that presents the searched related information object to the user.

[0008] The scenario creator defines in advance in the database 202 the link relationship between the fish in the video and the description of the fish as a pair of the object that triggers the event and the object that is presented as related information. In this example, since the subject (fish) in the image is used as the trigger of the event, it is defined as an object using the frame range or coordinate region in which the subject appears. Below,
A subject object is defined as such a subject.

For example, the subject "Angelfish"
5 frames to 10 frames of the video "movie.mpg" (video represented by the file name "movie.mpg". In the following, video and text will be displayed with the file name representing it.) When the coordinates "(50, 50)" and "(120, 120)" appear in a rectangle having diagonal vertices, the subject "angel fish" is defined as a subject object, and is an object that triggers an event. Also, "Angelfish"
Is the text "angelfilsh.txt", the text "angelfilsh.txt" is an object to be presented as related information.

Hereinafter, an object presented as such related information will be referred to as a related information object. Then, the pair of the subject object and the related information object is defined in the database 202. Similarly, with respect to other subject objects in the video, link relations are defined in the database 202 in pairs with the description of the fish serving as the related information object.

Here, when the same fish appears many times in the image, each time the fish appears, one subject object is defined in the frame in which the fish appears and paired with the description. Define it. For example, "Angelfish" has coordinates (30,10) and (50,7) in the 50th to 60th frames of the video "movie.mpg".
When the object object also appears inside a rectangle whose diagonal vertex is (0), this subject object is also defined in the database 202 as a pair with the text "angelfilsh.txt" which is the explanatory text.

The operation in this case will be described below.

In the example shown in FIG.
In the 10th frame of g ”, the coordinates (100,
It indicates that there is an external input 200 that indicates “instruction 100)”. As described above, the database 202 stores an object that triggers an event and an object that is presented as related information for the object as a pair.

The event detector 201 is a database 2
02, the external input 200 is the database 202
If it is an instruction to an object that triggers an event in, the occurrence of the event is detected. In the illustrated example, the external input 200 is the video “mo” defined in the database 202.
Since it is an instruction to a subject object defined by a rectangle whose diagonal vertices are coordinates (50, 50) and (120, 120) in the 5th to 10th frames of “vie.mpg”, the event detection unit 201 determines that the event When an event is detected, the event detection unit 201
Passes the object information that triggered the detected event to the related information object search unit 203.

The related information object search unit 203 refers to the database 202 and searches for an object to be presented to the object that triggered the event that occurred. In the example of FIG. 2, the text “angelfilsh.txt” defined in combination with the object that triggered the event that has occurred is searched as the related information object. The searched related information object is delivered to the related information presenting unit 205, and the related information presenting unit 205 receives the text “angelf” which is the searched related information object.
ilsh.txt ”is displayed on the screen.

[0016]

In the conventional multimedia information presenting apparatus described above, a link relationship is defined by storing in the database 202 a pair of an in-video object that triggers an event and its related information object. Was.

Here, the in-video object that triggers an event describes a subject or a scene by information unique to the video such as a spatial position or a temporal position in the video. Further, the related information object also uses the file name when the entire content is the presentation object as in the above-described example. In addition to this, there may be a case where a part of multimedia contents such as a partial image obtained by trimming a still image is used as a related information object. I was using the information.

Therefore, the link relationship defined in the database 202 strongly depends on individual contents such as video and text, and when the contents change,
It is necessary to redefine the link relationship.

For example, consider a case where a similar electronic aquarium service is to be implemented at another facility (another aquarium, etc.). At this time, in the newly created scenario, the image of the aquarium at the facility is used, but it is assumed that the fish appearing in the image are almost the same as before. In such a case, if the link relationship in the database 202 already defined can be reused, a scenario using the content for the facility can be created with a small number of operations. However, since the aquarium image contents to be used are different, the appearance position of the subject object is stored in the database 20.
It is easy to imagine that it is different from the information in 2. Such different information will be described with reference to FIG.

For example, when the definition of the subject object that triggers the event is as shown in the table indicated by reference numeral 306 in FIG. 3, the object information defined in the database 202 in FIG. Since they are different, the link relationship defined in the database 202 cannot be used.

Therefore, it is necessary not only to define the object with the content in the new facility but also to redefine the link relation as shown in the database 302.
In the case of the aquarium service, the subject object "Chowfish" that presents the explanation of the angelfish to the subject object "Angelfish"
For example, the link description between the object that triggers the event and the related information object for the event is obvious to the scenario creator, regardless of the video used. Conceivable. Nevertheless, it is a heavy burden for the scenario creator to redefine the link relationship between those objects.

Similarly, there are many cases where a subject common to each video appears, such as when presenting information about the subject in a continuous drama video or a baseball relay video. In such a case, the defined link relationship is used. Is not a problem.

It is also conceivable that the person who defines the in-video object that triggers the event, the person who defines the related information object, and the person who defines the link relationship between these objects may be different. However, in the conventional method, since the above three pieces of definition information are stored in the same database, it is premised that the same person defines these three pieces of information, which is also a problem to be solved. Can be mentioned.

Therefore, an object of the present invention is to make it possible to reuse the link relationship between the in-video object and the related information object, which is the trigger of the event, in a plurality of services, and to minimize the operation of the scenario creator. A media information providing method and device are provided.

It is another object of the present invention to make it possible to reuse a person who defines a link relationship and a person who defines an individual object without redefining the link relationship.

[0026]

A multimedia information providing method of the present invention is a multimedia information providing method for providing a content as a related information object as related information for each scene or subject in a video, and an event trigger. The procedure for accumulating the link relationship between the meta information of the in-video object to be displayed and the meta information of the related information object presented as related information to the in-video object, and the characteristics of the in-video object that triggers the event. A procedure for accumulating meta information, a procedure for accumulating meta information representing the characteristics of a related information object, a procedure for retrieving meta information describing a link relationship, and determining a link by matching the retrieved meta information. And a procedure for presenting related information according to the determined link information, A.

The multimedia information providing apparatus of the present invention is
A multimedia information providing apparatus that provides contents as related information objects as related information for each scene or subject in a video, and includes meta information of an in-video object that triggers an event and related information for the in-video object. First accumulating means for accumulating the link relationship with the meta information of the presented related information object, second accumulating means for accumulating the meta information representing the characteristics of the in-video object that triggers the event, and the related information. A third accumulating means for accumulating meta information representing the characteristics of the object and meta information describing the link relationship are searched by referring to the first accumulating means, and matching between the meta information searched among the accumulating means is performed. Retrieval means that determines the link by taking the information, and presents related information according to the determined link information. Having a related information providing unit.

That is, in the present invention, the definition information of the related information object presented as the in-video object or the related information that triggers the event and the definition information of the link relationship between these objects are separated. In addition, the link relationship between objects is described using meta information that represents the characteristics of the object, not information specific to the content that includes the object. With such a configuration, in the present invention, the link relationship between objects can be used regardless of the content used. Further, in the present invention, the definition information of the object and the definition information of the link relationship are linked by taking a match between the defined meta information. As a result, it is possible to deal with the case where each definer is different.

[0029]

BEST MODE FOR CARRYING OUT THE INVENTION Next, preferred embodiments of the present invention will be described with reference to the drawings. FIG. 4 is a diagram illustrating a multimedia information providing device according to an embodiment of the present invention. Here, the multimedia information providing apparatus will be described by taking the case of providing the electronic aquarium service shown in FIG. 1 as an example.

The multimedia information providing apparatus shown in FIG. 4 provides content such as video content, still image content, audio content or text content as related information for each scene or subject in the video. The multimedia information providing apparatus includes an event detection unit 401 that detects an event given as an external input 400, and a database 402 that defines a link relationship based on meta information of an object that triggers an event and meta information of a related information object.
And a database 406 that defines the correspondence between the object that triggers the event and the meta information of the object,
A database 407 that defines the correspondence between the related information objects and the meta information of the objects; and a related information object search unit 403 that finds the related information objects (links) corresponding to the event by searching these databases 402, 406, 407. A related information presenting unit 405 that presents the found related information object to the user.

Each database used in this embodiment will be described below.

The database 402 defines a link relationship by storing the meta information of an object that triggers an event and the meta information of an object presented as related information for the meta information as a pair. In the example of the electronic aquarium service, for example, the meta information "angel fish" of the subject object and the meta information "angel fish description" of the related information object to be presented thereto are stored as a pair. Here, even when the same fish appears in the video many times, the meta information of those subject objects is common to the keywords of the names of the subjects, so that the meta information of the object in the video that triggers the event is A pair of link relationships of the meta information of the related information object may be defined.

The database 406 stores, as a pair, an object that triggers an event and meta information that expresses the characteristic of the object for the video content used in the service to be used. That is, in the database 406, preset meta information representing the characteristics of objects in the image such as scenes and subjects in the image is stored. In the example of the electronic aquarium service, the fish appearing in the aquarium image to be used is defined as a subject object, which is the object that triggers the event. Further, here, a keyword of the name of a fish, which is a subject, is used as the meta information representing the characteristics of the object. The subject object and the keyword of the name are paired and stored in the database 406. If the same fish appears many times in the video, each time the fish appears,
A subject object is defined in the appearing frame, and a keyword of the subject name is stored in the database 406 as meta information.

Similarly, the database 407 stores, for each content used in the service to be used, a related information object and meta information expressing the characteristics of the object as a pair. That is, the database 407 stores meta information representing the characteristics of the object presented as the related information. In the example of the electronic aquarium service, the description of each fish is used as a related information object, and its feature is stored as meta information. Here, for example, the meta information of the explanation sentence of angel fish is defined as the text "Angel fish explanation sentence", and the meta information is similarly defined for the explanation sentences of other fish.

Next, the operation of this multimedia information providing apparatus will be described.

The event detection unit 401 uses an external input 400
When an event occurs, the database 40 that describes the correspondence between the object that triggers the event and its meta information
6, the external input 400 is the database 40
If it is an instruction to the object that is the trigger of the event stored in 6, the occurrence of the event is detected. In the example shown in FIG. 4, from the external input 400 to the database 40
The coordinates (50, 50) and (12) in 5 to 10 frames of the video "movie.mpg" defined in 6
The occurrence of an event triggered by a subject object defined by a rectangle whose diagonal vertex is (0, 120) is detected, and this object information is passed to the related information object search unit 403.

When the information of the object that triggers the event is delivered to the related information object searching unit 403, the related information object searching unit 403 searches for the object of the related information according to the procedure shown in the flowchart of FIG. .

First, in step 1, the related information object search unit 403 refers to the database 406 which describes the correspondence between the in-video object that triggers the event and its meta information, and finds the object that triggered the event that occurred. Search for meta information to represent. In the example of FIG. 4, the keyword “angel fish” is searched from the database 406 as the meta information expressing the object that triggered the event that occurred.

Next, in step 2, the related information object search unit 403 sets the meta information most similar to the meta information searched in step 1 to the meta information of the object that triggers the event and the meta information of the related information object. It is selected from the meta information of the object that triggers the event in the database 402 indicating the correspondence with. In this case, the keyword "angelfish" is selected as the meta information most similar to the keyword "angelfish". That is, the matching of the meta information is performed between the database 406 and the database 402.

Next, in step 3, the related information object search section 403 searches the database 402 for the meta information of the object to be presented for the occurred event, using the meta information selected in step 2 as a key. In the example of FIG. 4, the keyword “angel fish description” is searched as the meta information of the object presented to the event triggered by the object having the keyword “angel fish” as the meta information.

Next, in step 4, the related information object searching unit 403 describes the meta information most similar to the meta information searched in step 3 to the database 407 which describes the correspondence between the related information object and the meta information.
Select from the meta information inside. In this case, the keyword “explaining sentence of angel fish” is selected as the meta information most similar to the keyword “explaining sentence of angel fish”. That is, the matching of the meta information is performed between the database 402 and the database 407.

Finally, in step 5, the related information object search unit 403 searches the database 40 for the related information object having the meta information selected in step 4.
Search from 7. In the example of FIG. 4, the text “angelfish.txt” is searched as a related information object having the keyword “angel fish description” as meta information.

After all, the related information object search unit 403
Searches for meta information that describes link relationships,
The link is determined by matching the retrieved meta information. That is, the related information object is determined for the object that triggers the event.

By the above processing, the related information object to be presented for the detected event is determined, and the related information object presenting section 405 presents the related information object to the user. In the example of FIG. 4, the text “angelfish.txt” is displayed on the screen as the related information object.

In this multimedia information providing apparatus, the definition information of the in-video object or the related information object that triggers the event and the definition information of the link relationship between the objects are separately stored in the database. That is, the definition information of the object that triggers the event is stored in the database 406, and the definition information of the related information object is stored in the database 407. The definition information of the link relationship between the objects is stored in the database 402. In the database 402, the link relationship is described using meta information indicating the feature of the object, not information specific to the content including the object. ing. With this configuration, the link relationship between objects can be used regardless of the content to be used, and as a result, even if the content to be used is changed and the object information is changed, it is defined in the database 402. You will be able to use the relationships as they are.

For example, when an electronic aquarium service is being carried out in a certain aquarium using the aquarium image of the aquarium, when trying to carry out a similar service in another aquarium, information on the contents to be used in the new aquarium is provided. Object is defined by using and its characteristics are defined as meta information. At this time, when the video content “movie2.mpg” to be used different from the example in FIG. 4 is used, and the object object and meta information defined in this video are as in the database 606 of FIG. The link relationship defined in the database 402 of can be used as it is, and it is not necessary to redefine the link relationship. If there is an object for which a link relationship is not defined in the database 402, the relationship between the objects may be added by the meta information. Even in this case, the link definition work may be performed only for the undefined object, and the scenario creator may be less active.

In addition, in this embodiment, the description of the link relation and the description of the object information are separated and
By matching the defined meta information,
Object definition information and link relationship definition information are linked. Therefore, the description of the link relationship and the description of the object information can be separately defined and managed. Furthermore, by matching the defined meta information, it is possible to absorb the difference in the definer and the difference in the meta information depending on the content, and therefore it is possible to deal with the case where the respective definers are different.

The present invention has been described above based on the embodiments. However, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the invention. Hereinafter, the changeable parts in FIG. 4 will be described.

(1) Event Occurrence Detection Unit 401: In the above-mentioned embodiment, the external input 400 is input to the event occurrence detection unit 401 as the user interaction, but the external input is not always necessary. For example, with the progress of the internal timer as input, the video playback position is
It is also possible to detect the occurrence of an event when the frame in which a predefined object appears appears. Of course, it has both external input and internal input,
It is also possible to select the event generation condition by separately designating the event generation condition.

(2) A database 406 that describes the correspondence between the object that triggers an event and its meta information,
And the number of meta information per object in the database 407 that describes the correspondence between the related information object and its meta information: In the above embodiment, the meta information added to each object in these databases 406 and 407 is one object. There was one per
It is also possible to add a plurality of meta information to one object. Accordingly, it is possible to define the relationship between the event trigger and the action for the characteristics of each object from a plurality of viewpoints. FIG. 7 shows an example of such a case. In the example of the figure, in the database 706 (corresponding to the database 406 in FIG. 4), two object objects “angel fish” and “butterflyfish” are meta data from two viewpoints of “object name” and “object color”. It defines the information. Further, in the database 702 (corresponding to the database 402 in FIG. 4), the meta information of the action target is defined for the feature of each viewpoint. Database 70
7 (corresponding to the database 407 in FIG. 4) defines an object to be presented as an action target and its meta information.

(3) A database 406 that describes the correspondence between the object that triggers an event and its meta information,
And the form and method of expressing meta information in the database 407 that describes the correspondence between the related information object and its meta information: In the above embodiment, these database 4
As the meta information in 06 and 407, the color and the keyword of the subject object are used, but it is not particularly limited to these, and the form and method of expressing the feature of the object are not limited. Various other objects such as information on the shape of the object and the linguistic meaning vector of the text can be considered as the characteristics of the object.

(4) A database 406 that describes the correspondence between the object that triggers an event and its meta information,
And a method of managing meta information in the database 407 that describes the correspondence between the related information object and its meta information: In the above-described embodiment, these database 406,
Although the meta information is defined in advance in 407, it may be unnecessary to define it in advance depending on the characteristics of the object to be used. For example, when it is possible to obtain from the object at the time of execution by calculation, such as information about the color or shape of the subject object, it is not necessary to define in advance, and the database 406 and the database 4 are required.
07 is not always necessary.

The multimedia information providing apparatus described above can also be realized by causing a computer such as a personal computer to read a computer program for realizing it and executing the program. The program for implementing the multimedia information providing method is
It is read by a computer by a recording medium such as a flexible magnetic disk or a CD-ROM.

This computer is a central processing unit (CPU)
A hard disk device for storing programs and data, a main memory, an input device such as a keyboard and a mouse, a display device such as a CRT, and a reading device for reading a recording medium such as a flexible magnetic disk or a CD-ROM. It consists of The hard disk device, main memory, input device, display device, and reading device are all connected to the central processing unit. In this computer, a recording medium storing a program for implementing the multimedia information providing method is mounted on a reading device, the program is read from the recording medium and stored in a hard disk device,
When the central processing unit executes the program stored in the hard disk device, it functions as the above-mentioned multimedia information providing device.

[0055]

As described above, according to the present invention, a link between an object that triggers an event and a related information object for the event when the object information changes due to content replacement, which is required in the conventional method. There is no need to redefine the relationship, and the operation of the scenario creator can be minimized.

[Brief description of drawings]

FIG. 1 is a diagram illustrating an electronic aquarium service as an application example of a user interactive multimedia information providing method.

FIG. 2 is a diagram illustrating a configuration of a multimedia information providing apparatus that realizes a conventional multimedia information providing method and a data flow therein.

FIG. 3 is a diagram for explaining a problem in a conventional multimedia information providing device.

FIG. 4 is a diagram illustrating a multimedia information providing device according to an embodiment of the present invention.

FIG. 5 is a flowchart showing a process of searching for a related information object.

FIG. 6 is a diagram illustrating that a link relationship between objects can be used regardless of the content used.

FIG. 7 is a diagram illustrating an example in which a plurality of meta information is used for one object.

[Explanation of Codes] 100 Screen 101 Video 102 Subject 103 Text 200, 400 External Input 201, 401 Event Detection Unit 202, 302, 402, 406, 407, 606, 7
02, 706, 707 Database 203, 403 Related information Object search unit 205, 405 Related information presentation unit 306 Object information that triggers event

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Yasushi Hanago             2-3-1, Otemachi, Chiyoda-ku, Tokyo             Inside Telegraph and Telephone Corporation (72) Inventor Yoshinobu Tonomura             2-3-1, Otemachi, Chiyoda-ku, Tokyo             Inside Telegraph and Telephone Corporation F term (reference) 5B075 ND02 ND16 PQ32                 5B082 AA13 EA07

Claims (7)

[Claims] 1. A multimedia information providing method for providing a content as a related information object as related information for each scene or subject in a video, comprising: meta information of an in-video object that triggers an event; On the other hand, a procedure of accumulating a link relationship with the meta information of the related information object presented as related information, a procedure of accumulating meta information representing the characteristics of the in-video object that triggers the event, A procedure for accumulating meta information representing a feature, a procedure for searching the meta information describing the link relation and determining a link by matching the searched meta information, and related information according to the determined link information For presenting multimedia information, and a method for providing multimedia information having . 2. A multimedia information providing method for providing a content as a related information object as related information for each scene or subject in a video, comprising: meta information of an in-video object that triggers an event and the in-video object. On the other hand, a first database that stores a link relationship with the meta information of the related information object that is presented as related information, and a second database that stores the meta information that represents the characteristics of the in-video object that triggers the event. , A third database for accumulating meta information representing the characteristics of the related information object, a procedure for detecting an event, and a second database based on the object in the video corresponding to the detected event. A procedure for searching a database, Based on the meta information obtained by searching, a procedure for searching the first database, and based on the meta information obtained by searching the first database, search the third database, A multimedia information providing method comprising: a step of deciding a related information object to be presented as related information; and a step of presenting the decided related information object. 3. A multimedia information providing apparatus for providing contents as related information objects as related information for each scene or subject in a video, wherein meta information of an in-video object that triggers an event and said in-video object On the other hand, a first storage unit that stores a link relationship with the meta information of the related information object that is presented as related information, and a second storage that stores the meta information that represents the feature of the in-video object that triggers the event. Means, a third accumulating means for accumulating the meta information representing the characteristic of the related information object, and a meta information describing the link relationship with reference to the first accumulating means to search for the accumulating means. A search means for determining a link by matching the searched meta information between Multimedia information provision device having a related information providing means for presenting the related information in accordance with link information. 4. The multimedia information providing apparatus according to claim 3, further comprising an event detection unit that detects the occurrence of an event. 5. The searching means searches the second storage means based on an in-video object corresponding to the detected event, and based on meta information obtained by searching the second storage means. 5. The link is determined by searching the first storage means by searching for the third storage means based on meta information obtained by searching the first storage means. The described multimedia information providing device. 6. A program for causing a computer to execute the multimedia information providing method according to claim 1. 7. A computer-readable recording medium in which the program according to claim 6 is stored.