JP2001142453A - Display control method and display control device which display image related to the contents of all or portion of moving image, information providing server and image related to the contents of all or portion of the moving image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suitably display an image which makes the contents of all or a portion of moving image easily graspable at a glance. SOLUTION: One or more of objects which appear in all or a portion of original moving image are extracted and related information formed between the objects is obtained. The information includes information that regulates interaction among objects such as objects A and B are conducting conversation. Then, based on the appearing timing of each object and the related information among the objects, display position of each object is determined and an image in which each object is rarranged is synthesized. In the synthesized image, since the related information among the objects are indicated clearly or suggestively, by glancing at the synthesized image, the contents of the original image can be understood to a certain extent. In other words, a viewer easily comprehends the mage compared with the frame in which the original moving image is extracted as it is.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a display control method and a display control device for displaying an image related to the contents of a whole or a part of a moving image, and more particularly to a whole or part of a moving image. The present invention relates to a display control method and a display control device for displaying an image for easily grasping the contents of the display at a glance.

More specifically, the present invention provides a display for displaying, within a limited screen space, an image for easily grasping at a glance the whole or a part of a moving image over a relatively long time. The present invention relates to a control method and a display control device.

[0003]

2. Description of the Related Art When a moving image is processed by software on a device that handles a moving image such as a video playback or a computer system, the contents of a whole or a part of the moving image are summarized for the purpose of convenience of editing work. Or it is often necessary to provide a symbolic display. For example, in a device that performs non-linear editing of a moving image, it is common to display a short video with a large length on the screen as a video clip on the screen and perform video editing as an operation on the video clip. ing. In such a case, the question is how to display the video clip on the screen or how to make it.

FIG. 1 shows a display example (prior art) of a video clip. FIG. 7A shows a representative frame (for example, a leading frame or a key frame of the clip) of each clip arranged and displayed. According to such a method, a video clip could be produced relatively easily. However, FIG.
In the display described in (1), the length of each clip and the temporal relationship between the clips cannot be understood at a glance.

In the example shown in FIG. 1B, each clip is represented as a rectangle, one side of which is made to have a length proportional to the time of the clip, and a representative frame of the clip is displayed at one end of the rectangle. Let me. FIG.
In the example shown in (C), each clip is represented as a rectangular parallelepiped, the depth of the rectangular parallelepiped is given in proportion to the time of the clip, and a representative frame of the clip is mapped on the surface of the rectangular parallelepiped.

[0006] In each of the examples shown in FIGS. 1B and 1C, the visibility is improved as compared with the example shown in FIG. 1A.
The device is ingenious so that it can be understood intuitively. However, the content of each clip is displayed as an image of one frame per clip, and it is difficult to grasp the content of the clip only by looking at it.

FIG. 1D shows an example shown in FIG.
It can also be said to be an extended version of the example according to. That is, a plurality of frame images are extracted from one clip, and the respective frames are arranged and displayed in chronological order within a rectangle representing the clip. According to such a clip manufacturing method, it is much easier to understand than the above-described examples, and the contents of the clip can be grasped to a considerable extent at a glance. However, the method shown in FIG. 1D is effective only when the length of the video clip is not very long.

For example, for a plurality of movies, the contents of the story are displayed on the screen to allow the user to select a movie to be viewed, or a scene (scene) to be viewed of each movie is specified from the clip display. Let's consider an application that starts viewing from any point. In such a case, the length of the original movie may be as long as 1.5 to 3 hours, so that a large amount of screen space is required to display the video clip in the form shown in FIG. Is not practical. In addition, a solution of using a screen scrolling function is also conceivable, but it is difficult to see from the beginning to the ending of a long movie, and it is inevitable that visibility is lacking.

If the number of frames extracted from each clip is limited in order to display in a smaller screen space,
This causes another problem that the context between the displayed frames is not understood.

[0010] In other words, if the frames included in the original moving image are used as they are, long contents cannot be effectively expressed.

[0011]

SUMMARY OF THE INVENTION An object of the present invention is to provide an excellent display control method and an excellent display control device capable of displaying an image related to the contents of the whole or a part of a moving image. It is in.

It is a further object of the present invention to provide an excellent display control method and display control which can suitably display an image for easily grasping the contents of the whole or a part of the moving image at a glance. It is to provide a device.

A further object of the present invention is to suitably display an image for easily grasping at a glance the whole or a part of a moving image over a relatively long time in a limited screen space. An object of the present invention is to provide an excellent display control method and an excellent display control device.

[0014]

SUMMARY OF THE INVENTION The present invention has been made in consideration of the above problems, and a first aspect of the present invention is to display an image related to the contents of the whole or a part of a moving image. A display control method, comprising: (a) obtaining one or more objects appearing in the whole or a part of a moving image; and (b) forming between the objects extracted in the step (a). Obtaining relevant information; (c) determining a display position of each object based on a rule corresponding to the relevant information formed between the objects; and (d) according to the determination result in the step (c). Generating an image in which each object is arranged.

Here, the related information formed between the objects includes a temporal relationship in which each object appears in the original moving image, a positional relationship in a predetermined scene in the original moving image, and a relationship between the objects. It shall include at least one of a semantic relationship and a mutual relationship in the context of the original moving image.

In the display control method, when the display positions of the objects derived based on the rules corresponding to the two or more pieces of related information in the step (c) conflict, the priority of each rule May be arbitrated according to the following. Alternatively, when the display positions of the objects derived based on the rules corresponding to the two or more pieces of related information in the step (c) conflict, the sum of the costs of each rule is minimized. The conflict may be arbitrated.

Further, the display control method may further include a step of adding a background that matches the entirety of the moving image or a part of the moving image to the image synthesized in the step (d).

Further, related information formed between the objects may be handled in a meta data format.

Further, the images generated in the step (d) may be images arranged corresponding to a temporal relationship in which each object appears on a time axis set on the image plane. .

Alternatively, the image generated in the step (d) may be an image that is displayed in a perspective view in accordance with the passage of time on a time axis set on the image plane.

Alternatively, the image generated in the step (d) may be an image in which objects are arranged on two or more time axes set on the image plane.

Alternatively, the image generated in the step (d) may be an image in which objects are arranged on a substantially annular time axis set on the image plane.

According to a second aspect of the present invention, there is provided a display control device for displaying an image related to the contents of the whole or a part of a moving image, and (A) the whole or a part of the moving image. Means for acquiring one or more objects appearing in (A), (B) means for acquiring related information formed between the objects extracted in the means (A), and (C) means for forming between the objects. A rule base that stores multiple rules corresponding to related information,
(D) means for extracting a rule corresponding to the related information acquired in the means (B) from the rule base, and determining a display position of each object based on the rule; and (E) means for displaying the object (D). ) For generating an image in which the respective objects are arranged in accordance with the result of the determination according to (1).

Here, the related information formed between the objects includes a temporal relation in which each object appears in the original moving image, a positional relation in a predetermined scene in the original moving image, and a relation between the objects. It shall include at least one of a semantic relationship and a mutual relationship in the context of the original moving image.

Further, the display control device may further include, when the display positions of the objects derived by the means (D) based on the respective rules corresponding to the two or more pieces of related information conflict, the priority given to each rule. Means for arbitrating the competition according to the ranking may be provided. Alternatively, the means (D) is 2
When the display positions of the objects derived based on the rules corresponding to the related information described above conflict, the means for mediating the conflict by minimizing the sum of the costs of each rule is further provided. Is also good.

The display control device may further comprise means for adding, to the image synthesized by the means (E), a background corresponding to the entirety or a part of the moving image.

Further, on the display control device, related information formed between each object may be handled in a meta data format.

Further, on the display control device, the images generated by the means (E) are arranged in correspondence with the temporal relationship in which each object appears on a time axis set on the image plane. It may be an image.

Alternatively, the image generated by the means (E) may be an image that is displayed in a perspective view in accordance with the passage of time on a time axis set on the image plane.

Alternatively, the image generated by the means (E) may be an image in which each object is arranged on two or more time axes set on the image plane.

Alternatively, the image generated by the means (E) may be an image in which objects are arranged on a substantially annular time axis set on the image plane.

According to a third aspect of the present invention, there is provided an image relating to the contents of the entire moving image or a part thereof, wherein one or more objects appearing in the moving image are represented by:
This is an image characterized by being arranged corresponding to the temporal relationship in which each object appears on a time axis set on the image plane.

According to a fourth aspect of the present invention, there is provided an image relating to the contents of the entire moving image or a part thereof, wherein one or more objects appearing in the moving image are represented by:
This is an image characterized in that it is displayed on the time axis set on the image plane in a perspective view in accordance with the passage of time.

According to a fifth aspect of the present invention, there is provided an image relating to the contents of the entire moving image or a part thereof, wherein one or more objects appearing in the moving image are represented by:
An image characterized by being arranged on two or more time axes set on the image plane.

According to a sixth aspect of the present invention, there is provided an image relating to the contents of the entire moving image or a part thereof, wherein one or more objects appearing in the moving image are represented by:
This is an image characterized by being arranged on a substantially annular time axis set on the image plane.

[0036]

According to the present invention, a video clip is not displayed by simply using a representative frame such as a key frame as it is, but one or more objects appearing in the entire moving image or a part thereof are extracted and each object is extracted. By rearranging the images, an image (for example, a still image) from which the whole or a part of the moving image can be easily understood is newly synthesized.

That is, one or more appearing objects are extracted from the whole or a part of the original moving image, and related information formed between the objects is obtained. The related information referred to here is, for example, an object A
Information that defines the interaction between the objects, such as that the object and the object B are in a conversation (or have a quarrel) or are performing a predetermined collaborative work.

Then, the display position of each object is determined based on the appearance time of each object and the related information formed between the objects, and an image (for example, a still image) in which each object is rearranged according to the determined content is synthesized. I do.

Since the related information between the objects is explicitly or implicitly expressed in the newly synthesized image in this manner, the user can simply look at the synthesized image to see the contents of the original image. And the context of stories. That is, it is much easier to understand than a frame simply extracted from the original moving image.

Still other objects, features and advantages of the present invention are:
It will become apparent from the following more detailed description based on the embodiments of the present invention and the accompanying drawings.

[0041]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 2 shows an example in which a conventional display system and a display system according to the present invention are compared for a scene in which two characters A and B are talking as objects.

FIG. 2A shows the conversation scene in the same manner as that described with reference to FIG. 1D. Generally, in the conversation scene, the respective characters A and B alternately appear, so that the respective characters A and B
, A video clip will be created in which the frames showing are alternated. In such a case, FIG.
As shown in (a), the display time tends to be long.

On the other hand, in the present invention, FIG.
As shown in (b), the character A reflected in the conversation scene
And B are taken out as objects, and these objects are rearranged on the screen so as to face each other and the images are synthesized, so that A and B are obtained on a smaller screen space.
It is possible to accurately represent that it is a video of the conversation scene.

Also, at the time of image synthesis, each of A and B is rearranged so as to reproduce the positional relationship between the characters A and B in the original scene, and a background (bitmap) suitable for the scene is added. This makes it possible to more accurately and accurately convey information and atmosphere of the original video.

The points of the above display control method are summarized as follows.

The objects shown in the video are extracted and re-mapped to a two-dimensional image (still image) based on each of the following relationships formed between the objects. (1) Temporal relationship: "B appears before A", "B appears longer than A", "A
(2) Positional relationship in the original scene: "A is located before B", "A and B face each other",
"The traveling direction of A crosses the traveling direction of B" (3) Interaction on the story (context): "A
And B are talking "," A and B are cooperating (cooperatively performing a predetermined task) "," A and B are hostile ", etc.

FIGS. 3 to 6 show examples in which the objects extracted from the whole or a part of the moving image are arranged and displayed in a picture scroll type.

FIG. 3 shows the most basic form, in which objects appearing are arranged and displayed in chronological order to the right on the paper as the story progresses like a normal picture scroll. It is.

FIG. 4 is a modification of the one shown in FIG. 3. In FIG. 4, the objects are arranged on a straight line in the order of appearance in a three-dimensional space and are statistically arranged on a two-dimensional plane. . According to the example shown in this figure, the passage of time can be represented by a perspective, that is, the size of the object. Further, the same contents can be displayed with a smaller area than the example shown in FIG. In addition, by zooming up the screen, it is possible to sequentially progress to the previous story.

In FIG. 5, as in FIG. 4, the objects are arranged along the time axis in the order in which they appear, and the lapse of time is represented by the perspective, that is, the size of the objects. By zooming up on the screen, it is possible to sequentially progress to the previous story. In this case, the time axis is oriented from the periphery of the screen toward the center.

FIG. 6 shows the picture scroll-like image shown in FIG. 3 formed in a drum shape. In this case, the time axis is formed in a ring shape and is directed to the rotation axis direction of the drum, and the progress of the story can be expressed by the rotation of the drum.

In each of the examples shown in FIGS. 3 to 6, the moving picture is far more in comparison with the conventional display method in which the frames in the moving picture are directly taken out (see FIGS. 1 and 2). It can express the contents of the image richly, and
The user can easily understand the story at a glance.

[0054]

Embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 7 shows an outline configuration of a system using the display control device 10 for displaying an image related to the content of a moving image according to the present invention.

In the example shown in the figure, a moving image is supplied in the form of audio and video data (AV data) of a broadcast program distributed from a broadcast station 1 via a broadcast satellite 5 by a satellite wave (or a terrestrial wave) or the like. Is done. In addition, the display control device 10
Is implemented in the form of a receiver (for example, a set-top box) for satellite broadcasting or the like which is disposed in a general home or the like.

Next, a method of constructing information for realizing "picture scroll display" (see FIGS. 3 to 6) will be described with reference to the flowchart shown in FIG. In order to realize image composition and display representing the content of a moving image according to the present invention, the broadcast station 1 performs an extraction process of each object in the moving image (step S1), and additionally, extracts each object. The related information formed between the objects appearing in the whole or part of the moving image, such as the role of the moving image in the story, the relationship with other objects, the scene setting, and the relation to the previous and next scenes, It is acquired appropriately (step S2).

In realizing the present invention, the form for supplying the relevant information to the display control device and the data format of the relevant information are not particularly limited. In the example shown in FIG. 7, the objects and the related information between the objects are “meta data”.
, And is provided in the form of data broadcasting content served by digital satellite broadcasting. Here, “meta data” refers to data attributes, semantic content,
Refers to information for managing data, such as the acquisition destination and storage location.

Regarding the extraction of an object from a moving image, many proposals have been made so far, including manual and automatic. Also, regarding the generation of meta data,
A method of generating the video based on the editing data used when the video is generated can be considered. However, since these matters are not directly related to the gist of the present invention, they will not be further described herein.

Further, information of the extracted object is set to 1
(Step S)
3) Transmit as a transmission wave and distribute via satellite broadcast 5 (step S4). Regarding the encoding method, MPE
G (Motion Picture Experts
Group-4), and can be realized by applying existing technology.

Similarly, meta data is encoded and transmitted (step S5). Regarding multimedia coding of digital satellite broadcasting, ARIB (Association of Radii) is used in Japan.
o Industry and Business
s: The Radio Industries and Businesses Association) is leading the standardization work.

FIG. 9 schematically shows a functional block configuration of the display control device 10. As shown in FIG. 1, the display control device 10 includes a receiving unit 11, a decoding unit 12, an object position determining unit 13, an image combining unit 14, a video signal converting unit 15, and a rule base 16. Is done.
After receiving information such as a moving image and meta data transmitted from the broadcast station 1, the display control device 10 uses, for example, a picture scroll type (see FIGS. 3 to 6) based on the rule base 16 held by itself. ), And outputs the image to a display locally connected to the display control device 10.

FIG. 10 schematically shows the operation procedure of the display control device 10 according to the present embodiment in the form of a flowchart. Hereinafter, each step of this flowchart will be described.

First, the receiving unit 11 receives information transmitted from the broadcasting station 1 via the broadcasting satellite 5 (step S11).

Next, the decoding section 12 decodes the data received by the receiving section 11 to extract information on objects in the video and meta data (step S).
12).

Next, the object position determining unit 13
Based on the extracted meta data, the arrangement position of each object is determined with reference to the rule base 16 (step S13). As will be described later, the rule base 16 is a database that collects rules defining how to combine objects based on meta data.

In the image synthesizing section 14, the object information passed from the decoding section 12 and the object position determining section 1
Based on the position information determined in Step 3, an actual image is synthesized (Step S14).

Next, the video converter 15 converts the synthesized image (for example, a picture scroll type image as shown in FIGS. 3 to 6) into an appropriate video signal,
CRT (Cathode Ray Tube) display and liquid crystal display (LCD: L
The display is output to a display (e.g., liquid crystal display) (step S15).
In order to suitably realize the present invention, the video conversion unit 15
A (Video Graphic Array) or higher display function (for example, SVGA (Super Video
Graphic Array) or XGA (extend)
ed video Graphic Array)). When displaying, another G
UI (Graphic User Interface)
e) It may be displayed as an overlay on a screen or video image.

The flowchart of FIG. 11 shows the processing of steps S12 to S14 relating to image synthesis in more detail. Hereinafter, each step of this flowchart will be described.

[0070] First, the object o _1, o ₂ from the received decoded data, ..., acquires information of o _m (step S21).

Next, meta data is obtained from the received and encoded data (step S22). In this specification, for example, meta data indicating related information formed between objects is described as, for example, R _i (o ₁ , o ₂ ) (the subscript i added to R is a serial number). Please understand.)

[0072] _{R i (o p, o q} ) meta-data that is,
Relation R _i between the object o _p and o _q means that satisfied (R _i is described by the meta-data format). For example, the relationship of "o _p is to appear before the o _q" is described as the following equation.

[0073]

[Number 1] appearEarlierThan (o _p, o _q)

[0074] In addition, the relationship of "o _p and o _q is interacting." Is described as the following equation.

[0075]

[Number 2] dialogue (o _p, o _q)

In general, it should be understood that the relationship as shown in the above equations can be defined between an arbitrary number (including zero) of objects.

Next, in the FOR loop composed of steps S23 to S26, all the rules R _i stored in the rule base 16 (provisionally n in this embodiment).
(O _p, o _q) based on (however, the 0 <i ≦ n),
Each object o _1, o _2, ..., determining a relative positional relationship between o _m.

[0078] For _{example, R i (o p, o} q) rule corresponding to the relevant information of the meta data format, each object o _p, of whether may be disposed in any relative positional relationship o _q Defines constraints. For example,
"If the o _p is to appear before the o _q is, to place the o _p to the left of the o _q" there is such a rule that defines the effect that. The rule can be described as:

[0079]

(Equation 3)

[0080] However, in the above equation, o _p. x and o _q . x
Respectively show the x-coordinate of the object o _p, o _q are the. In particular, the above equation (1), as the relationship is established following o _p. x and o _q . It specifies that x must be placed.

[0081]

(Equation 4)

Next, in step S27, step S27
Based on the relative position information between the objects obtained by the FOR loop according to 26, the arrangement of all the objects, that is, the absolute coordinate position and the image size are determined.

At this time, there is a possibility that a conflict occurs between the pieces of relative position information. For example, each of the two objects o _p
And o _q, the relative position between the o _q and o _r is obtained automatically determined the positional relationship between o _r and o _p. In contrast, if the different relative positions between the o _r and o _p In other rules are defined, the inconsistent i.e. between rules conflict occurs. In such a case, a competitive relationship can be arbitrated by setting priorities among the rules in advance. Alternatively, “cost” (or “weighting factor”) is predetermined for each rule, and the arrangement between the objects is determined so that the total cost or weight is minimized, thereby arbitrating inconsistency, that is, competing relation. You can also.

For example, in the FOR loop according to step S26, it is assumed that the following constraint condition has been obtained.

[0085]

(Equation 5)

Obviously, the above three conditions (3) to (3)
(5) cannot be satisfied at the same time. Therefore, priorities (values in square brackets at the right end in the expressions) are assigned to the conditional expressions (3) to (5), and the higher the value, the higher the priority. Then, the conditional expression (5) having the highest priority is prioritized, and the conditions (3) and (4) that conflict with the conditional expression are ignored. Alternatively, assuming that the value in square brackets is regarded as a cost and a value whose sum is low in cost is selected, the conditional expressions (3) and (4) are adopted, and the conditional expressions ( 5) will be ignored.

Next, in step S28, drawing is performed on the screen buffer according to the arrangement and size of each object obtained in the previous step S27. The drawing information on the screen buffer is output on a display or the like.

In the example shown in FIG. 9, the rule base is stored locally in the terminal device 10 that controls the display of a moving image, but the gist of the present invention is not necessarily limited to this. As another example, for example, this rule base may be distributed from the broadcast station 1 side together with the original moving image as encoded broadcast content, similarly to the object information and the meta data. In such a case, the broadcasting station changes the rule base in accordance with the broadcast content, that is, tunes, so that the picture scroll image matching the content of the broadcast content (more specifically, the service content provided in the broadcast program) can be obtained. It can be generated and displayed. Also, on the display control device 10 side, by switching the rule base according to the user's preference,
It is possible to select a display that meets the needs of each user.

Further, in the above, the embodiment of the present invention has been described specifically for a system form based on broadcasting, but the gist of the present invention is not necessarily limited to this. That is, the object information and the meta data do not necessarily need to be distributed in the form of broadcast as broadcast content, and may be distributed in another form.

For example, object information and meta data can be stored as a package medium together with a moving image, that is, video content, or as a separate package.
Can be distributed to each user.

Alternatively, the object information and meta data may be obtained from an external network instead of a broadcast wave. External network is LAN
(Local Area Network), or a wide area network such as the Internet in which countless hosts are interconnected without limit.

A part of the host on the Internet operates as a server 20 for providing information contents, and it is possible to provide such information providing server 20 with a paid or free service of providing object information and meta data. It is. Individual client users can use WWW (W
The location of desired object information and meta data is located using an information retrieval system such as the Old Wide Web, and HTTP (Hyper Text T) is used.
transfer Protocol) or FTP (Fil)
This information can be downloaded according to a protocol such as eTransfer Protocol.

FIG. 12 shows a server (information providing site) on the Internet for storing object information and meta data.
20 schematically shows an example of downloading from the server 20.

Alternatively, the object information and meta data may be automatically generated by a computer instead of being acquired as broadcast waves.

However, the display control device 10 according to the present embodiment specifically assumes a device having a relatively low calculation capability such as a set-top box (STB), and can store such information only by itself. It is difficult to generate. In such a case, a computer (server) 30 externally connected to the display control device 10 automatically generates object information and meta data, and the generated data is stored in the calculation server 3.
You may make it receive from 0.

FIG. 13 schematically shows an example in which object information and meta data are automatically generated by the calculation server 30 connected to the display control device 10. The display control device 10 and the calculation server 30 are, for example, SCSI (Sma
ll Computer System Interf
ace) or via a LAN such as Ethernet. Or,
As in the case of FIG. 12, the connection may be made via the Internet.

[Supplement] The present invention has been described in detail with reference to the specific embodiments. However, it is obvious that those skilled in the art can modify or substitute the embodiment without departing from the spirit of the present invention. That is, the present invention has been disclosed by way of example, and should not be construed as limiting. In order to determine the gist of the present invention, the claims described at the beginning should be considered.

[0098]

As described above in detail, according to the present invention,
An excellent display control method and display control device capable of displaying an image related to the contents of the whole or a part of a moving image can be provided.

Further, according to the present invention, an excellent display control method and display control which can suitably display an image for easily grasping the contents of the whole or a part of the moving image at a glance. An apparatus can be provided.

Further, according to the present invention, an image for easily grasping at a glance the contents of the whole or a part of a moving image over a relatively long time is suitably displayed in a limited screen space. It is possible to provide an excellent display control method and an excellent display control device.

According to the display control method and the display control device according to the present invention, a more appropriate display method can be achieved by switching the arrangement rule of objects relating to related information between objects in accordance with the contents of the distribution content and the needs of the user. Can be selected.

[Brief description of the drawings]

FIG. 1 is a diagram showing a display example (prior art) of a video clip.

FIG. 2 is a diagram comparing a conventional display method and a display method according to the present invention with respect to a scene in which two characters A and B are talking as objects, and more specifically,
FIG. 2A is a diagram showing a conventional display system, and FIG. 2B is a diagram showing a display system according to the present invention.

FIG. 3 is a diagram showing an example of a display method according to the present invention.

FIG. 4 is a diagram showing an example of a display method according to the present invention.

FIG. 5 is a diagram showing an example of a display method according to the present invention.

FIG. 6 is a diagram showing an example of a display method according to the present invention.

FIG. 7 is a diagram showing an overview configuration (an example using satellite broadcasting) of a system using a display control device 10 that displays an image related to the content of a moving image according to the present invention.

FIG. 8 is a flowchart schematically illustrating a method of configuring information for performing image composition according to the present embodiment.

FIG. 9 is a diagram schematically illustrating a functional block configuration of a display control device 10 according to the present embodiment.

FIG. 10 is a flowchart schematically showing an operation procedure of the display control device 10 according to the embodiment.

FIG. 11 is a flowchart showing the processing of steps S12 to S14 relating to image composition in more detail.

FIG. 12 illustrates another embodiment of the present invention. More specifically, object information and meta data are transferred to a server (information providing site) 20 on the Internet.
FIG. 5 is a diagram showing an example of downloading from a.

FIG. 13 illustrates another embodiment of the present invention, and more specifically, an example in which object information and meta data are automatically generated by a calculation server 30 connected to the display control device 10. FIG.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Broadcasting station 5 ... Broadcast satellite 10 ... Display control device 11 ... Receiving part 12 ... Decoding part 13 ... Object position determining part 14 ... Image synthesizing part 15 ... Video signal converting part 16 ... Rule base 20 ... Information providing site 30 ... Calculation server

 ──────────────────────────────────────────────────続 き Continuation of the front page (54) [Title of the Invention] A display control method and a display control device for displaying an image related to the contents of the whole or a part of the moving image, an information providing server, and a moving image Images related to the content of the whole or part of it

Claims (25)

[Claims] 1. A display control method for displaying an image related to the contents of a whole or a part of a moving image,
(A) acquiring one or more objects appearing in the whole or a part of the moving image; and (b) acquiring related information formed between the objects extracted in the step (a). (C) determining a display position of each object based on a rule corresponding to related information formed between the objects, and (d) determining an image in which each object is arranged according to the determination result in the step (c). Generating a display control method. 2. The related information formed between the objects includes a temporal relationship in which each object appears in an original moving image, a positional relationship in a predetermined scene in the original moving image,
The display control method according to claim 1, further comprising at least one of a semantic relationship between the objects and a mutual relationship in a context of the original moving image. 3. In the case where the display positions of the objects derived based on each rule corresponding to two or more related information in the step (c) conflict, the conflict is determined in accordance with the priority order of each rule. The display control method according to claim 1, wherein arbitration is performed. 4. When the display positions of the objects derived based on each rule corresponding to two or more related information in the step (c) conflict, the total sum of the costs of each rule is minimized. 2. The display control method according to claim 1, wherein the competition is arbitrated in such a manner as to be performed. 5. The method according to claim 1, further comprising the step of adding a background that matches the entirety or a part of the moving image to the image synthesized in step (d). Display control method. 6. The display control method according to claim 1, wherein related information formed between each object is handled in a meta data format. 7. The image generated in the step (d) is an image arranged corresponding to a temporal relationship in which each object appears on a time axis set on the image plane. The display control method according to claim 1, wherein 8. The image generated in the step (d) is an image displayed in a perspective view corresponding to a lapse of time on a time axis set on the image plane. 2. The display control method according to 1. 9. The image according to claim 1, wherein the image generated in the step (d) is an image in which objects are arranged on two or more time axes set on the image plane. Display control method described. 10. An image according to claim 1, wherein the image generated in said step (d) is an image in which objects are arranged on a substantially annular time axis set on said image plane. Display control method described. 11. A display control device for displaying an image related to the contents of the whole or a part of a moving image, wherein (A) acquiring one or more objects appearing in the whole or a part of the moving image (B) means for acquiring the related information formed between the objects extracted in the means (A), and (C) a plurality of rules corresponding to the related information formed between the objects. A rule base to be accumulated; and (D) means for extracting a rule corresponding to the related information acquired in the means (B) from the rule base and determining a display position of each object based on the rule. E) means for generating an image in which each object is arranged according to the result of the determination by means (D). 12. The related information formed between the objects includes a temporal relationship in which each object appears in the original moving image, a positional relationship in a predetermined scene in the original moving image, and a meaning between the objects. The display control device according to claim 11, wherein the display control device includes at least one of a dynamic relationship and a mutual relationship in the context of the original moving image. 13. When the display positions of objects derived by the means (D) based on each rule corresponding to two or more pieces of related information conflict, the conflict is arbitrated in accordance with the priority of each rule. The display control device according to claim 11, further comprising: 14. When the display position of an object derived based on each rule corresponding to two or more pieces of related information conflicts with the means (D), the sum of the costs of each rule is minimized. 12. The display control device according to claim 11, further comprising means for arbitrating the competition in such a manner. 15. The apparatus according to claim 11, further comprising means for adding a background matching the whole or a part of the moving image to the image synthesized by said means (E). Display control device. 16. The method according to claim 11, wherein related information formed between the objects is handled in a meta data format.
3. The display control device according to 1. 17. An image generated by said means (E) is an image arranged corresponding to a temporal relationship in which each object appears on a time axis set on the image plane. The display control device according to claim 11, wherein: 18. An image generated by the means (E) is an image displayed in a perspective view corresponding to a lapse of time on a time axis set on the image plane.
3. The display control device according to 1. 19. An image generated by said means (E) is an image in which each object is arranged on two or more time axes set on said image plane.
3. The display control device according to 1. 20. An apparatus according to claim 11, wherein said image generated by said means (E) is an image in which objects are arranged on a substantially annular time axis set on said image plane.
3. The display control device according to 1. 21. The display control device according to claim 11, wherein the display control device is an information providing server connected to a network, and transmits object information and / or related information between objects to the display control device. Information providing server. 22. An image relating to the contents of the whole or a part of the moving image, wherein one or more objects appearing in the moving image are displayed on a time axis set on the image plane. An image characterized by being arranged corresponding to a temporal relationship in which objects appear. 23. An image relating to the contents of the whole or a part of a moving image, wherein one or more objects appearing in the moving image are displayed on a time axis set on the image plane. An image characterized by being displayed in a perspective view in accordance with the progress of. 24. An image related to the contents of the whole or a part of the moving image, wherein one or more objects appearing in the moving image are set on the image plane.
An image characterized by being arranged on the above time axis. 25. An image relating to the contents of the whole or a part of a moving image, wherein one or more objects appearing in the moving image are displayed on a substantially annular time axis set on the image plane. An image characterized by being arranged above.