JP2007174477A - Object-related automatic display system in video reproduction system, player device and server device used for display system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an object-related automatic display system in a video reproduction system in which a motion picture can be reproduced in a manner to be comprehended easily for a user by automatically displaying a relational diagram of an object being displayed in a video image at present by the side of the video image or the like synchronously with video streaming. <P>SOLUTION: A player device 3a fetches a video image from a content server device 1b and simultaneously acquires the relation information of an object corresponding to that video image and track information from a metadata management server device 1a. The relational diagram of the object appearing in the acquired video image is also automatically displayed on a display device 11a together with the video image. The relational diagram displays a relationship between objects with pictures, characters, symbols and so on and enables the user to easily grasp the video image. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a video playback system having a player function and a server function that can dynamically display information (metadata) related to video content in synchronization with a playback video, and more specifically, a display device of a player device. The method of automatically displaying a relational diagram of objects (characters, items, BGM, etc.) related to the currently displayed moving image together with the moving image.

For reproduction of still images, information relating to still images is added to the file, and property information and explanatory text of the still images are displayed on the screen in accordance with user operations. With this additional information, the user can obtain more detailed information about a photograph of a photographed landscape or person.
Regarding the reproduction of video (moving image), information (metadata) related to the reproduced video is added, and various usage forms have been proposed along with video reproduction.

  As one of the above-mentioned usage forms, an information processing system has been proposed in which video content is received from a content storage portion, played back by a recording / playback device, and distributed to a number of information processing devices via a network (Patent Document 1). This is to disclose an example of the usage form of metadata, and a user acquires content from a broadcast station, a net broadcast station, or a content recording medium acquired by the user himself, and identification information for specifying the acquired content. This is a system for recording and reproducing on an internal recording medium of a recording / reproducing apparatus. Based on the user's instructions, the recording / playback device connects to the search server via the network as necessary, and makes a metadata acquisition request including related information related to the object appearing in the specific scene of the content, and acquires the metadata. To do. A technique for acquiring more detailed information about an object included in a specific scene of content from a WEB server via a network as needed based on the acquired metadata is disclosed.

  This technique achieves an object of providing related information of objects included in a specific scene of content to more users than before.

By the way, Patent Document 1 does not disclose how metadata is displayed on the playback screen. For example, when a user operates a scene of a video that is being played back, the related metadata is displayed at what position and at what timing with respect to the video that is being played back, It is unclear whether it will be displayed in such a form.
JP 2004-272490 A

In general, in a video such as a drama or animation, when a large number of characters appear from one to the next and move and disappear at high speed, the user (such as the relationship and name of the character, how the character appears) It ’s not easy to understand.
In such a case, on the image display screen, if a relationship diagram showing the relationship and current status can be displayed for an object (character, item, BGM, etc.) appearing in the playback image in a part of the screen in synchronization with the playback image, It seems possible to solve the problem.
The present invention has been made in view of the above situation, and the purpose of the present invention can be understood by the user by automatically displaying a relationship diagram of objects currently displayed on the video on the side of the video in synchronization with the flow of the video. It is an object to provide an object-related automatic display method in a video playback system that can easily perform video playback.
Another object of the present invention is to provide a player device that reproduces video used in the object-related automatic display method and a server device that supplies metadata related to the video.

In order to achieve the above object, claim 1 of the present invention provides video playback having video content and metadata corresponding to each video of the video content, and displaying the video and metadata on a display device as the video is played back. A system for drawing a video, a relationship drawing creation / drawing unit for creating a relationship diagram showing a relationship between objects appearing in a video from relationship information constituting metadata related to the video, Synthesizing drawing data of the image drawing means and the relation drawing creation drawing means, and partially overlaps with a display position or video different from the moving image display area in synchronization with the moving image displayed on the display device In this way, a relationship diagram of objects is displayed.
Claim 2 of the present invention is a video reproduction system comprising a content supply device for supplying video, a player device for reproducing video, and a server device for distributing metadata corresponding to each video in response to a request from the player device. The player device requests video rendering means for rendering a video and metadata related to the video to the server device, and from the relation information constituting the metadata sent from the server device, A display device of a player device, comprising: a relationship diagram creation / drawing means for creating a relationship diagram showing a relationship between objects appearing on the display; and a composition processing means for synthesizing drawing data of the video drawing means and the relationship diagram creation / drawing means. In synchronization with the displayed video, the object relationship diagram is displayed so that it partially overlaps the display position or video different from the video display area. It is characterized in.
Claim 3 of the present invention has object trajectory creation / drawing means for creating and drawing a trajectory of each object appearing in the video from the object trajectory information constituting the metadata in the invention of claim 1 or 2, The object trajectory is overlaid on the video.
According to a fourth aspect of the present invention, in the first, second, or third aspect, the metadata includes at least one object relation diagram and trajectory information for each frame of video.
According to a fifth aspect of the present invention, in the invention according to the first, second, third, or fourth aspect, the object relation diagram is formed as a node by forming an area so as to surround a character meaning the object for each object, The nodes are related by drawing with line segments, and the relationship between the nodes is displayed by characters in the vicinity of the line segments.
According to a sixth aspect of the present invention, in the invention according to the third, fourth, or fifth aspect, the image drawing means, the relationship diagram creating / drawing means, and the object locus creating / drawing means are each configured by a frame buffer, and each frame buffer has Each drawing is created by writing a bit representing the image.
According to a seventh aspect of the present invention, in the invention according to the second aspect, the player device includes a metadata storage unit and a metadata synthesis unit, and the metadata synthesis unit configures object relation information and trajectory information. When knowledge information to be supplied is supplied, if the metadata does not exist in the metadata storage means, metadata is stored by storing relationship information or trajectory information of the supplied object, and in the combined metadata The object relation information is passed to the relation diagram creation / drawing means, and the object locus information in the synthesized metadata is passed to the object locus creation / drawing means.
An eighth aspect of the present invention is characterized in that, in any one of the first to sixth aspects, the relationship diagram between the video and the object corresponding to the video is displayed on different display devices in synchronization.
Claim 9 of the present invention is a player device used in a video reproduction system, wherein the player device requests video rendering means for rendering a video and metadata relating to the video to the server device, and the server Relationship diagram creation / drawing means for creating a relationship diagram showing a relationship between objects appearing in a video based on relation information constituting metadata sent from the apparatus, and drawing data of the video drawing means and the relationship diagram creation / drawing means And a synthesis processing means for synthesizing.
According to a tenth aspect of the present invention, in the ninth aspect of the invention, an object trajectory creation / drawing is performed in which a trajectory of each object appearing in the video is created and drawn from the object trajectory information constituting the metadata sent from the server device. It has the means.
An eleventh aspect of the present invention is a server device used in the video reproduction system according to the second aspect of the present invention, wherein the server device is connected to the player device via a network or directly, and the player device. It is characterized in that video-compatible metadata is supplied in response to a request from.
According to a twelfth aspect of the present invention, in the invention according to the eleventh aspect, the server device has the database, and retrieves object-related information from the database, and retrieves object trajectory information from the database. Object trajectory information search means and the object search information are converted into XML format and supplied to the player device.

  According to the above configuration, since the relationship and state between objects such as characters and items appearing on the moving image are automatically displayed as a relationship diagram on the same display screen in synchronization with the moving image, the user can Thus, it is possible to easily determine the state of the main character. For example, in the case of a drama, it is possible to deepen understanding of the transition of the contents for each screen.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a system schematic diagram for explaining an object-related automatic display method in a video reproduction system according to the present invention.
A large number of player devices 3 a to 3 n can access the server device 1 via the network 2 to obtain video and metadata related to the video. The player devices 3a to 3n can download video content from the content server device 1b by operating the operation units 5a to 5n, and can display the downloaded video content on the display devices 11a to 11n. For each video content displayed on the display devices 11a to 11n, the metadata is requested from the metadata management server device 1a, and a moving image is displayed on the screens of the display devices 11a to 11n by the metadata and is synchronized with the moving image. The relationship diagram of the objects appearing in the video can be displayed.

The metadata distributed by the metadata management server device 1a can be created by the authoring device 4, and can be updated to appropriate metadata corresponding to each video content.
The content server device 1b has a database including an HDD or the like that stores a large number of video contents such as dramas, and can provide video content in accordance with the requests of the player devices 3a to 3n. Since the content server device 1b provides video content to the player devices 3a to 3n, it may be stored in a storage medium that can be directly connected to the player devices 3a to 3n, or the player devices 3a to 3n. In addition, a DVD disk deck or a large-capacity semiconductor memory device can be built in and read from the disk or semiconductor memory to obtain video content.
The server device 1 and the player devices 3a to 3n can be connected directly without going through a network.

FIG. 2 is a block diagram showing an embodiment of a physical configuration of the player device used in the present invention.
The player devices 3a to 3n include a player control circuit 8, a timer 7, an input processing unit 6, a memory unit 9, an image processing unit 10, and a network communication unit 12.
The player control circuit 8 controls the entire player device, and also includes a control unit 8a, a video control unit 8b, an object locus hit determination unit (key selection rectangle) 8c, an object locus hit determination unit (mouse coordinates) 8d, and a metadata analysis unit. 8e and the metadata composition unit 8f. The memory unit 9 includes areas of a metadata storage unit 9a for storing metadata acquired from the server device 1 and a selection information storage unit 9b for storing information selected by the user through the operation of the operation unit 5. It has a storage area for data necessary for controlling the devices 3a to 3n.

  The image processing unit 10 includes a video drawing unit 10a that draws the acquired video, a grid selection region drawing unit 10b that draws a grid selection region, a relationship diagram drawing unit 10c that draws a relationship diagram on another part of the screen that displays a moving image, and a moving image. The object locus drawing unit 10d for drawing the locus of the object and the composition processing unit 10e are configured. The relationship diagram creation / drawing means for creating and drawing the relationship diagram is realized by cooperation of the metadata composition unit 8f, the metadata storage unit 9a, the relationship diagram drawing unit 10c, and the like. Similarly, object trajectory creation / drawing means for creating and drawing a trajectory of a moving image object is realized by cooperation of the metadata composition unit 8f, the metadata storage unit 9a, the object trajectory drawing unit 10d, and the like.

FIG. 3 is a block diagram showing an embodiment of a physical configuration of a server device used in the present invention.
The server device 1 includes a network communication unit 21, a server control circuit 22, a database control unit 24, a memory unit 25, an input control unit 27, and an image processing unit 28.
The server control circuit 22 controls the entire server device, and also includes a parameter analysis unit 22a, an image information analysis unit 22b, an object information search unit 22c, an object trajectory information search unit 22d, an object area collision determination unit 22e, and a response output generation unit. 22f and an XML conversion unit 22g.

The network communication unit 21 includes an output unit 21a, a network interface 21b, and an input unit 21c.
The database 23 includes storage areas for a moving image data storage unit 23a for storing video content and a metadata storage unit 23b for storing metadata of the video content. The database 23 also stores moving images, and the server device 1 also functions as the content server device 1b.
A display device 29 is connected to the image processing unit 28, and an operation device 26 is connected to the input control unit 27. Data is input to the server device 1 from the operation device 26 to control the server device. Can be displayed on the display device 29 via the image processing unit 28. The memory unit 25 is a RAM or the like used for the processing. These device parts constituting the input / output are for adjusting the server device 1 separately from the authoring device 4 shown in FIG.

Next, transmission / reception of signals between the functional units of the video reproduction system according to the present invention and data structures handled will be described.
FIG. 4 shows the information transmission / reception relationship between the function units of the player device and the server device for easy understanding.
An input signal such as an operation from the input processing unit 6 is supplied to the control unit 8a, and the output of the control unit 8a is an image control unit 8b, an object locus hit determination unit (key selection rectangle) 8c or an object locus hit determination unit (mouse coordinates). ) Input to 8d. The output of the timer 7 is supplied to the video control unit 8b and the network communication unit 12. The selection information storage unit 9b is for storing selection information used by the object locus hit determination unit (key selection rectangle) 8c for determining whether or not the selection region and the object locus are hit. The metadata storage unit 9a is used by the object locus hit determination unit (key selection rectangle) 8c, the object locus hit determination unit (mouse coordinates) 8d, and the metadata composition unit 8f. The network communication unit 12 transmits a trajectory hit determination to the network interface 21b of the server device based on outputs from the object trajectory hit determination unit (key selection rectangle) 8c, the object trajectory hit determination unit (mouse coordinates) 8d, and the timer 7. The information received from the server device is analyzed by the metadata analysis unit 8e and sent to the metadata synthesis unit 8f. The video drawing unit 10a inputs the moving image information from the video control unit 8b, and the grid selection area drawing unit 10b inputs the grid information from the object locus hit determination unit (key selection rectangle) 8c. Further, the relationship diagram drawing unit 10c and the object trajectory drawing unit 10d receive the metadata from the metadata synthesis unit 8f, create respective drawing information, and pass it to the synthesis processing unit 10e. The composition processing unit 10e is connected to a display device and displays a moving image and a related diagram thereof.

  In the server device, the network interface 21b exchanges information with the network communication unit 12, and sends the received signal to the parameter analysis unit 22a via the input unit 21c. As a result of analyzing the received information, the parameter analysis unit 22a sends each information to the image information analysis unit 22b, the object information search unit 22c, or the object trajectory information search unit 22d according to the analysis content. The pixel information analysis unit 22b reads the corresponding moving image from the moving image data storage unit 23a according to the pixel information, and sends it to the object area collision determination unit 22e. The object information search unit 22c searches the metadata storage unit 23b for the object information requested via the database control unit 24. Further, the object trajectory information search unit 22d searches the metadata storage unit 23b for the object trajectory information requested via the database control unit 24. The object area collision determination unit 22e determines whether or not the object trajectory information from the object trajectory information search unit 22d overlaps with the grid information from the pixel information analysis unit 22b. Send the object information. The response output generation unit 22f aggregates the object information and the object trajectory information from the object trajectory information search unit 22d and the object information search unit 22c and sends them to the XML conversion unit 22g. The XML conversion unit 22g converts each information into the XML format. The data is transferred to the network interface 21b via the output unit 21a.

FIG. 5 is a diagram illustrating an example of a data structure of metadata transmitted and received according to the present invention.
The moving image file 40 has knowledge information 50, and the knowledge information 50 is configured by information obtained by collecting a large number of user information 60, relationship information 70, and trajectory information 80. The relation information 70 is formed from a relation sentence 75, a relation information section 76, and object information 77, and the trajectory information 80 is formed from a trajectory information section 85 and trajectory area information 86.
The moving image file 40 includes an ID item indicating the location, a name item indicating the moving image file, a URL item of the moving image file indicating the file name of the moving image file, and an associated knowledge information 50 item. A plurality of pieces of knowledge information can be listed.
The knowledge information 50 includes an ID item of the knowledge information, a name item such as a drama name, an ID of knowledge information of a moving image file, an item of a creator such as a drama, and a document used for authoring. (Multiple description is possible), items that list the relationship information 70, and items that list the trajectory information 80.

The user information 60 includes items of a user ID, a user name, a login ID for identifying the user, and a password.
The relational information 70 is centered on the relational sentence 75, and the object information 77 defined by the relational sentence 75 is defined by the predicate relationship indicated in the relational information section 76. , A related sentence name item, a main case object ID, a target case object ID item, and a predicate ID item that explains them. The object information 77 is, for example, a character such as an actor to be clicked by the user, an item used by the character, BGM, and the like. The object ID item, the object name item, and the object description image displayed in the relationship diagram And items for explaining the object (words displayed in the relation diagram when clicked) are described. A plurality of items of the image and description can be listed. The relationship information section 76 describes the relationship between objects, and prescribes “use” or the like if the relationship between a person and an item or the like if it is a person relationship diagram. It is composed of an ID item such as a person, a name item such as a person, an image item for easy understanding by authoring, and an explanatory item in which an explanatory text for making the relationship easy to understand by authoring or the like.
The relationship information 70 is aggregate data obtained by stacking these data.

The trajectory information 80 is information for moving the trajectory of the object in accordance with the motion of the video, and the trajectory information unit 85 is an information unit listing the movements of a certain object at a certain time.
The trajectory information unit 85 is an item for identifying an object, a URI item indicating the position of the object, a position in the Z direction of the object (when multiple objects appear and overlap on the same screen, it is for determining which is the front side. ) Indicating the Z depth, the time when the object appears and the time when the object disappears in a part of the scene, and the area information 86 at a certain time. The area information 86 defines the movement of the area at a certain time, and a plurality of areas can be listed. The area information 86 is a sequence number indicating the item of the ID of the object, the area is displaced, and the order in which the area is displaced (for example, if the moving area has changed five times from the appearance to the disappearance of the object). It consists of items, items indicating the time of reproduction at the time of displacement (number of frames from the start of reproduction) and area parameters.

  The area parameter is an item that defines a shape such as an ellipse or a rectangle. If the shape is a quadrangle, for example, an X coordinate and a Y coordinate item that indicate the center coordinate, and a width and height that indicate the width and height of the rectangle. It consists of an item and a rotation angle item indicating a square rotation angle. When the shape is an ellipse, it is similarly defined. In such a case, a rectangle circumscribing the ellipse is used as a reference. The trajectory information 80 is aggregate data obtained by stacking these data.

4 will be described with reference to FIGS. 6A to 6I (player device) and FIGS. 7A to 7F (server device).
FIG. 6A is a flowchart for explaining the configuration of the input processing unit of the player device.
The input processing unit 6 monitors whether or not the user has operated the operation unit (step (hereinafter referred to as “S”) 6A01), and if there is an input, passes the input information to the control unit 8a (S6A02). The control unit 8a in FIG. 6B receives the input information (S6B01), and determines whether the input information has 10 keys or direction keys (S6B02). If the key is 10 keys or direction keys, the input information is passed to the object locus hit determination unit (key selection rectangle) 8c (S6B03).
On the other hand, if the input information is a mouse or a button, it is determined whether the input data is a mouse click on the video (S6B04), and the mouse coordinate (input information) is set in the object locus hit determination unit (mouse coordinate) 8d. Hand it over (S6B05). If it is not a mouse click, it is determined whether the video is to be reproduced or the stop button is pressed down as input information by the button (S6B06). If the video is to be reproduced, the information is passed to the video controller 8b. If (S6B07), video playback is not performed, the process ends.

When receiving the button information from the control unit 8a (S6C01), the video control unit 8b in FIG. 6C determines whether or not it is a playback button (S6C02), and if it is a playback button, performs video data playback processing (S6C03). Then, it is passed to the video drawing unit 10a (S6C06). On the other hand, if it is not a playback button, it is determined whether or not a stop button has been pressed (S6C04). If it is a stop button, a video stop process is performed (S6C05). If the stop button has not been pressed, the video data is continuously passed to the video drawing unit 10a (S6C06).
When the object locus hit determination unit (mouse coordinates) 8d in FIG. 6D receives the mouse coordinates from the control unit 8a (S6D01), the locus information is read from the metadata storage unit 9a (S6D02). Then, the optimum radius r of the circle is acquired from the current movement of the object locus.
For example, if the object is stop information, the radius r is small, and if the object is moving, the radius r is increased according to the speed of movement (S6D03). As a result, a circle area having a radius r centered on the mouse coordinates is determined as the selected area (S6D04). Then, the process proceeds to determination of whether or not the trajectory information of the object in the video overlaps the selection area selected by the user with the mouse, that is, whether or not it is hit (S6D05, S6D06). If there is even a slight overlap, a hit determination is made. For example, in the case of FIG. 6D-1, since the selection area and the trajectory area slightly overlap, it is determined that the hit has occurred.
In the case of a hit, the object information is passed to the network communication unit 12 (S6D09). If not hit, the video is returned by one frame time and the same determination is made in S6D06. If there is still no hit, a similar determination is attempted up to a predetermined n frame (S6D06, S6D07, S6D08). The n frame value is set to the time that the object will move from when the user recognizes the object until the user clicks the mouse. If there is no hit even after repeating up to n frames, it is assumed that the user has not caught the object and the process ends.

On the other hand, when the object locus hit determination unit (key selection rectangle) 8c in FIG. 6E receives key information (input information) received from the control unit 8a (S6E01), Otherwise, it is determined whether it is a mouse button (S6E02). If it is a 10 key / direction key, it is determined whether the key operation is a determination key or a selected key (S6E03). If it is not a key, it is determined whether or not it has been pressed with the same key as before (S6E04). If it is determined that the same key as the previous operation has been operated (determining operation is performed if the same key is pressed twice in succession), the process proceeds to S6E11 because it has been determined. If it is not the same key operation as the previous time, it is determined whether or not there is an already stored selection start time (time when the area selection key is pressed) (S6E05), and the selection start time is stored. If so, the area corresponding to the pressed down 10 key / direction key is selected (S6E08). If the selection start time is not stored (nothing is selected by the 10 key / direction key), the time when the 10 key / direction key is pressed this time is stored as the selection start time, and is pressed down for the entire screen. The area corresponding to the 10 key / direction key is set as the selection area (S6E06, S6E07). Then, the selection area determined in S6E07 or S6E08 is stored as the current selection area (S6E09), and a selection range change event is issued to the grid selection area drawing unit 10b (S6E10). Thereby, the grid area of the video is displayed in, for example, a specific color (red or the like).
FIG. 6E-1 shows a display example of the grid area in the video. The video display area is divided into a large number, and a plurality of 10 keys are assigned to the divided areas, and the divided area to which the pressed key belongs when one of the keys is pressed is, for example, red. This indicates that it has been selected.
When this process ends, the process returns to S6E02 for determining whether or not the key is a region selection key.
If the determination key is determined in S6E03, the time is returned to the stored selection start time (S6E11), and only the time from the user's recognition of the object to the key input is returned (S6E12). The time from the user's object recognition to key input takes into account the human reaction time, and is returned, for example, from about 0.2 seconds to about 0.5 seconds.
Then, the position information of the object trajectory at the returned time is acquired, and hit determination is performed based on whether or not there is an overlapping portion between the object trajectory and the current selection area (S6E13, S6E14), and a selection range change event is generated. It is issued to the grid selection area drawing unit 10b, and the display indicating that it is selected is canceled (red is eliminated) (S6E15). In the case of a hit, the hit object information is passed to the network communication unit 12. If there is no hit, the process ends (S6E16, S6E17).
By this processing, it is determined whether or not the area selected by the user has captured an object in the video.

The network communication unit 12 in FIG. 6F determines whether or not time information from the start of playback to the present time has been passed from the timer 7 for video playback (S6F01), and if the current video playback time is passed, Request the object trajectory information of time to the server device (network interface 21b) (S6F02, S6F03). When the XML of the object trajectory information is received from the server device, the acquired XML is passed to the metadata analysis unit 8e (S6F04, S6F05). On the other hand, when a click is made, it is determined whether or not the object information is passed from the object locus hit determination unit (mouse coordinates) 8d. When the object information is passed, this information is received and the object information 77 Using the ID as a key, the server device is requested for the related information 70 (S6F06, S6F07, S6F08). When the XML of the object relation information is received from the server device, the acquired XML is passed to the metadata analysis unit 8e (S6F09, S6F10).
If mouse coordinates instead of object information are passed from the object trajectory hit determination unit 8d in S6F06, the mouse coordinates are received (S6F11), and a hit result object is requested to the server device (S6F12). When the hit result of the object is received from the server device, it is determined whether or not it is hit. If it is hit, the process proceeds to S6F08 (S6F13, S6F14).
In this network communication, information is obtained little by little from the server device while being reproduced by the player device, and information is obtained every few seconds (for example, every 2 seconds).

The metadata analysis unit 8e in FIG. 6G to which the XML data is passed from the network communication unit 12 extracts the metadata (object trajectory information and relation information) from the XML (S6G01, S6G02), and the metadata is the metadata composition unit 8f. (S6G03).
When the metadata composition unit 8f in FIG. 6H receives the metadata (S6H01), it is checked whether or not there is metadata to be added to the metadata storage unit 9a for all metadata, and the metadata is stored. If it is not stored, it is stored, and if the same metadata is already stored, it is discarded. The above processing is repeated for the metadata sent in predetermined units (a group of result data for the request) (S6H02, S6H03, S6H04). By this processing, for example, data between 100 frames to 150 frames is added to the object trajectory information 80 in time series, and the object relation information 70 is similarly added and synthesized in time series.
When the synthesis of a set of metadata is completed, the synthesized metadata is read from the metadata storage unit 9a, the relationship information 70 is sent to the relationship diagram drawing unit 10c, and the object track information 80 is sent to the object track drawing unit 10d. Drawing data is written (S6H05, S6H06, S6H07). The metadata composition unit 8f repeatedly performs the composition process on the metadata sent next.

When the relationship information is passed from the metadata synthesis unit 8f, the relationship diagram drawing unit 10c in FIG. 6I creates an image for drawing the relationship diagram by writing the relationship information in the frame buffer (S6I01, S6I02, S6I02). Similarly, the object trajectory drawing unit 10d in FIG. 6I creates an image for drawing a trajectory on the video by writing to the frame buffer (S6I04, S6I05, S6I06). Further, when the grid area information is passed from the object locus hit determination unit (key selection rectangle) 8c, the grid selection area drawing unit 10b creates an image for drawing the grid by writing the grid area information in the frame buffer ( S6I07, S6I08, S6I09).
The composition processing unit 10e performs a process of superimposing the image of the image drawing unit 10a, the grid image of the grid selection region drawing unit 10b, the locus image of the object locus drawing unit 10d, and the relationship diagram image of the relationship diagram drawing unit 10c. (S6I10, S6I11, S6I12, S6I13, S6I14), and this image is displayed on the display device (S6I15).
FIG. 6I-1 shows an example of a composite image displayed on the display device.
The ellipse circumscribing the object is the trajectory area, and the circle area partially overlapping the trajectory area is the grid area clicked by the user. The relationship diagram is displayed on the right side of the video. The relationship diagram is displayed at the same time corresponding to the object of the image, and can be displayed on other parts (upper side, left side, right side, etc.) of the video portion of the display position, and is partially displayed. It is also possible. If the display device has two displays, it is possible to display an image on the first display and a relationship diagram on a second display installed nearby. Moreover, the screen display of the trajectory may be displayed as necessary, so that the user can visually confirm it, or may not be displayed at all. If no display is made, the locus drawing unit is not operated.

Next, the configuration of the server device will be described.
The input unit in FIG. 7A is a flowchart for explaining the configuration of the input unit and the parameter analysis unit of the server device.
The input unit 21c starts waiting for a request, and when receiving a request from the player device, passes the request to the parameter analysis unit 22a (S7A01, S7A02, S7A03). The parameter analysis unit 22a receives a request from the input unit 21c and analyzes a character example of the request. That is, it interprets what kind of request is made (S7A04, S7A05).
The parameter analysis unit 22a functions as a controller of the server device 1 and executes processing for the four requests (1), (2), (3), and (4) from the player devices 3a to 3n. The request (1) requests static knowledge information from the player device, and in the case of such a request, the request is passed to the object information search unit 22c (S7A06, S7A07). If it is not a request for static knowledge information, it is determined whether it is a request for object trajectory information. If it is a request for object trajectory information 80, the request is passed to the object trajectory information search unit 22d (S7A08, S7A09). . The request (3) is a flow when the server device has an object locus hit determination unit. In this embodiment, the server device has an object area collision determination unit 22e (if the player device has an object hit determination unit, it is actually In the following description, the case of hit determination on the server apparatus side will also be described. As an example in which the server device has an object locus hit determination unit, this is applied to a case where the player device is small and the load is reduced.

FIG. 8 shows specific examples of various requests (1), (2), (3), and (4).
The static information acquisition request (1) is made up of video content ID and user ID data.
The object trajectory data acquisition request (2) is a request for the number of frames to which trajectory information, and is composed of video content ID, user ID, start time and end time of the requested video frame. Yes.
The object hit determination request (3) is a case where the server apparatus determines a hit as described above.
There are the following six request contents, and the description contents are an example. Note that the delayed click described below is to perform determination while returning one frame at a time from the input of the click signal until a predetermined time, and complements the delay in the operation of the user.
Each of them includes the video content ID, the user ID, and the hit time. The request on the first line shows the start time 628 of delayed click (the time to go back in consideration of the delay), the end time 688 and the position of the target coordinate (X coordinate 382, Y coordinate 257), and only 60 frames. The determination is made retroactively. In the second line, there is no delayed click, only the clicked position (X coordinate 416 and Y coordinate 154) of the target time 520 is shown, and it does not go back. The third line performs hit determination with a determination width of radius 10 in addition to the description of the second line (mouse click). The fourth line is used for determining hit of an area, and is a determination request for a rectangular area having a width of 213 and a height of 160 as an area (area click). The fifth line is a request that the delayed click goes back to 0 frame and the determination range is radius 8 (mouse click). The sixth line is a determination request for a square area with a delayed click going back to 441 and a width of 80 and a height of 60 (area click).
The request (3) is a case where delayed click and area click can be turned on / off by the player device.
The object detail data acquisition request (4) is a request for an object hit by the player device to the server device. The hit video content ID, the user ID, the clicked video frame, and the object in the clicked video Thus, the relation information of the object hit by clicking is obtained.

When the image information analysis unit 22b in FIG. 7B receives the search condition from the parameter analysis unit 22a, the image information analysis unit 22b reads the moving image data according to the search condition from the moving image data storage unit 23a (S7B01, S7B02). Then, the motion analysis of the image is performed, the motion speed per second of the image is calculated, and passed to the object region collision determination unit 22e as a region parameter (S7B03, S7B04).
This is a process in which the server device creates the click area radius r when it is not specified. In other words, it corresponds to the request (3) and determines how much radius should be taken from the motion of the screen and the motion of the trajectory (which video of which video in the video data), and is included in the video data. The moving image is analyzed, the motion is analyzed, and the optimum r is calculated.
When the object information search unit 22c in FIG. 7B receives the search condition from the parameter analysis unit 22a, the object information search unit 22c passes the data to the database control unit 24 and acquires the result (S7B05, S7B06, S7B07). Then, it is determined whether or not there is related information in the object information (S7B08). If there is related information, the database control unit 24 is requested to search for related information (S7B09), and the result is acquired (S7B10). The obtained result is passed to the response output generation unit 22f (S7B11). The request (4) is for obtaining detailed object information and for obtaining a relational sentence of the object. That is, the object information 77 and the information of the relationship information section 76 are obtained for the object in the relationship sentence 75.

  The object trajectory information search unit 22d in FIG. 7C performs the same search for the object trajectory as the object information search unit 22c. The object trajectory information search unit 22d receives a search condition from what frame to what frame from the parameter analysis unit 22a. (S7C01, S7C02). The database control unit 24 accesses the metadata storage unit 23b with the conditions and reads the trajectory information 80 that meets the search conditions (S7C03). Here, it is determined whether or not the search is for the hit determination request (3) (S7C04). If it is for the hit determination, the trajectory information is sent to the object collision determination unit 22e as a candidate for the hit object. hand over. In the case of the object locus data acquisition request (2), the obtained result is passed to the response output unit 22f (S7C06).

When the object region collision determination unit 22e in FIG. 7D receives the object locus information of a candidate for determination (a locus that may exist at that time) from the object locus information search unit 22d (S7D01), the object region collision determination unit 22e collides with respect to which coordinate from the parameter analysis unit 22a. Information on whether to make a determination (whether it overlaps with the trajectory information) is received (S7D02). Then, it is determined whether the object of collision determination is a point or an area (S7D03), and if it is an area, the collision determination target area is set as a click recognition area (S7D04). On the other hand, if it is a point, it is further determined whether the area click function is enabled (ON state) (S7D05). If it is not enabled, an area consisting only of the collision determination point is created and clicked. It is set as a recognition area (S7D06). If it is valid, the area click parameter (radius r, etc.) is received from the pixel information analysis unit 22b, and the click recognition area is calculated (S7D07, S7D08).
After performing the click recognition area in this way, the collision object trajectory list is initialized (S7D09). Then, the position of the object at the determination time (frame number) is calculated, and it is determined whether the object trajectory and the click recognition area have a common area (overlapping part) for each of the collision candidate object trajectories. If it exists, it is added to the collision object trajectory list (S7D10, S7D11, S7D12, S7D13).
If such an operation is repeated for each of the collision candidate object trajectories and terminated (S7D14), it is determined whether or not the delayed click function (returning from the time when the user clicked) is enabled (S7D15). If it is not valid, the collision object trajectory list is directly passed to the object information search unit 22c (S7D18).
When the delayed click is valid, the determination time is returned by one frame (S7D16), and it is determined whether this frame is (initially given determination time-current determination time), that is, n seconds (S7D17), and n If it is not (number of frames to be returned), the process returns to S7D10 to make a collision determination. When it is n seconds, the process exits the loop of S7D10 to S7D17 and passes the collision object trajectory list to the object information search unit 22d (S7D18).

Here, the details of the calculation of the position of the object at the determination time of S7D10 will be described using the A flow of FIG. 7E. The target object and the time are received, and the trajectory information including the time is acquired from the object trajectory information search unit 22d (S7E01, S7E02). That is, information for calculating what frame position for which object information is acquired.
First, it is determined whether or not the trajectory information of the target object exists (S7E03), and if the object trajectory does not exist (such as when the object trajectory has returned too much), the process ends (S7E04). When the trajectory information exists, it is determined whether the time coincides with the time indicated by the definition area in the trajectory information (S7E05). If there is a matching area, the definition area is directly used as the object area (S7E09). For example, if the definition area includes the 100th frame, the 150th frame, and the 180th frame, and the clicked determination time is the 150th frame, the 150th frame exists in the definition area. Is used as is.
If it does not exist in the definition area, the definition area at the time immediately before (= a) and the time immediately after (= b) from the time (= t) is extracted (S7E06). For example, if the determination time is the 130th frame, it does not exist. Therefore, in order to calculate the trajectory information of the 130th frame, the information of the immediately preceding 100th frame and the immediately following 150th frame is extracted. Hereinafter, a proportional coefficient ρ is calculated from the ratio of (ta) and (b−t), and an area is created by firstly complementing each parameter of the area with the proportional coefficient ρ, which is defined as an object area (S7E07, S7E08). The specific position of the definition area to be created varies, for example, in units of pixels with respect to the previous and subsequent definition areas depending on the speed of change of the image.
The object area thus obtained is called and returned to the original (S7E10).

FIG. 7F is a flowchart for explaining the configuration of the response output generation unit, the XML conversion unit, and the output unit.
The response output generation unit 22f acquires the object information from the object information search unit 22c and the object trajectory information search unit 22d (S7F01), and acquires the request type from the original request (S7F02). Then, the object is shaped according to the request type (S7F03). For example, if the object trajectory information has a definition area of 50 frames, 100 frames, 200 frames, 300 frames, and 400 frames, and the request is from 100 frames to 300 frames, the 50 frames and 400 frames are discarded.
The shaped object information is passed to the XML conversion unit 22g (S7F04).
When the XML conversion unit 22g receives the object information from the response output generation unit 22f (S7F05), the XML conversion unit 22g reads the XML schema, activates the object-XML conversion engine, converts the object information into XML data, and outputs the obtained XML data. To the unit 21a (S7F06, S7F07, S7F08). When acquiring the XML data, the output unit 21a outputs the XML data to the output stream (S7F09, S7F10).

The above is the description of the configuration and operation of each functional unit of the player device and the server device. Next, the processing flow of the entire video reproduction system according to the embodiment of the present invention will be described with reference to FIGS. 9 and 9A to 9F. The flow in FIG. 9 is mainly based on the processing of the server device, and the processing sequence is as follows.
When the user operates the player device 3 to request a desired video content (S901), the request is sent to the content supply device 1b, and the content supply device 1b transmits the video content to the player device 3 (S902). ). The player device 3 requests the server device 1a for the relationship information 70 of the video content (video data) (S903). When the server apparatus 1a receives a search request for the related information 70, the server apparatus 1a starts a database search process along the flow of FIG. 9A. Thus, the static relationship information 70 is searched according to the sequence of FIG. 9C (S904), and the obtained relationship information 70 is supplied to the player device 3 (S905). Upon receiving the relation information 70, the player device 3 starts playing the video (S906). The display device automatically displays the relationship diagram together with the video. Next, the player device 3 requests acquisition of the object trajectory information 80 (S907).

The server apparatus 1 starts the database search process along the flow of FIG. 9A. Thereby, the object trajectory information is searched according to the sequence of FIG. 9D (S908), and the obtained object trajectory information is supplied to the player device 3 (S909). In the sequence up to this point, the related information and object trajectory information of the video are sequentially taken into the player device and displayed as the video progresses.
The player device 3 determines whether or not the video is currently being played back, and if the video is not being played back, the player device 3 ends there (S910). If the video is being reproduced, the user's input operation (such as clicking on an object in the video by the operation unit) will be monitored.
In this state, when the area clicked by the user using the input device hits the object locus area, the server apparatus 1 is requested for information on the hit object (S912).
The server device 1 starts the database search process along the flow of FIG. 9A. Thereby, hit object relation information is supplied to the player apparatus 3 (S914). Note that S913 is a step performed when the server device 1b is provided with the object hit determination unit instead of the player device 3, and the operation sequence thereof is shown in FIG. 9E.

  When the hit object information is supplied, the player devices 3a to 3n request the server device 1 for detailed information (object information 77) of the object (S915). The server device 1 starts the database search process along the flow of FIG. 9A. Thereby, the detailed information of the object is searched according to the sequence of FIG. 9F (S916), and the obtained detailed information of the object is supplied to the player device 3 (S917). When the detailed information of the object is supplied, the player device 3 returns to the step of S910, monitors the user's input device during the video reproduction, and repeats the sequence of S912 to S917 if there is an input operation.

Next, FIGS. 9A to 9F showing sub-flowcharts of FIG. 9 will be described.
FIG. 9A shows a flow of standby processing of the server device, which starts at the positions S61, 64, 67, and 70 in FIG. When processing requests are received from the player devices 3a to 3n and the requests can be received, the requests are analyzed. If the content of the request is appropriate, the next database processing is started. If not appropriate, error information is created (S9A01 to S9A06).
FIG. 9B is a flowchart of the database search process in S9A05 of FIG. 9A.
When the server device 1b receives the search condition, it connects to the database 23. If the connection to the database 23 is established, the database 23 is searched for the content and user with the given ID. If there is one that satisfies the above conditions, the database 23 is searched according to the search condition. Then, the user operation history is recorded and the process ends (S9B01 to S9B07).
If it is not possible to connect to the database 23 in S9B03, if there is no condition that satisfies the condition in S9B05, error processing is performed (S9B08).

FIG. 9C is a flowchart of the static relationship information search process in step S904 of FIG.
The search condition is received and the result list is initialized (S9C01, S9C02). Then, all the knowledge information 50 belonging to the content is retrieved and stored in the result list as follows (S9C03). That is, the object information 77 included in the knowledge information is searched as a subject (main character, etc.) and object (enemy character, item, etc.) for all searched knowledge information (S9C04, S9C05). If the retrieved object information is not in the result list, the object information is added to the result list (S9C06, S9C07). Next, the information of the relation information part 76 included in the knowledge information as a predicate is searched. If the searched relation information is not in the result list, the relation information is added to the result list (S9C08, S9C09, S9C10). The steps of S9C04 to S9C10 are repeatedly executed, and when the search is completed for all knowledge information (S9C11), the result list is called back and returned to the original (S9C12).

FIG. 9D is a flowchart of object trajectory information search processing in step S908 of FIG.
The search condition is received and the result list is initialized (S9D01, S9D02). Then, the object trajectory information 80 belonging to the content is retrieved and stored in the result list as follows (S9D03). That is, it is determined whether or not the start time / end time of the trajectory information overlaps the start time / end time given to the search for all the trajectory information searched. If there is no overlap, the process proceeds to S9D10. When overlapping, the area information of the locus is searched (S9D04, S9D05). This is to exclude the condition in this case when the trajectory information is from 100 frames to 200 frames and the frames given by the search are 1000 to 1500, for example. Of the searched area information, information within the time given by the search condition is added to the result list (S9D07). Then, it is determined whether there is a region that matches the search start or end time in the region information in the result list. If there is no region information that matches the search start or end time, the result is the region information that is closest in time outside the search condition. Add to list (S9D08, S9D09). In this process, for example, when region information of 100 frames, 160 frames, and 210 frames exists, if the search conditions are from 100 frames to 200 frames, 210 frames are outside the search conditions, but there are no 210 frames. This is because the area information of 200 frames cannot be calculated.
When the search from S9D05 to S9D09 is repeated in this way and all the trajectory information is searched, the result list is returned to the caller (S9D10, S9D11).

FIG. 9E is a flowchart of the object trajectory information search process of the process of performing hit determination in the server device in the step of S913 in FIG.
The search condition is received and the result list is initialized (S9E01, S9E02). Then, the object trajectory information 80 belonging to the content is searched (S9E03), and an object area collision is determined (S9E04). The result is saved in the list, and the result list is returned to the caller (S9E05, S9E06).

FIG. 9F is a flowchart for explaining the flow of object detailed information search processing in step S916 of FIG.
The search condition (object ID, time) is received and the result list is initialized (S9F01, S9F02).
An object having the object ID specified in the condition is searched (S9F03), and all knowledge information having the object as a subject or object is searched as follows (S9F04). That is, the object information 77 included in the knowledge information is searched for all the searched knowledge information as a subject (main character, etc.) and an object (enemy character, item, etc.) (S9F05, S9F06). If the retrieved object information is not in the result list, the object information is added to the result list (S9F07, S9F08). Next, the information of the relation information part 76 included in the knowledge information is searched as a predicate. If the searched relation information is not in the result list, the relation information is added to the result list (S9F09, S9F10, S9F11). The steps of S9F05 to S9F11 are repeatedly executed, and when the search is completed for all knowledge information (S9F12), the result list is called back to the original (S9F13).

FIG. 10 is a flowchart for explaining the flow of the object-related automatic display method in the video reproduction system according to the present invention.
For example, an explanation will be given of a sequence in which an actor appears on the video being played and at the same time, a relation diagram of actor nodes (objects displayed in the relation diagram) is automatically displayed on the display device. , The relationship diagram drawing unit 10c operates mainly. Note that the control unit 8a mainly performs the following monitoring and determination.
When the user operates the player device 3 for video reproduction, the player device 3 accesses the content supply device and the server device 1, and video data and related information are supplied to prepare the content (S1001). That is, when the user performs a key operation for selecting the video content A1 to be played by operating the player device, a setting file (each data for selecting the video content A1) is formed in the player device. The setting file is read, the content supply device 1b on the Internet is accessed, and the video content A1 is acquired and stored in the memory unit 9 of the player device. Further, the server device 1 is accessed, and the static intelligent data (relation information 70) and object trajectory data (trajectory information 80) corresponding to the video content A1 are retrieved, read from the database 23, and transmitted to the player device 3. The player device 3 also stores this information in the memory unit (S1001).
If the player performs a reproduction operation by this content preparation, the video content A1 is displayed on the display screen. The control unit 8a determines whether or not the acquired video content A1 is being reproduced (S1002). If it has not been reproduced, the sequence ends, and if it is being reproduced, it is monitored whether or not a new object locus has appeared on the video (S1003).

If a new object locus does not appear on the video, the control unit 8a next monitors whether or not the currently appearing object has disappeared from the video (S1007). As long as an object exists in the video, the relationship diagram drawing unit 10c displays the relationship diagram of the object in synchronization with, for example, a video in which an actor appears (S1011). When this operation ends, the process returns to S1002 again, confirms the reproduction of the video, monitors the appearance of a new object, and repeats the drawing process of the relationship diagram (S1003, S1007, S1011).
In step S1003, when a new object locus appears on the video, the control unit 8a acquires the object detailed information from the server device 1 (S1004), and the relationship diagram drawing unit 10c to which the detailed information is passed is related. A node to be displayed in the figure is created (S1005), this node is added to the relationship diagram, and a relationship diagram with an already existing node is created (S1006). This relationship diagram is displayed on the display device.
In step S1007, when the currently appearing object locus disappears from the video, the control unit 8a acquires a node corresponding to the disappeared object (S1008). Then, it is determined whether or not the disappeared object is automatically added (S1009). This is to distinguish between a node added to the relationship diagram by the user's click and a node automatically added to the relationship diagram by this flow. If not automatically added, the process proceeds to S1011 for drawing a relationship diagram. If it has been automatically added, the timer 7 is set, and the node is deleted from the relationship diagram after a lapse of a fixed time (S1010).
As described above, the relationship diagram can be automatically displayed in synchronization with the video reproduced on the display device.

FIG. 11 is a diagram showing an example of a relationship diagram related to a reproduced video and characters displayed on the display device according to the present invention.
In this example, a video 100 at a certain time is simultaneously displayed on the left side of the screen of the display device, and the relationship diagram 102 is simultaneously displayed on the right side of the screen. A main character 101 appears in the video 100. On the other hand, in the relationship diagram 102, a main character (node) 102a in which characters are enclosed in an ellipse is drawn in the center, and trackers (nodes) 102b and 102c in which characters are enclosed in an ellipse are shown on the upper side. Each is drawn. An arrow is drawn from the tracker 102b to the node of the main character 102a, and is displayed with the characters “track”. This indicates that the tracker 102b is tracking the main character 102a. In addition, arrows are displayed between the main character 102a and the tracker 102c node, and the states “attack” and “track” are displayed as characters.

Further, an item 102d and a pistol 102e in which characters are surrounded by an ellipse are displayed below the main character 102a. An arrow is drawn from the main character 102a to the item 102d and the pistol 102e, and the state is displayed as “use”.
In this way, by displaying each node in the relationship diagram and displaying the relationship with arrows, characters, etc., the user can easily understand the state of the video 100 displayed at the current time.

FIG. 12 is a diagram illustrating a state in which the relational diagram is dynamically displayed in synchronization with the movement of the character in the reproduced video.
FIG. 11 shows an example in which the video is displayed on the display device at a certain time. This example shows that the relationship diagram is also changed and displayed in accordance with the motion of the video.
FIG. 12A shows an image 103 at the moment when the main character enters the passage, and a relationship diagram in which the node of the character 104a uses the item 104b is shown correspondingly. Even when there are a large number of characters, only the relationship diagram of objects appearing at that time is displayed, so the relationship diagram itself is not complicated and the user can easily grasp the situation.

FIG. 12B is a screen further advanced from the video in FIG. 12A, and the relationship diagram changes in accordance with the change in the video.
The video 105 is a video of the moment when the main character enters the passage deeply and heads for another branch passage. The relationship diagram 106 shows a situation in which the main character 106a is using the pistol 106b against the tracker 106c. Even if the video 105 alone does not understand what the main character is doing, it can be easily understood from the relationship diagram that the main character uses the pistol for the tracker. Furthermore, it is clear that the tracker 106c is tracking the main character 106a. In the video 105, a tracker 107 is starting to be seen near the entrance of a distant passage.

In the above embodiment, the node displayed as the relationship diagram is described as an example in which the character name is described and surrounded by an ellipse. However, the display method of the node is not limited to such display, and various forms are possible. Can be displayed. For example, a character or symbol may be surrounded by other shapes (circle, square, etc.) instead of an ellipse, or may be described in an animated manner. Moreover, although the arrow was shown as a display example which shows a relationship, you may use another code | symbol. Furthermore, although the state is described in characters, it is also possible to display it more easily by displaying simple pictures such as icons.
Moreover, although the example in which the video content is stored in the database of the content server device on the network and the database of the server device that manages the metadata has been shown, the storage location is a local memory, DVD, or HDD that can be directly connected to the player device It may be stored in a disk device such as Further, the player device itself may be built in.

In addition, the object trajectory hit determination is mainly described in the case where it is provided in the player device (which also describes the case where it exists on the server device side), but the sequence of the object trajectory determination means is not included in the player device. You may provide in the server apparatus side.
Further, the embodiment of the video reproduction system configured by connecting the content server device, the server device, and the player device via the network has been described. However, the functional units of the content server device, the server device, and the player device are included in one device. It is also possible to configure as a built-in device, for example, a video playback device system. In such a case, the data structure relating to the video is the structure shown in the example of FIG. 5, and each functional unit is substantially the same as the configuration described in FIGS. 6A to 6I and FIGS. 7A to 7F. However, the network interface 21b and the network communication unit 12 (see FIG. 4), which are means for communicating between the server device and the player device, are unnecessary. Since this video playback device also has video and metadata, a search unit, etc., the processing burden on the device itself becomes larger than when it is distributed to the server device and the player device, but the video communication processing can be omitted to save the network communication processing. And the display processing speed for displaying the relationship diagram is improved. Furthermore, video, metadata, control data, selection data, etc. can be recorded in a memory medium for each data purpose, and an efficient memory means can be constructed.

  An object-related automatic display method in a video playback system that plays back video content on a player device, and a server device and a player device used in the video playback system, are used for home use and business use.

1 is a system schematic diagram for explaining an object-related automatic display method in a video reproduction system according to the present invention. It is a block diagram which shows embodiment of the physical structure of the player apparatus used for this invention. It is a block diagram which shows embodiment of the physical structure of the server apparatus used for this invention. It is a figure for demonstrating the relationship of the information transmission / reception of each function part in a player apparatus and a server apparatus. It is a figure which shows an example of the data structure of the moving image metadata transmitted / received by this invention. It is a flowchart for demonstrating the structure of the input process part of a player apparatus. It is a flowchart for demonstrating the structure of the control part of a player apparatus. It is a flowchart for demonstrating the structure of the video control part of a player apparatus. It is a flowchart for demonstrating the structure of the object hit determination part (mouse coordinate) of a player apparatus. It is a figure which shows the example judged to have hit because the selection area | region and the locus | trajectory area | region have overlapped slightly. It is a flowchart for demonstrating the structure of the object hit determination part (key selection rectangle) of a player apparatus. It is a figure which shows the example of a display of the grid area | region in an image | video. It is a flowchart for demonstrating the structure of the network communication part of a player apparatus. It is a flowchart for demonstrating the structure of the metadata analysis part of a player apparatus. It is a flowchart for demonstrating the structure of the metadata synthetic | combination part of a player apparatus. It is a flowchart for demonstrating the structure of the grid selection area | region drawing part of a player apparatus, a relationship figure drawing part, an object locus | trajectory drawing part, and a synthetic | combination process part. It is a figure which shows an example of the synthesized image displayed with a display apparatus. It is a flowchart for demonstrating the structure of the input part and parameter analysis part of a server apparatus. It is a flowchart for demonstrating the structure of the image information analysis part of a server apparatus, and an object information search part. It is a flowchart for demonstrating the structure of the object locus | trajectory information search part of a server apparatus. It is a flowchart for demonstrating the structure of the object area | region collision determination part of a server apparatus. It is a flowchart for demonstrating the calculation method of the object position in the specific time of the subflow A of FIG. 7D. It is a flowchart for demonstrating the structure of the response output production | generation part of a server apparatus, an XML conversion part, and an output part. It is a figure which shows an example of the content of each acquisition request and determination request | requirement in this invention. It is a figure for demonstrating the processing sequence of the video reproduction system to which the object related automatic display system by this invention is applied. It is a flowchart for demonstrating the flow of the standby process of a server apparatus. It is a flowchart for demonstrating the flow of a database search process. It is a flowchart for demonstrating the flow of a static knowledge information search process. It is a flowchart for demonstrating the flow of an object locus | trajectory information search process. It is a flowchart for demonstrating the flow of an object hit determination process. It is a flowchart for demonstrating the flow of an object detailed information search process. 5 is a flowchart for explaining a flow of an object-related automatic display method in the video reproduction system according to the present invention. It is a figure which shows the example of the related figure regarding the example of the reproduction | regeneration image | video displayed on a display apparatus by this invention, and its character. It is a figure which shows the example which synchronizes a relation figure according to a motion of the character in a reproduction | regeneration image | video, and displays it dynamically.

Explanation of symbols

1 Server device 1a Metadata management server device
1b Content server device
2 network
3 (3a-3n) Player device
4 authoring device 5 operation unit 6 input processing unit 7 timer 8 player control circuit 8a control unit 8b video control unit 8c object locus hit determination unit (key selection rectangle)
8d Object locus hit determination part (mouse coordinates)
8e Metadata analysis unit
8f Metadata composition part 9 Memory part (semiconductor memory)
9a Metadata storage unit 9b Selection information storage unit 10 Image processing unit 10a Video drawing unit 10b Grid selection area drawing unit 10c Relationship diagram drawing unit 10d Object locus drawing unit 11 Display device 12, 21 Network communication unit 21a Output unit
21b Network interface 21c Input unit 22 Server control circuit 22a Parameter analysis unit
22b Pixel information analysis unit
22c Object information search unit 22d Object trajectory information search unit 22e Object area collision determination unit
22f response output generation unit 22g XML conversion unit 23 database 24 database control unit 25 memory unit 26 operation device 27 input control unit 28 image processing unit 29 display device 40 moving image file 50 knowledge information 60 user information 70 relation information 75 relation sentence 76 relation information Part 77 Object information 80 Trajectory information 85 Trajectory information part 86 Trajectory area information

Claims (12)

A video playback system having video content and metadata corresponding to each video of the video content, and displaying the video and metadata on a display device as the video is played back,
Video rendering means for rendering video;
A relationship diagram creating / drawing means for creating a relationship diagram showing a relationship between objects appearing in the video from relationship information constituting metadata related to the video;
Combining processing means for combining drawing data of the video drawing means and the relation diagram creation drawing means,
An object-related automatic display in a video playback system, wherein a relational diagram of objects is displayed so as to partially overlap a display position or video different from the video display area in synchronization with the video displayed on the display device method. A video playback system comprising a content supply device that supplies video, a player device that plays back video, and a server device that delivers metadata corresponding to each video in response to a request from the player device,
The player device includes:
Video rendering means for rendering video;
Requests metadata related to the video to the server device, and creates a relationship diagram drawing that creates a relationship diagram showing the relationship between objects appearing in the video from the relationship information constituting the metadata sent from the server device Means,
Combining processing means for combining drawing data of the video drawing means and the relation diagram creation drawing means,
An object relation in a video reproduction system, wherein a relation diagram of objects is displayed so as to partially overlap a display position or video different from the video display area in synchronization with a video displayed on a display device of a player device Automatic display method. Object trajectory creation and drawing means for creating and drawing a trajectory of each object appearing in the video from the object trajectory information constituting the metadata,
3. The object-related automatic display method in the video reproduction system according to claim 1, wherein the object trajectory is displayed as an overlay on the video.   4. The object-related automatic display method in a video reproduction system according to claim 1, wherein the metadata includes an object relation diagram and trajectory information in at least one frame unit of the video.   In the object relationship diagram, for each object, an area is formed so as to enclose characters that represent the object, and nodes are formed. The nodes are drawn by line segments, and the relationship between the nodes in the vicinity of the line segment. 5. The object-related automatic display method in the video reproduction system according to claim 1, 2, 3, or 4, wherein the character is displayed in characters. The video drawing means, the relationship diagram creation drawing means, and the object trajectory creation drawing means are each constituted by a frame buffer,
6. The object-related automatic display method in the video reproduction system according to claim 3, wherein each drawing is created by writing a bit representing each image in each frame buffer. The player device has metadata storage means and metadata composition means,
When the metadata composition means is supplied with knowledge information constituting object relation information and trajectory information, if the metadata does not exist in the metadata storage means, the supplied object relation information or trajectory information Synthesize metadata by storing
The relationship information of the objects in the synthesized metadata is sent to the relationship diagram creation / drawing means.
3. The object-related automatic display method in the video reproduction system according to claim 2, wherein the object locus information in the synthesized metadata is passed to the object locus creation / drawing means.   7. The object-related automatic display method in the video reproduction system according to claim 1, wherein the relationship diagram between the video and the object corresponding to the video is displayed on different display devices in synchronization with each other. . A player device used in a video reproduction system,
The player device includes:
Video rendering means for rendering video;
A relationship diagram creation / drawing means for requesting metadata related to the video to a server device and creating a relationship diagram showing a relationship between objects appearing in the video based on relationship information constituting metadata sent from the server device. When,
A player apparatus in a video reproduction system, comprising: synthesis processing means for synthesizing drawing data of the video drawing means and the relation diagram creation drawing means.   10. The video reproduction according to claim 9, further comprising object trajectory creation / drawing means for creating and drawing a trajectory of each object appearing in the video from object trajectory information constituting the metadata sent from the server device. A player device in the system. A server device used in the video reproduction system according to claim 2,
The server device in a video playback system, wherein the server device is connected to the player device via a network or directly and supplies video-compatible metadata in response to a request from the player device. The server device
Having the database;
Object information retrieval means for retrieving object relation information from the database;
Object trajectory information search means for searching object trajectory information from the database;
12. The server device in the video reproduction system according to claim 11, wherein the object search information is converted into an XML format and supplied to the player device.

Images