JP2004297448A - Device for superimposing and displaying information - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce excess and deficiency in information displayed for viewers by reducing a man-hour to optimize information displayed by a program provider. <P>SOLUTION: A receiving part 12 receives data broadcasting program information, gives image information to a display information determining part 15, and gives information related to an object in an image to a related information holding part 13. A pointing recognizing part 14 recognizes which one of objects displayed in the image is pointed by a user, and gives the recognition result to the display information determining part 15. The part 15 determines the degree of minuteness concerning the related information at each object based on image information, related information with the minuteness degree of each object recorded in the related information holding part 13, and the recognition result given from the pointing recognizing part 14. The display information determining part 15 gives the image information and the related information determined at each object to a display part 16. The display part 16 superimposes the image information on the related information so as to display the resultant information. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an information superimposed display device, and particularly to an information superimposed display device for information related to an object included in a video of a data broadcast program.
[0002]
[Prior art]
In a data broadcast program, information related to an object in a video is transmitted at the same time as the video information, and the receiver displays the relevant information according to the progress of the video.
[0003]
For example, a program receiving unit that receives information including a program and auxiliary information that is information supplementing the program, a display flag management unit that manages a display flag that is a flag indicating whether or not to display the auxiliary information; A separating unit that separates the program and the auxiliary information received by the receiving unit, a program display unit that displays the program separated by the separating unit, and an auxiliary information display unit that displays the auxiliary information separated by the separating unit. There is a technology including an auxiliary information display control unit that reads a display flag of the display flag management unit and instructs the auxiliary information display unit to display auxiliary information when the display flag is on. (Patent Document 1)
[Patent Document 1]
JP 2001-245234 A
[0004]
[Problems to be solved by the invention]
The problem with the above technique is that only a small amount of related information that the viewer wants to know in detail is displayed, or conversely, a large amount of extra related information is displayed.
[0005]
Also, in order to provide as much or as little information as desired by the viewer, the program provider needs to optimize the amount and time of information to be displayed and the priority order with other information in advance. There is an additional problem that the man-hour is enormous.
[0006]
An object of the present invention is to prepare a plurality of pieces of information related to objects in a video for each level of detail, and determine the level of detail of information displayed by the system from the state of the viewer and the operation history, so that the program provider can It is an object of the present invention to reduce the number of steps for optimizing the information to be displayed and to reduce the excess or deficiency of information displayed to a viewer.
[0007]
[Means for Solving the Problems]
A first information superimposing display device of the present invention, which receives video information, data broadcast program information including related information with an object detail level, and outputs video information and related information with an object detail level; A related information holding unit for holding related information with a degree of detail from a receiving unit, a pointing recognition unit for recognizing a pointing position on a screen designated by a user and outputting pointing position information, and a pointing from the pointing recognition unit Based on the position information, the video information from the receiving unit, and the related information with the degree of detail of the object from the related information holding unit, the importance of each object included in the video information is calculated, and based on the calculated importance. The degree of detail for each object is calculated, and the object is determined from the related information with the degree of detail of the object based on the calculated degree of detail. A display information determining unit that determines relevant information to be displayed corresponding to the object, and outputs video information and related information to be displayed corresponding to the object; and a video information from the display information determining unit, and an object corresponding to the object information from the display information determining unit. And a display unit that superimposes and displays the related information displayed by the display unit.
[0008]
A second information superimposed display device of the present invention, which receives video information, data broadcast program information including related information with an object detail level, and outputs video information and related information with an object detail level; A related information holding unit that holds related information with a degree of detail from a receiving unit, a pointing recognition unit that recognizes a pointing position on a screen designated by a user, and outputs pointing position information, a face direction information, and the pointing recognition. The importance calculated for each object included in the video information is calculated based on the pointing position information from the unit, the video information from the reception unit, and the related information with the degree of detail of the object from the related information holding unit. Calculates the level of detail for each object based on the level of detail, and based on the calculated level of detail, the association with the level of detail of the object The display information determining unit that determines the relevant information to be displayed corresponding to the object from the information, and outputs the video information, the relevant information to be displayed corresponding to the object, and the video information from the display information determining unit. A display unit for superimposing and displaying related information to be displayed corresponding to an object; and face image recognition for creating face direction information indicating whether a user's face faces the display unit and outputting the face direction information to the display information determination unit And a part.
[0009]
The third information superimposed display device of the present invention is the first information superimposed display device, wherein (a) if there is an object at a position where the user is pointing, the object is the most important object. (B) If there is no object at the position where the user is pointing, a score is calculated for each object, the one with the highest score is made the most important object, and (c) the score for each object is Is determined in consideration of the size of the display area, the proximity to the center of the screen, and the ratio of continuous display, and (object importance) = (weight of the increase when the object is the most important object) ) * (Increase when the object is the most important object) + (weight of object score) * (object score + (Weight of increase when object is pointed) * (increase when object is pointed) + (weight of number of frames in which screen is continuously pointed) * (continuous screen) The display information determination unit is calculated by the following formula: (the number of frames pointing to) + (weight of the number of objects included in the video) * (numerical value determined from the number of objects included in the video) And
[0010]
A fourth information superimposed display device according to the present invention is the third information superimposed display device, wherein the size of the display area of the object, the proximity to the center of the screen, and the ratio of continuous display are weighted. And a display information determining unit that calculates the score of the object by adding them together.
[0011]
The fifth information superimposed display device of the present invention is the fourth information superimposed display device, wherein (a) the importance is increased if the object is the most important object, and (b) the importance is increased according to the score of the object. (C) If the object is pointed, the importance is increased by a certain amount. (D) If somewhere on the screen is pointed continuously, the importance is increased. ) The importance is increased or decreased according to the number of objects included in the video. (F) Whether the object is the most important object, the score of the object, whether the object is pointed, and how continuously the screen is pointed , And the value to be increased or decreased from the number of objects included in the video, weighted and added, It characterized by having the display information determination unit which determines the.
[0012]
A sixth information superimposed display device of the present invention is the fifth information superimposed display device, wherein (importance of the object) = (weight of increase when the object is the most important object) * (object is the most important) (Increase in important object) + (weight of object score) * (score of object) + (weight of increase when object is pointed) * (increase when object is pointed) + (Weight of the number of frames that continuously point the screen) * (number of frames that continuously point the screen) + (weight of the number of objects included in the video) * (included in the video) The display information determining unit that calculates the importance of the object by a numerical value determined from the number of objects). That.
[0013]
The seventh information superimposed display device of the present invention is the second information superimposed display device, wherein (a) if the object is located at the position where the user is pointing, the object is the most important object. (B) If there is no object at the position where the user is pointing, a score is calculated for each object, the one with the highest score is made the most important object, and (c) the score for each object is Is determined in consideration of the size of the display area, the proximity to the center of the screen, and the ratio of continuous display, and (object importance) = (weight of the increase when the object is the most important object) ) * (Increase when the object is the most important object) + (weight of object score) * (object score + (Weight of increase when object is pointed) * (increase when object is pointed) + (weight of number of frames in which screen is continuously pointed) * (continuous screen) The display information determination unit is calculated by the following formula: (the number of frames pointing to) + (weight of the number of objects included in the video) * (numerical value determined from the number of objects included in the video) And
[0014]
An eighth information superimposed display device according to the present invention is the seventh information superimposed display device, wherein the size of the display area of the object, the proximity to the center of the screen, and the ratio of continuous display are weighted. And a display information determining unit that calculates the score of the object by adding them together.
[0015]
A ninth information superimposed display device according to the present invention is the eighth information superimposed display device, wherein (a) the importance is increased if the object is the most important object, and (b) the importance is increased according to the score of the object. (C) If the object is pointed, the importance is increased by a certain amount. (D) If somewhere on the screen is pointed continuously, the importance is increased. ) The importance is increased or decreased according to the number of objects included in the video. (F) Whether the object is the most important object, the score of the object, whether the object is pointed, and how continuously the screen is pointed , And the value to be increased or decreased from the number of objects included in the video, weighted and added, It characterized by having the display information determination unit which determines the.
[0016]
A tenth information superimposed display device according to the present invention is the ninth information superimposed display device, wherein (object importance) = (weight of increase when the object is the most important object) * (object is the most important) (Increase in important object) + (weight of object score) * (score of object) + (weight of increase when object is pointed) * (increase when object is pointed) + (Weight of the number of frames that continuously point the screen) * (number of frames that continuously point the screen) + (weight of the number of objects included in the video) * (included in the video) Numerical value determined from the number of objects) + (weight of increase when the user's face direction is continuously facing the screen direction) * ( Continued increase when the user's face orientation points to the screen direction) + (When the user's face orientation continues to face the screen direction but the screen is not pointed for a long time, The display information for calculating the importance of the object based on the weight of the decrease) * (the decrease when the screen is not pointed to for a long time even if the user's face is facing the screen continuously) It has a decision part.
[0017]
BEST MODE FOR CARRYING OUT THE INVENTION
A first embodiment of the present invention will be described in detail with reference to the drawings.
[0018]
FIG. 1 is a block diagram showing a configuration of the first exemplary embodiment of the present invention.
[0019]
Referring to FIG. 1, a first embodiment of the present invention includes a receiving unit 12 for receiving data broadcast program information, and a related information holding unit for holding related information with details of objects included in the data broadcast program information. 13, a pointing recognizing unit 14 for recognizing which object the user is pointing to, a display information determining unit 15 for determining the level of detail of related information to be displayed from the user's operation and its history, and a video of the data broadcast program. A display unit 16 for superimposing and displaying related information of objects included in the video.
[0020]
In addition, a program recording medium 11 that records a system program is provided. The program read from the program recording medium 11 controls the operation of the system.
[0021]
Next, a first embodiment of the present invention will be described with reference to the drawings.
[0022]
The receiving unit 12 receives the data broadcast program information, and passes the video information included in the data broadcast program information to the display information determination unit 15 (S205 in FIG. 1), and at the same time, the related information with the detail level of the object included in the data broadcast program information. Is passed to the related information holding unit 13 (S203 in FIG. 1).
[0023]
FIG. 2 is an explanatory diagram illustrating an example of the content of the related information with detail level.
[0024]
The related information with the degree of detail enables the user to know which object is displayed at which position and in what size in the video at a certain point in time, and to find the related information of those objects according to several levels of detail. It just needs to be.
[0025]
Referring to FIG. 2, the related information holding unit 13 holds the related information with detail level separately into object information and related information.
[0026]
The object information includes, for example, a frame number of video information, an ID of an object displayed in the frame, a position and a size of the object.
[0027]
For example, if two or more objects are displayed in a certain frame, data of the same frame number and different object IDs exist as shown in FIG.
[0028]
The position and size of the object may be expressed as a width and height, for example, with the shape of the object being a rectangle, the coordinates of the upper left vertex of the object as the position, or the upper left vertex and lower right of the object. The coordinates of the vertices may be used, or, for example, the shape of the object may be expressed as a polygon and the coordinates of vertices clockwise or counterclockwise from a certain vertex may be arranged.
[0029]
The related information only needs to be able to determine the related information to be displayed when the level of detail is designated for each object. For example, as shown in FIG. 2, each object is managed separately for each level of detail. Alternatively, it may be determined that when the level of detail is designated, all information below the level of detail is displayed.
[0030]
The display information determining unit 15 passes the video information passed from the receiving unit 12 to the display unit 16 (S506 in FIG. 1), and the display unit 16 displays the video information.
[0031]
The user looks at the video information displayed on the display unit 16 and specifies an object of interest included in the video using the pointing recognition unit 14. The pointing recognition unit 14 may be, for example, that the user specifies an object on the screen using a laser pointer and recognizes the position pointed by the laser pointer, or the user moves the mouse cursor over the object using a mouse. , And click the mouse to specify the position. Alternatively, the position where the user points at the screen may be recognized by image recognition using a camera.
[0032]
The position specified by the user on the screen through the pointing recognition unit 14 is passed to the display information determination unit 15 (S405 in FIG. 1).
[0033]
Information is passed from the pointing recognition unit 14 to the display information determination unit 15 at regular intervals, and the display information determination unit 15 accumulates the information passed from the pointing recognition unit 14. When no pointing is performed, information may not be sent, or information indicating that no pointing may be passed. Using the accumulated information, the display information determination unit 15 determines whether or not pointing is performed continuously.
[0034]
For example, when information is passed from the pointing recognizing unit 14 to the display information determining unit 15 at intervals of 1/30 second, which is the timing synchronized with the video frame, information that is pointed 30 times continuously is passed. For example, it is determined that pointing has been performed continuously for 30 frames (1 second). Also, for example, when information is passed at 1/10 second intervals, if information pointed continuously 10 times is passed, pointing is performed continuously for 30 frames (1 second). It is determined that there is.
[0035]
Using the current frame number of the video information as a key, the display information determination unit 15 receives the ID of the object displayed in the current frame, and the position and size of the object from the related information holding unit 13 (S305 in FIG. 1).
[0036]
The display information determination unit 15 also stores the user's designated position passed from the pointing recognition unit 14 and the ID of the object displayed in the current frame passed from the related information holding unit 13 and the history of the position and size. Hold.
[0037]
The display information determination unit 15 determines the relevant information to be displayed for each object from the position where the user is pointing the screen and the information on the position and size of the object, and transfers the relevant information to the display unit 16 together with the video information (S506 in FIG. 1). ), The display section 16 displays the related information of the object superimposed on the video information.
[0038]
The display information determining unit 15 determines the related information to be displayed for each object by first determining the most important object from the objects included in the current video information, and then determining the object other than the most important object. Determine importance. The detail of the related information of the object is determined according to the importance of the object.
[0039]
The most important object is determined in the following manner.
[0040]
(1) If there is an object at the position where the user points, the object is set as the most important object.
[0041]
(2) If there is no object at the position where the user is pointing, a score is calculated for each object, and the object with the highest score is determined as the most important object.
[0042]
The score for each object is calculated in consideration of the size of the display area of the object, the proximity to the center of the screen, and the ratio of continuous display. For example, the following is performed.
[0043]
(1) Find the center of gravity and area of the object from the position and size of the object. For simplicity, for example, a rectangle circumscribing the object may be determined and substituted by the center of gravity and area of the rectangle.
[0044]
(2) The index of the proximity of the object to the center of the screen is, for example, the reciprocal of the distance from the center of gravity of the object to the center of the screen.
[0045]
(3) The rate of continuous display is, for example, the rate of how many frames have been displayed continuously in the video of the last minute.
[0046]
(4) The weight of the display area of the object, the proximity to the center of the screen, and the ratio of continuously displayed objects are weighted and added to obtain an object score.
[0047]
The weight may be set to 0 when calculating the score of the object.
[0048]
In other words, the score may be determined based on only one of the size of the display area of the object, the proximity to the center of the screen, and the ratio of continuous display, or the score may be determined based on only some combinations but not all. May be determined.
[0049]
In addition, the importance of each object takes into account whether the object is the most important object, the score of the object, whether the object is pointed, how long the screen is pointed continuously, and the number of objects included in the video. To decide. For example, it is determined as follows.
[0050]
(1) If the object is the most important object, increase the importance.
[0051]
(2) The importance is increased according to the score of the object.
[0052]
(3) If the object is pointed, the importance is increased by a certain amount.
[0053]
(4) If any part of the screen is continuously pointed, the importance is increased. For example, a threshold and a change rate are defined, and if the number of seconds or frames that are continuously pointed does not exceed the threshold, the importance is gradually increased according to the change rate. Increase importance by a certain amount.
[0054]
(5) The importance is increased or decreased according to the number of objects included in the video. For example, if the number of objects is small, the importance of each object is relatively increased, and if the number of objects is large, the importance of each object is decreased.
[0055]
(6) Determine a value to be increased or decreased based on whether the object is the most important object, the score of the object, whether the object is pointed, how much the screen is continuously pointed, and the number of objects included in the video. And add them together to determine the importance of each object.
[0056]
The weight may be set to 0 when calculating the importance of the object.
[0057]
That is, the score is determined from only the most important object, the score of the object, whether the object is pointed, how much the screen is pointed continuously, or the number of objects included in the video. Alternatively, the score may be determined from only some combinations but not all.
[0058]
The calculation of the importance of the object is, for example, as follows.
[0059]
(Object importance) = (weight of increase when object is the most important object) * (increase when object is the most important object) + (weight of object score) * (object score) + ( Weight of the increment when the object is pointed) * (increase when the object is pointed) + (weight of the number of frames in which the screen is continuously pointed) * (pointing of the screen continuously) Number of frames in the video) + (weight of the number of objects included in the video) * (numerical value determined from the number of objects included in the video)
For example, the display information determination unit 15 determines the detail level of each object as follows.
[0060]
It is assumed that the user points at a certain point on the screen.
[0061]
The position pointed by the user is passed to the display information determination unit 15 through the pointing recognition unit 14 (S405 in FIG. 1). For example, it is assumed that the pointing position is (600, 600).
[0062]
Using the current frame number of the video information as a key, the display information determination unit 15 receives the ID of the object displayed in the current frame, and the position and size of the object from the related information holding unit 13 (S305 in FIG. 1). It is assumed that the related information holding unit 13 holds data as shown in FIG. 2 and that the current frame number of the video information is 1000. The related information holding unit 13 searches for data whose frame number is 1000.
[0063]
From the related information holding unit 13 to the display information determining unit 15, the position of the object 1 is (80, 80) and the size is (100, 200), the position of the object 2 is (300, 400), and the size is (200). , 100) is passed (S305 in FIG. 1).
[0064]
The display information determination unit 15 first determines the most important object.
[0065]
Since the position of the object 1 is (80, 80) and the size is (100, 200), and the position of the object 2 is (300, 400) and the size is (200, 100), the position (600) where the user is pointing , 600) do not point to any objects and the most important object has not yet been determined.
[0066]
Since the most important object has not been determined, the score for each object is calculated next.
[0067]
For example, assume that the size of the screen is (1000, 800).
[0068]
Regarding the object 1, the center of gravity of the object 1 is (130, 180) and the area is 20,000.
[0069]
Since the center of the screen is (500, 400), the distance from the center of gravity to the center of the screen is √ ((500−130) ＾ 2 + (400−180) ＾ 2) = 430.5, and the proximity to the center of the screen is It is the reciprocal.
[0070]
Also, assuming that the video is displayed at 30 frames per second, 1800 frames will be displayed in one minute. Assuming that the object 1 has been continuously displayed for 360 frames in the video of the last minute, the ratio is 0.2.
[0071]
Regarding the object 2, the center of gravity of the object 2 is (400,450) and the area is 20,000.
[0072]
Since the center of the screen is (500, 400), the distance from the center of gravity to the center of the screen is √ ((500−400) ＾ 2 + (400−450) ＾ 2) = 111.8, and the proximity to the center of the screen is It is the reciprocal.
[0073]
If the object 2 has been displayed for 360 frames consecutively to the video of the last minute, the ratio is 0.2.
[0074]
Assuming that the weight of the display area of the object, the proximity to the center of the screen, and the ratio of continuous display are 0.0001, 500, and 1,
The calculation of the score is (score) = 0.0001 * (the size of the display area of the object) + 500 / (closeness to the center of the screen) + 1 * (the ratio of continuously displayed images).
[0075]
The score of object 1 is 0.0001 * 20000 + 500 / 430.5 + 1 * 0.2 = 3.36, and the score of object 2 is 0.0001 * 20000 + 500 / 111.8 + 1 * 0.2 = 6.67. Therefore, the object 2 is determined as the most important object.
[0076]
Next, the importance of each object is determined.
[0077]
The increment of the importance when the object is pointed is set to 1, and the increment is 0 when the object is not pointed.
[0078]
The threshold value of the number of frames when any part of the screen is continuously pointed is 900, and the change rate is 0.002. At this point, it is assumed that the user has pointed somewhere on the screen continuously for 600 frames. The amount of increase or decrease in importance according to the number of objects included in the video is, for example, -1 / (11-N), where N is the number of objects included in the video. However, when N is 10 or more, it is assumed that N is set to 10. Also, the weights for the addition are 0.2 for the object score and 1 for all others.
[0079]
(Object importance) = (weight of increase when object is the most important object) * (increase when object is the most important object) + (weight of object score) * (object score) + ( Weight of the increment when the object is pointed) * (increase when the object is pointed) + (weight of the number of frames in which the screen is continuously pointed) * (pointing of the screen continuously) The number of frames in the video) + (weight of the number of objects included in the video) * (numerical value determined from the number of objects included in the video) ) = 1 * (increase when the object is the most important object) + 0.2 * (object A) + 1 * to become (the number of increments) + 0.002 * (objects contained in successive number of frames that are pointing the screen) -1 / (11-image when it is pointing).
[0080]
At this time, the importance of the object 1 is that the object 1 is not the most important object, the score of the object is 3.36, the object is not pointed, and the user points at some point on the screen for 600 consecutive frames. Since the number of objects included in the video is two, 0 + 0.2 * 3.36 + 0 + 600 * 0.002-1 / (11-2) = 1.76.
[0081]
The importance of the object 2 is that the object 2 is the most important object, the score of the object is 6.67, the object is not pointed, and the user points at some point on the screen for 600 consecutive frames. Since the number of objects included in the video is two, 1 + 0.2 * 6.67 + 0 + 600 * 0.002-1 / (11-2) = 3.42.
[0082]
Assuming that the degree of detail of the related information of the object is obtained by cutting off the decimal part of the importance of the object, the degree of detail of the related information of the object 1 is 1 and the degree of detail of the related information of the object 2 is 3.
[0083]
The display information determination unit 15 receives the related information from the related information holding unit 13 according to the level of detail for each object (S305 in FIG. 1).
[0084]
Since the detail of the related information of the object 1 is 1, the related information is “mobile phone”, and the detail of the related information of the object 2 is 3, so the related information is “PDA, pocket gear, 30,000 yen”.
[0085]
The display information determination unit 15 passes the related information determined for each object together with the video information to the display unit 16 (S506 in FIG. 1), and the display unit 16 superimposes and displays the relevant information of the object on the video information.
[0086]
FIG. 3 is an explanatory diagram illustrating an example of related information superimposed on video information.
[0087]
Referring to FIG. 3, related information is displayed below the object.
[0088]
Next, a second embodiment of the present invention will be described in detail with reference to the drawings.
[0089]
FIG. 4 is an explanatory diagram showing the configuration of the second exemplary embodiment of the present invention.
[0090]
Referring to FIG. 4, a second embodiment of the present invention, in addition to the components of the first embodiment, includes a face image recognition unit that recognizes whether or not the user faces the display unit 16. 17 is provided. When the recognition result from the pointing recognition unit 14 is passed to the display information determination unit 15, the recognition result as to whether the user faces the display unit 16 is also passed from the face image recognition unit 17 to the display information determination unit 15. It is used to determine how detailed related information is displayed for each object.
[0091]
Information is passed from the face image recognition unit 17 to the display information determination unit 15 at regular intervals, and the display information determination unit 15 stores the information passed from the face image recognition unit 17. When the user does not face the display unit 16, the information may not be sent, or information indicating that the user does not face the display unit 16 may be passed. Using the stored information, the display information determination unit 15 determines whether the user continuously faces his / her face to the display unit 16.
[0092]
For example, when information is passed from the face image recognition unit 17 to the display information determination unit 15 at intervals of 1/30 seconds, which is the timing synchronized with the video frame, the user displays the face on the display unit 16 continuously 30 times. If the information indicating that the user is pointing is passed, it is determined that the user continuously faces the display unit 16 for 30 frames (1 second). Further, for example, when information is passed at 1/10 second intervals, if information indicating that the user turns his / her face to the display unit 16 is passed ten times in succession, the number of frames is 30 frames (1 second). It is determined that the user continuously faces his / her face to the display unit 16.
[0093]
In the second embodiment of the present invention, processing is performed in the same manner as in the first embodiment of the present invention. In the second and second embodiments of the present invention, the method of determining the importance for each object is different from that of the second and first embodiments of the present invention.
[0094]
In the second embodiment of the present invention, the degree of importance of each object is determined based on whether the object is the most important object, the score of the object, whether the object is pointed, how much the screen is continuously pointed, or Is determined in consideration of the number of objects included in the image and the recognition result of the face image. For example, it is determined as follows.
[0095]
(1) If the direction of the user's face is continuously facing the screen direction, the importance of the object is increased.
[0096]
(2) Even if the user's face is facing the screen direction continuously, if the time during which the screen is not pointed is long, it is considered that the user concentrates on the contents of the program rather than the related information, and the importance of the object Decrease.
[0097]
(3) As in the first embodiment, whether the object is the most important object, the score of the object, whether the object is pointed, how long the screen is continuously pointed, the number of objects included in the video, In addition to the above, the face is facing the screen direction, and a value to be increased or decreased from the time when the screen is not pointed is determined, and the weight is added to determine the importance of each object.
[0098]
The weight may be set to 0 when calculating the importance of the object. In other words, whether the object is the most important object, the score of the object, whether the object is pointed, how much continuous screen pointing, the number of objects included in the video, whether the face is facing the screen, The score may be determined from only one of the times when the screen is not pointed, or the score may be determined from only some combinations but not all.
[0099]
The importance of an object is calculated, for example, as follows.
[0100]
(Object importance) = (weight of increase when object is the most important object) * (increase when object is the most important object) + (weight of object score) * (object score) + ( Weight of the increment when the object is pointed) * (increase when the object is pointed) + (weight of the number of frames in which the screen is continuously pointed) * (pointing of the screen continuously) Number of frames being performed) + (weight of the number of objects included in the video) * (numerical value determined from the number of objects included in the video) + (the direction of the user's face continuously faces the screen direction) Weight when the user is facing) * (increase when the user's face is facing the screen direction) + (continuously) Even if the user's face is facing the screen direction, the weight of the decrease when the screen is not pointed for a long time) * (even if the user's face is facing the screen continuously, Decrease when the screen is not pointed at for a long time).
[0101]
For example, the receiving unit 12 receives the data broadcast program information, passes the video information included in the data broadcast program information to the display information determination unit 15 (S205 in FIG. 4), and adds the level of detail of the object included in the data broadcast program information. The related information is passed to the related information holding unit 13 (S203 in FIG. 4). The display information determining unit 15 passes the video information passed from the receiving unit 12 to the display unit 16 (S506 in FIG. 4), and the display unit 16 displays the video information. The recognition result of the face direction of the user is transmitted from the face image recognition unit 17 to the display information determination unit 15 (S705 in FIG. 4).
[0102]
In addition, the user looks at the video information displayed on the display unit 16, specifies the object of interest included in the video using the pointing recognition unit 14, and specifies the object on the screen through the pointing recognition unit 14. The position is passed to the display information determination unit 15 (S405 in FIG. 4).
[0103]
Further, for example, the determination of the detail level of each object in the display information determination unit 15 is performed as follows.
[0104]
It is assumed that the user points at a certain point on the screen. The position pointed by the user is passed to the display information determination unit 15 through the pointing recognition unit 14 (S405 in FIG. 4).
[0105]
For example, it is assumed that the pointing position is (600, 600).
[0106]
Using the current frame number of the video information as a key, the display information determination unit 15 receives the ID of the object displayed in the current frame and the position and size of the object from the related information holding unit 13 (S305 in FIG. 4). It is assumed that the related information holding unit 13 holds data as shown in FIG. 2 and that the current frame number of the video information is 1000. The related information holding unit 13 searches for data whose frame number is 1000. The position of the object 1 is (80, 80) and the size is (100, 200), the position of the object 2 is (300, 400) and the size is (200) from the related information holding unit 13 to the display information determination unit 15. , 100) (S305 in FIG. 4).
[0107]
As in the first embodiment of the present invention, the object 2 is determined as the most important object.
[0108]
Next, the importance of each object is determined.
[0109]
Assume that the increment of the importance when the object is pointed is 1. The threshold value of the number of frames when any part of the screen is continuously pointed is 900, and the change rate is 0.002. At this point, it is assumed that the user has pointed somewhere on the screen continuously for 600 frames. The amount of increase or decrease in importance according to the number of objects included in the video is, for example, -1 / (11-N), where N is the number of objects included in the video. However, when N is 10 or more, it is assumed that N is set to 10.
[0110]
The threshold value of the number of frames when the direction of the user's face is continuously pointing to the screen direction is 900, and the change rate is 0.001. When the face direction of the user is continuously facing the screen direction and the screen is not pointed, the threshold value of the number of frames is 900 and the change rate is 0.001. At this point, it is assumed that the user has turned the face toward the screen continuously for 900 frames or more.
[0111]
Also, the weights for the addition are 0.2 for the object score and 1 for all others.
[0112]
(Object importance) = (weight of increase when object is the most important object) * (increase when object is the most important object) + (weight of object score) * (object score) + ( Weight of the increment when the object is pointed) * (increase when the object is pointed) + (weight of the number of frames in which the screen is continuously pointed) * (pointing of the screen continuously) Number of frames being performed) + (weight of the number of objects included in the video) * (numerical value determined from the number of objects included in the video) + (the direction of the user's face continuously faces the screen direction) Weight when the user is facing) * (increase when the user's face is facing the screen direction) + (continuously) Even if the user's face is facing the screen direction, the weight of the decrease when the user is not pointing at the screen for a long time) * (even if the user's face is facing the screen direction continuously, Since the amount of decrease when the screen is not pointed at for a long time), the importance of the object is calculated as (the importance of the object) = 1 * (the increase when the object is the most important object) + 0.2 * ( (Score of object) + 1 * (increase when pointing) + 0.002 * (number of frames in which screen is continuously pointed) -1 / (11-number of objects included in video) + 1 * (Number of frames when the face direction of the user is continuously facing the screen direction * change rate) -1 * (points in the screen when the face direction of the user is continuously facing the screen direction) The number of frames * rate of change) and made of when not in Ingu.
[0113]
At this time, the importance of the object 1 is that the object 1 is not the most important object, the score of the object is 3.36, the object is not pointed, and the user points at some point on the screen for 600 consecutive frames. Since the number of objects included in the video is two and the face is facing the screen direction, 0 + 0.2 * 3.36 + 0 + 600 * 0.002-1 / (11-2) + 900 * 0.001 −0 = 2.66.
[0114]
The importance of the object 2 is that the object 2 is the most important object, the score of the object is 6.67, the object is not pointed, and the user points at some point on the screen for 600 consecutive frames. Since the number of objects included in the video is two and the face is facing the screen direction, 1 + 0.2 * 6.67 + 0 + 600 * 0.002-1 / (11-2) + 900 * 0.001- 0 = 4.32.
[0115]
Assuming that the degree of detail of the related information of the object is obtained by cutting off the decimal part of the importance of the object, the degree of detail of the related information of the object 1 is 2 and the degree of detail of the related information of the object 2 is 4.
Here, it is assumed that the pointing of the screen is stopped with the face facing the screen and 900 frames are passed. The number of objects in the video remains two.
[0116]
At this time, the importance of the object 1 is such that the object 1 is not the most important object, the score of the object is 3.36, the object is not pointed, and the user does not point anywhere on the screen. Since the number of objects included in the video image is two and the face faces the screen direction, 0 + 0.2 * 3.36 + 0 + 0-1 / (11-2) + 900 * 0.001-900 * 0.001 = 0.56.
[0117]
The importance of the object 2 is as follows. The object 2 is the most important object, the score of the object is 6.67, the object is not pointed, and the user is not pointing anywhere on the screen. Since the number of objects included in the video is two and the face is facing the screen direction, 1 + 0.2 * 6.67 + 0 + 0-1 / (11-2) + 900 * 0.001-900 * 0.001 = 2 .22.
[0118]
Assuming that the degree of detail of the related information of the object is obtained by truncating the decimal part of the importance of the object, the degree of detail of the related information of the object 1 is 0, and the degree of detail of the related information of the object 2 is 2.
[0119]
The display information determination unit 15 receives the related information from the related information holding unit 13 according to the level of detail for each object (S305 in FIG. 4). The display information determination unit 15 passes the relevant information determined for each object together with the video information to the display unit 16 (S506 in FIG. 4), and the display unit 16 superimposes and displays the relevant information of the object on the video information.
[0120]
In this way, it becomes possible to automatically adjust the related information of the object presented to the viewer.
[0121]
【The invention's effect】
An advantage of the present invention is that the number of steps for optimizing information displayed by a program provider can be reduced, and the amount of information displayed to a viewer can be reduced.
[0122]
The reason is that a plurality of pieces of information related to the objects in the video are prepared for each level of detail, and the level of detail of the information displayed by the system is determined from the state of the viewer and the operation history.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of a first exemplary embodiment of the present invention.
FIG. 2 is an explanatory diagram showing an example of object information and related information.
FIG. 3 is an explanatory diagram showing a display state in which related information is superimposed on an object.
FIG. 4 is a block diagram showing a configuration of a second exemplary embodiment of the present invention.
[Explanation of symbols]
11 Program recording medium
12 Receiver
13 Related information storage
14. Pointing Recognition Unit
15 Display information judgment unit
16 Display
17 Face image recognition unit

Claims (10)

A receiving unit that receives video information, data broadcast program information including related information with an object detail level, and outputs video information and related information with an object detail level;
A related information holding unit that holds related information with a degree of detail from the receiving unit,
A pointing recognition unit that recognizes a pointing position on the screen specified by the user and outputs pointing position information;
Pointing position information from the pointing recognition unit, video information from the receiving unit, based on the related information with the degree of detail of the object from the related information holding unit, to calculate the importance of each object included in the video information, The degree of detail for each object is calculated based on the calculated importance, the related information to be displayed corresponding to the object is determined from the related information with the degree of detail of the object based on the calculated degree of detail, and the video information and the corresponding object are displayed. A display information determining unit that outputs related information;
A display unit that superimposes and displays relevant information corresponding to the object corresponding to the object from the display information determination unit on the video information from the display information determination unit;
An information superimposing display device comprising: A receiving unit that receives video information, data broadcast program information including related information with an object detail level, and outputs video information and related information with an object detail level;
A related information holding unit that holds related information with a degree of detail from the receiving unit,
A pointing recognition unit that recognizes a pointing position on the screen specified by the user and outputs pointing position information;
Importance of each object included in video information based on face direction information, pointing position information from the pointing recognition unit, video information from the reception unit, and related information with detail of the object from the relevant information holding unit Is calculated, the detail level of each object is calculated based on the calculated importance level, the related information to be displayed corresponding to the object is determined from the related information with the detail level of the object based on the calculated detail level, and the video information, the object A display information determining unit that outputs relevant information to be displayed,
A display unit that superimposes and displays relevant information corresponding to the object corresponding to the object from the display information determination unit on the video information from the display information determination unit;
A face image recognition unit that creates face direction information indicating whether the user's face is facing the display unit, and outputs the information to the display information determination unit;
An information superimposing display device comprising: The most important objects,
(A) If there is an object at the position where the user points, the object is regarded as the most important object,
(B) If there is no object at the position where the user is pointing, a score is calculated for each object, and the object with the highest score is determined as the most important object,
(C) The score for each object is calculated in consideration of the size of the display area of the object, the proximity to the center of the screen, and the ratio of continuous display.
Decide,
(Object importance) = (weight of increase when object is the most important object) * (increase when object is the most important object) + (weight of object score) * (object score) + ( Weight of the increment when the object is pointed) * (increase when the object is pointed) + (weight of the number of frames in which the screen is continuously pointed) * (pointing of the screen continuously) The display information determination unit is calculated by: (the number of frames being performed) + (weight of the number of objects included in the video) * (numerical value determined from the number of objects included in the video) The information superimposed display device according to claim 1. The display information determining unit is configured to calculate the score of the object by adding a weight to the size of the display area of the object, the proximity to the center of the screen, and the ratio of continuous display. The information superimposed display device according to claim 3. (A) If the object is the most important object, increase the importance,
(B) increasing the importance according to the score of the object,
(C) If the object is pointed, the importance is increased by a certain amount,
(D) If somewhere on the screen is continuously pointed, the importance is increased, (e) the importance is increased or decreased according to the number of objects included in the video,
(F) Determine whether the object is the most important object, the score of the object, whether the object is pointed, how much the screen is continuously pointed, and a value to be increased or decreased from the number of objects included in the video, 5. The information superimposed display device according to claim 4, further comprising the display information determining unit that determines the importance of each object by adding weights. (Object importance) = (weight of increase when object is the most important object) * (increase when object is the most important object) + (weight of object score) * (object score) + ( Weight of the increment when the object is pointed) * (increase when the object is pointed) + (weight of the number of frames in which the screen is continuously pointed) * (pointing of the screen continuously) The display information determination unit that calculates the importance of an object by (the number of frames being performed) + (weight of the number of objects included in the video) * (numerical value determined from the number of objects included in the video) 6. The information superimposed display device according to claim 5, comprising: The most important objects,
(A) If there is an object at the position where the user points, the object is regarded as the most important object,
(B) If there is no object at the position where the user is pointing, a score is calculated for each object, and the object with the highest score is determined as the most important object,
(C) The score for each object is calculated in consideration of the size of the display area of the object, the proximity to the center of the screen, and the ratio of continuous display.
Decide,
(Object importance) = (weight of increase when object is the most important object) * (increase when object is the most important object) + (weight of object score) * (object score) + ( Weight of the increment when the object is pointed) * (increase when the object is pointed) + (weight of the number of frames in which the screen is continuously pointed) * (pointing of the screen continuously) The display information determination unit is calculated by: (the number of frames being performed) + (weight of the number of objects included in the video) * (numerical value determined from the number of objects included in the video) The information superimposed display device according to claim 2. The display information determining unit is configured to calculate the score of the object by adding a weight to the size of the display area of the object, the proximity to the center of the screen, and the ratio of continuous display. The information superimposed display device according to claim 7. (A) If the object is the most important object, increase the importance,
(B) increasing the importance according to the score of the object,
(C) If the object is pointed, the importance is increased by a certain amount,
(D) If somewhere on the screen is continuously pointed, the importance is increased, (e) the importance is increased or decreased according to the number of objects included in the video,
(F) Determine whether the object is the most important object, the score of the object, whether the object is pointed, how much the screen is continuously pointed, and a value to be increased or decreased from the number of objects included in the video, 9. The information superimposed display device according to claim 8, further comprising the display information determination unit that determines the importance of each object by adding weights. (Object importance) = (weight of increase when object is the most important object) * (increase when object is the most important object) + (weight of object score) * (object score) + ( Weight of the increment when the object is pointed) * (increase when the object is pointed) + (weight of the number of frames in which the screen is continuously pointed) * (pointing of the screen continuously) Number of frames being performed) + (weight of the number of objects included in the video) * (numerical value determined from the number of objects included in the video) + (the direction of the user's face continuously faces the screen direction) Weight when the face is facing the screen direction) + (continuously when the user's face is facing the screen direction) + (continuously) Even if the user's face is facing the screen, the weight of the decrease when the screen is not pointed at for a long time) * (even if the user's face is continuously facing the screen, 10. The information superimposed display device according to claim 9, further comprising the display information determination unit that calculates the importance of the object based on a decrease when the screen is not pointed at for a long time.

Images