JP4103652B2 - Information superimposed display device, information superimposed display method, and information superimposed display program - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an information superimposed display device, and more particularly to a superimposed display device for information related to an object included in a video of a data broadcast program.
[0002]
[Prior art]
In the data broadcasting program, information related to the object in the video is transmitted simultaneously with the video information, and the related information is displayed on the receiver side as the video progresses.
[0003]
For example, a program receiving unit that receives information including a program and auxiliary information that supplements the program, a display flag management unit that manages a display flag that is a flag indicating whether or not auxiliary information is displayed, and the program A separation unit that separates the program and auxiliary information received by the reception unit, a program display unit that displays the program separated by the separation unit, an auxiliary information display unit that displays the auxiliary information separated by the separation unit, There is a technique including an auxiliary information display control unit that reads a display flag of the display flag management unit and instructs the auxiliary information display unit to display auxiliary information when the display flag is on. (Patent Document 1)
[Patent Document 1]
JP 2001-245234 A
[0004]
[Problems to be solved by the invention]
The problem with the above-described technique is that only a small amount of related information that the viewer wants to know in detail is displayed, or conversely, a lot of extra related information is displayed.
[0005]
In addition, in order to provide as much information as possible as desired by the viewer, the program provider needs to optimize the amount and time of information to be displayed, the priority with other information in advance, and so on. There is an accompanying problem that man-hours are enormous.
[0006]
It is an object of the present invention to prepare a plurality of pieces of information related to objects in a video for each degree of detail, and to determine the degree of detail of information displayed by the system from the viewer's state and operation history, so that the program provider can This is to reduce the man-hours for optimizing the information to be displayed and to reduce the excess or deficiency of the information displayed to the viewer.
[0007]
[Means for Solving the Problems]
The first information superimposing display device of the present invention receives video broadcast program information including video information and related information with details of objects, and outputs video information and related information with details of objects, A related information holding unit that holds related information with a detail level from the receiving unit, a pointing recognition unit that recognizes a pointing position on a screen specified by a user, and outputs pointing position information, and a pointing from the pointing recognition unit Based on the positional information, the video information from the receiving unit, and the related information with the detail level of the object from the related information holding unit, the importance for each object included in the video information is calculated, and based on the calculated importance The level of detail for each object is calculated, and based on the calculated level of detail, A display information determination unit that determines related information to be displayed corresponding to the object, and outputs video information and related information to be displayed corresponding to the object, and the video information from the display information determination unit corresponds to the object corresponding from the display information determination unit. And a display unit that superimposes and displays related information to be displayed.
[0008]
The second information superimposing display device of the present invention receives the data broadcast program information including the video information and the related information with the detail level of the object, and outputs the video information and the related information with the detail level of the object, A related information holding unit for holding related information with a degree of detail from the receiving unit, a pointing recognition unit for recognizing a pointing position on a screen designated by the user, and outputting pointing position information, face direction information, and the pointing recognition The importance calculated for each object included in the video information is calculated based on the pointing position information from the unit, the video information from the receiving unit, and the related information with the detail level of the object from the related information holding unit. The level of detail for each object is calculated based on the degree of detail, and the relationship with the level of detail of the object is calculated based on the calculated level of detail. The display information determination unit for determining the related information to be displayed corresponding to the object from the report, and outputting the video information and the related information to be displayed for the object, and the video information from the display information determination unit. Face image recognition that creates a display unit that superimposes related information to be displayed corresponding to an object and displays face direction information that indicates whether a user's face is facing the display unit, and outputs the information to the display information determination unit And a portion.
[0009]
A third information superimposing display device of the present invention is the first information superimposing display device, wherein the most important object is (a) if there is an object at the position where the user is pointing, the object is the highest. (B) If there is no object at the position where the user is pointing, the score is calculated for each object, the object with the highest score is the most important object, and (c) the score for each object is the object Determined by taking into account the display area size, proximity to the center of the screen, and the rate of continuous display, (object importance) = (weight of increase when the object is the most important object) ) * (Increment when the object is the most important object) + (Object score weight) * (Object score) + (Weight increase when the object is pointing) * (weight increase when the object is pointed) + (weight of the number of frames pointing the screen continuously) * (screen continuously The display information determination unit is calculated by: (the number of frames pointing to) + (weight of the number of objects included in the video) * (numerical value determined from the number of objects included in the video) And
[0010]
A fourth information superimposing display device of the present invention is the third information superimposing display device, wherein a weight is given to the size of the display area of the object, the proximity to the center of the screen, and the ratio displayed continuously. The display information determination unit calculates the score of the object by adding and adding.
[0011]
A fifth information superimposed display device of the present invention is the fourth information superimposed display device,
(A) Increase the importance if the object is the most important object, (b) Increase the importance according to the score of the object, (c) If the object is pointed, increase the importance by a certain amount (D) If any part of the screen is continuously pointed, increase the importance, (e) increase or decrease the importance according to the number of objects included in the video, and (f) Determine the most important object, the object score, whether the object is pointed, how long the screen is pointing, and the value to be increased or decreased from the number of objects included in the video, and add the weight. In addition, the display information determining unit that determines the importance of each object is provided.
[0012]
The sixth information superimposing display device of the present invention is the fifth information superimposing display device, wherein (the importance of the object) = (the weight of the increment when the object is the most important object) * (the object is the most important). Increment when important object) + (Object score weight) * (Object score) + (Increment weight when object is pointing) * (Increment when object is pointing) + (Weight of the number of frames that point to the screen continuously) * (number of frames that point to the screen continuously) + (weight of the number of objects included in the video) * (included in the video) The display information determination unit for calculating the importance of the object by a numerical value determined from the number of objects), That.
[0013]
The seventh information superimposed display device of the present invention is the second information superimposed display device, wherein the most important object is (a) if there is an object at the position where the user is pointing, that object is (B) If there is no object at the position where the user is pointing, the score is calculated for each object, the object with the highest score is the most important object, and (c) the score for each object is the object Determined by taking into account the display area size, proximity to the center of the screen, and the rate of continuous display, (object importance) = (weight of increase when the object is the most important object) ) * (Increment when the object is the most important object) + (Object score weight) * (Object score) + (Weight increase when the object is pointing) * (weight increase when the object is pointed) + (weight of the number of frames pointing the screen continuously) * (screen continuously The display information determination unit is calculated by: (the number of frames pointing to) + (weight of the number of objects included in the video) * (numerical value determined from the number of objects included in the video) And
[0014]
The eighth information superimposing display device of the present invention is the seventh information superimposing display device, wherein the weight of the display area size of the object, the proximity to the center of the screen, and the ratio displayed continuously are weighted. The display information determination unit calculates the score of the object by adding and adding.
[0015]
The ninth information superimposed display device of the present invention is the eighth information superimposed display device, wherein (a) the importance is increased if the object is the most important object, and (b) the importance is increased according to the score of the object. (C) If the object is pointed, the importance is increased by a certain amount. (D) If any part of the screen is continuously pointed, the importance is increased. ) Increase or decrease the importance according to the number of objects included in the video, and (f) How the object is the most important object, the object score, whether the object is pointed, and how long it is pointing to the screen Determine the value to be increased / decreased from the number of objects included in the video, and add the weights to add importance It characterized by having the display information determination unit which determines the.
[0016]
The tenth information superimposing display device of the present invention is the ninth information superimposing display device, wherein (the importance of the object) = (the weight of the increment when the object is the most important object) * (the object is the most important) Increment when important object) + (Object score weight) * (Object score) + (Increment weight when object is pointing) * (Increment when object is pointing) + (Weight of the number of frames that point to the screen continuously) * (number of frames that point to the screen continuously) + (weight of the number of objects included in the video) * (included in the video) (Numerical value determined from the number of objects) + (weight of the increment when the user's face is continuously facing the screen) * ( (Increment when the user's face is facing the screen direction) + (When the time when the screen is not pointing is long even if the user's face is facing the screen direction continuously) The display information for calculating the importance of the object based on the weight of the decrease) * (the decrease when the time during which the user's face is continuously pointing to the screen is not pointing on the screen) It has a determination part.
[0017]
DETAILED DESCRIPTION OF THE INVENTION
A first embodiment of the present invention will be described in detail with reference to the drawings.
[0018]
FIG. 1 is a block diagram showing the configuration of the first exemplary embodiment of the present invention.
[0019]
Referring to FIG. 1, in the first embodiment of the present invention, a receiving unit 12 that receives data broadcast program information, and a related information holding unit that holds related information with details of objects included in the data broadcast program information. 13, a pointing recognition unit 14 for recognizing which object the user is pointing to, a display information determination unit 15 for determining the level of detail of related information to be displayed based on the user's operation and its history, and a video of a data broadcast program And a display unit 16 that superimposes and displays related information of objects included in the video.
[0020]
Also provided is a program recording medium 11 on which a system program is recorded. A program read from the program recording medium 11 controls the operation of the system.
[0021]
Next, a first embodiment of the present invention will be described with reference to the drawings.
[0022]
The receiving unit 12 receives the data broadcast program information and passes the video information included in the data broadcast program information to the display information determination unit 15 (S205 in FIG. 1), and at the same time, the related information with the detail level of the object included in the data broadcast program information Is transferred to the related information holding unit 13 (S203 in FIG. 1).
[0023]
FIG. 2 is an explanatory diagram illustrating an example of the content of related information with a degree of detail.
[0024]
As for the related information with detail level, it is possible to know which object is displayed at what position and in what position in the video at a certain point of time, and related information of those objects can be understood according to the level of detail. It only has to be.
[0025]
Referring to FIG. 2, the related information holding unit 13 holds related information with a detail level separately for object information and related information.
[0026]
The object information includes, for example, the frame number of the video information, the ID of the object displayed in the frame, and the position and size of the object.
[0027]
For example, if two or more objects are displayed in a certain frame, data of different object IDs exist with the same frame number as shown in FIG.
[0028]
The position and size of the object may be expressed by, for example, a rectangle with the shape of the object, the coordinates of the top left vertex of the object as a position, the width and height, or the top left vertex and the bottom right corner of the object. For example, the coordinates of the vertices may be expressed by arranging the coordinates of the vertices clockwise or counterclockwise from a certain vertex as a polygon.
[0029]
The related information only needs to be able to determine the related information to be displayed when the detail level is designated for each object. For example, as shown in FIG. 2, each object is managed separately for each degree of detail. Further, when the level of detail is specified, it may be determined that all information below the level of detail is displayed.
[0030]
The display information determination unit 15 passes the video information passed from the reception unit 12 to the display unit 16 (S506 in FIG. 1), and the display unit 16 displays the video information.
[0031]
The user views the video information displayed on the display unit 16 and designates an object of interest contained in the video using the pointing recognition unit 14. For example, the pointing recognition unit 14 may be such that the user designates an object on the screen using a laser pointer and recognizes the position pointed to by the laser pointer, or the user moves the mouse cursor over the object using a mouse. You can also move to and click the mouse to specify the position. Alternatively, the position where the user is pointing at the screen may be recognized by image recognition using a camera.
[0032]
The position designated on the screen by the user through the pointing recognition unit 14 is passed to the display information determination unit 15 (S405 in FIG. 1).
[0033]
Information is passed from the pointing recognition unit 14 to the display information determination unit 15 at regular intervals, and the display information determination unit 15 accumulates information passed from the pointing recognition unit 14. Further, when not pointing, information may not be sent, or information indicating that pointing is not performed may be passed. Using this accumulated information, the display information determination unit 15 determines whether or not the pointing is continuously performed.
[0034]
For example, when information is passed from the pointing recognition unit 14 to the display information determination unit 15 at 1/30 second intervals that are synchronized with the video frame, the information pointed 30 times continuously is passed. For example, it is determined that the pointing is continued for 30 frames (1 second). Also, for example, when information is passed at intervals of 1/10 second, if information pointed 10 times continuously is passed, it is continuously pointed for 30 frames (1 second). It is determined that
[0035]
The display information determination unit 15 receives the ID, the position and the size of the object displayed in the current frame from the related information holding unit 13 using the current frame number of the video information as a key (S305 in FIG. 1).
[0036]
In addition, the display information determination unit 15 stores the user's designated position passed from the pointing recognition unit 14, the ID of the object displayed in the current frame passed from the related information holding unit 13, and the history of the position and size. Hold.
[0037]
The display information determination unit 15 determines related information to be displayed for each object from the position where the user is pointing on the screen and the information on the position and size of the object, and passes them to the display unit 16 together with the video information (S506 in FIG. 1). The display unit 16 superimposes and displays the object related information on the video information.
[0038]
The display information determination unit 15 determines the related information to be displayed for each object by first determining the most important object from the objects included in the current video information, and then determining the object other than the most important object. Determine the importance. The degree of detail of the related information of the object is determined according to the importance of the object.
[0039]
The most important object is determined as follows.
[0040]
(1) If there is an object at the position where the user is pointing, that object is made the most important object.
[0041]
(2) If there is no object at the position where the user is pointing, the score is calculated for each object, and the object with the highest score is set as the most important object.
[0042]
The score for each object is calculated in consideration of the size of the display area of the object, the closeness to the center of the screen, and the rate of continuous display. For example, this is performed as follows.
[0043]
(1) The center of gravity and area of the object are obtained from the position and size of the object. For simplicity, for example, a rectangle circumscribing the object may be determined, and the center of gravity and area of the rectangle may be substituted.
[0044]
(2) The index of the proximity of the object to the screen center is, for example, the reciprocal of the distance from the center of gravity of the object to the screen center.
[0045]
(3) The ratio of continuously displayed is, for example, the ratio of how many frames have been continuously displayed in the video for the last minute.
[0046]
(4) A weight is added to the size of the display area of the object, the proximity to the center of the screen, and the ratio of continuous display to obtain the object score.
[0047]
The weight may be set to 0 when calculating the score of the object.
[0048]
In other words, the score may be determined based on only the size of the display area of the object, the proximity to the center of the screen, the ratio of being displayed continuously, or the score based on only some combinations instead of all. May be determined.
[0049]
In addition, the importance of each object takes into account whether the object is the most important object, the score of the object, how much the object is pointing, how long the object is pointing to the screen, and the number of objects included in the video. And decide. For example, it is determined as follows.
[0050]
(1) If the object is the most important object, increase the importance.
[0051]
(2) Increase the importance according to the score of the object.
[0052]
(3) If the object is pointed, increase the importance by a certain amount.
[0053]
(4) If the point is continuously pointed on the screen, the importance is increased. For example, if a threshold value and a change rate are set, and the number of seconds or frames that are continuously pointed to does not exceed the threshold value, the importance is gradually increased according to the change rate, and if the threshold value is exceeded, Increase importance by a certain amount.
[0054]
(5) Increase or decrease the importance according to the number of objects included in the video. For example, if the number of objects is small, the importance of each object is relatively increased, and if the number of objects is large, the importance of the object is decreased.
[0055]
(6) Determine the value to be increased or decreased from whether the object is the most important object, the object score, whether the object is pointed, how long the object is pointing to the screen, or the number of objects included in the video, and the weight And add together to determine the importance of each object.
[0056]
The weight may be set to 0 when calculating the importance of the object.
[0057]
In other words, the score is determined based only on whether the object is the most important object, the object score, whether the object is pointed, how long the object is pointing to the screen, or the number of objects included in the video. Alternatively, the score may be determined only from some combinations instead of all.
[0058]
For example, the importance of the object is calculated as follows.
[0059]
(Object importance) = (weight of increase when object is the most important object) * (increase when object is the most important object) + (weight of object score) * (object score) + ( Weight of the increment when the object is pointed) * (Increment when the object is pointed) + (Weight of the number of frames pointing the screen continuously) * (Pointing the screen continuously Number of frames) + (weight of the number of objects included in the video) * (number determined from the number of objects included in the video)
For example, the level of detail of each object in the display information determination unit 15 is determined as follows.
[0060]
Assume that the user points on the screen at a certain point in time.
[0061]
The position pointed by the user is passed to the display information determination unit 15 through the pointing recognition unit 14 (S405 in FIG. 1). For example, assume that the pointing position is (600, 600).
[0062]
The display information determination unit 15 receives the ID, the position and the size of the object displayed in the current frame from the related information holding unit 13 using the current frame number of the video information as a key (S305 in FIG. 1). Assume that the related information holding unit 13 holds data as shown in FIG. 2 and the frame number of the current video information is 1000. The related information holding unit 13 searches for data having a frame number of 1000.
[0063]
From the related information holding unit 13 to the display information determination unit 15, the position of the object 1 is (80, 80) and the size is (100, 200), the position of the object 2 is (300, 400), and the size is (200 , 100) (S305 in FIG. 1).
[0064]
The display information determination unit 15 first determines the most important object.
[0065]
Since the position of the object 1 is (80, 80) and the size is (100, 200), the position of the object 2 is (300, 400) and the size is (200, 100), the position where the user is pointing (600 , 600) does not refer to any object, and the most important object has not yet been determined.
[0066]
Since the most important object has not been determined, the score for each object is calculated next.
[0067]
For example, assume that the screen size is (1000, 800).
[0068]
For object 1, the center of gravity of object 1 is (130, 180) and the area is 20000.
[0069]
Since the center of the screen is (500,400), the distance from the center of gravity to the center of the screen is √ ((500−130) ^ 2 + (400−180) ^ 2) = 430.5, and the proximity to the center of the screen is Reciprocal.
[0070]
Also, assuming that the video is displayed at 30 frames per second, the video is 1800 frames per minute. If object 1 has been displayed continuously for 360 frames in the last minute, the ratio is 0.2.
[0071]
For object 2, the center of gravity of object 2 is (400, 450) and the area is 20000.
[0072]
Since the center of the screen is (500, 400), the distance from the center of gravity to the center of the screen is √ ((500−400) ^ 2 + (400−450) ^ 2) = 111.8, and the proximity to the center of the screen is Reciprocal.
[0073]
Further, if the object 2 has been displayed continuously for 360 frames in the video for the last minute, the ratio is 0.2.
[0074]
Assuming that the weights of the display area size of the object, the proximity to the center of the screen, and the ratio of continuously displayed are 0.0001, 500, and 1, respectively,
The score is calculated as (score) = 0.0001 * (size of the display area of the object) + 500 / (closeness to the center of the screen) + 1 * (ratio continuously displayed).
[0075]
The score of object 1 is 0.0001 * 20000 + 500 / 430.5 + 1 * 0.2 = 3.36, and the score of object 2 is 0.0001 * 20000 + 500 / 111.8 + 1 * 0.2 = 6.67. Therefore, the object 2 is determined as the most important object.
[0076]
Next, the importance for each object is determined.
[0077]
The increment of importance when the object is pointed is set to 1, and the increment is set to 0 when the object is not pointed.
[0078]
The threshold for the number of frames when a point is continuously pointed on the screen is 900, and the rate of change is 0.002. Assume that the user is pointing somewhere on the screen continuously for 600 frames at this time. The increase / decrease amount of importance according to the number of objects included in the video is, for example, -1 / (11-N), where N is the number of objects included in the video. However, when N is 10 or more, N is calculated as 10. In addition, the weights for addition are 0.2 for the object score and 1 for all others.
[0079]
(Object importance) = (weight of increase when object is the most important object) * (increase when object is the most important object) + (weight of object score) * (object score) + ( Weight of the increment when the object is pointed) * (Increment when the object is pointed) + (Weight of the number of frames pointing the screen continuously) * (Pointing the screen continuously Number of frames in the video) + (weight of the number of objects included in the video) * (numerical value determined from the number of objects included in the video). ) = 1 * (increment when the object is the most important object) + 0.2 * (object scan) A) + 1 * to become (the number of increments) + 0.002 * (objects contained in successive number of frames that are pointing the screen) -1 / (11-image when it is pointing).
[0080]
At this time, the importance of the object 1 is that the object 1 is not the most important object, the score of the object is 3.36, the object is not pointing, and the user points somewhere on the screen for 600 consecutive frames. Since the number of objects included in the video is two, 0 + 0.2 * 3.36 + 0 + 600 * 0.002-1 / (11-2) = 1.76.
[0081]
The importance of the object 2 is that the object 2 is the most important object, the score of the object is 6.67, the object is not pointing, and the user points somewhere on the screen for 600 consecutive frames. Since the number of objects included in the video is two, 1 + 0.2 * 6.67 + 0 + 600 * 0.002-1 / (11-2) = 3.42.
[0082]
Assuming that the degree of detail of the related information of the object is obtained by rounding down the decimal point of the importance of the object, the degree of detail of the related information of the object 1 is 1, and the degree of detail of the related information of the object 2 is 3.
[0083]
The display information determination unit 15 receives related information according to the degree of detail for each object from the related information holding unit 13 (S305 in FIG. 1).
[0084]
Since the detail level of the related information of the object 1 is 1, the related information is “mobile phone”, and the detail level of the related information of the object 2 is 3, so the related information receives “PDA, pocket gear, 30,000 yen”.
[0085]
The display information determination unit 15 passes the related information determined for each object together with the video information to the display unit 16 (S506 in FIG. 1), and the display unit 16 displays the related information of the object superimposed on the video information.
[0086]
FIG. 3 is an explanatory diagram illustrating an example of related information superimposed on video information.
[0087]
Referring to FIG. 3, related information is displayed below the object.
[0088]
Next, a second embodiment of the present invention will be described in detail with reference to the drawings.
[0089]
FIG. 4 is an explanatory diagram showing the configuration of the second exemplary embodiment of the present invention.
[0090]
Referring to FIG. 4, the second embodiment of the present invention is a face image recognition unit for recognizing whether or not the user is turning his / her face toward the display unit 16 in addition to the components of the first embodiment. 17. When the recognition result from the pointing recognition unit 14 is passed to the display information determination unit 15, the recognition result as to whether or not the user faces the display unit 16 is also passed from the face image recognition unit 17 to the display information determination unit 15 at the same time. And used to determine how detailed relevant information is displayed for each object.
[0091]
Information is passed from the face image recognition unit 17 to the display information determination unit 15 at regular intervals, and the display information determination unit 15 accumulates information passed from the face image recognition unit 17. Further, when the user does not face the display unit 16, information may not be sent, or information that the user does not face the display unit 16 may be passed. Using this accumulated information, the display information determination unit 15 determines whether or not the user is continuously facing the display unit 16.
[0092]
For example, when information is passed from the face image recognition unit 17 to the display information determination unit 15 at 1/30 second intervals, which is a timing synchronized with the video frame, the user places the face on the display unit 16 continuously 30 times. If the information indicating that the user is facing is passed, it is determined that the user is continuously facing the display unit 16 for 30 frames (1 second). Also, for example, when information is passed at intervals of 1/10 second, if information that the user is facing the display unit 16 is passed 10 times in succession, the number of frames is 30 frames (1 second). It is determined that the user is continuously facing the display unit 16.
[0093]
In the second embodiment of the present invention, processing is performed in the same manner as in the first embodiment of the present invention. In the second and second embodiments of the present invention, the method of determining the importance for each object is different from that of the second and first embodiments of the present invention.
[0094]
In the second embodiment of the present invention, the importance for each object is determined based on whether the object is the most important object, the score of the object, whether the object is pointed, how long the object is pointing on the screen, In addition to the number of objects included in the image, it is determined in consideration of the recognition result of the face image. For example, it is determined as follows.
[0095]
(1) If the orientation of the user's face is continuously directed toward the screen, the importance of the object is increased.
[0096]
(2) Even if the user's face is continuously facing the screen direction, if the time during which the screen is not pointing is long, it is considered that the user's face is more concentrated than the related information, and the importance of the object Decrease.
[0097]
(3) Similar to the first embodiment, whether the object is the most important object, the score of the object, whether the object is pointed, how long the object is pointing to the screen, the number of objects included in the video, In addition to this, a value to be increased / decreased from a time when the face is directed to the screen or from a time when the screen is not pointing is determined, and the importance of each object is determined by adding weights.
[0098]
The weight may be set to 0 when calculating the importance of the object. In other words, whether the object is the most important object, the object score, whether the object is pointing, how long the object is pointing to the screen, the number of objects included in the video, whether the face is facing the screen, The score may be determined only from any of the times when the screen is not pointing, or the score may be determined from only some combinations instead of all of them.
[0099]
The importance of an object is calculated as follows, for example.
[0100]
(Object importance) = (weight of increase when object is the most important object) * (increase when object is the most important object) + (weight of object score) * (object score) + ( Weight of the increment when the object is pointed) * (Increment when the object is pointed) + (Weight of the number of frames pointing the screen continuously) * (Pointing the screen continuously Number of frames in the video) + (weight of the number of objects included in the video) * (numerical value determined from the number of objects included in the video) + (the user's face is continuously directed toward the screen) The weight when the user's face is continuously facing the screen) + (the continuous weight) -Even if the face of the user is facing the screen, the weight of the decrease when the screen is not pointing for a long time) * (Even if the user's face is continuously facing the screen, Decrease when the screen is not pointing for a long time).
[0101]
For example, the receiving unit 12 receives the data broadcast program information and passes the video information included in the data broadcast program information to the display information determination unit 15 (S205 in FIG. 4). At the same time, the detail level of the object included in the data broadcast program information is attached. The related information is transferred to the related information holding unit 13 (S203 in FIG. 4). The display information determination unit 15 passes the video information passed from the reception unit 12 to the display unit 16 (S506 in FIG. 4), and the display unit 16 displays the video information. The recognition result of the user's face orientation is transmitted from the face image recognition unit 17 to the display information determination unit 15 (S705 in FIG. 4).
[0102]
In addition, the user looks at the video information displayed on the display unit 16 and designates an object of interest contained in the video using the pointing recognition unit 14, and the user designates on the screen through the pointing recognition unit 14. The position is passed to the display information determination unit 15 (S405 in FIG. 4).
[0103]
For example, the display information determination unit 15 determines the detail level of each object as follows.
[0104]
Assume that the user points on the screen at a certain point in time. The position pointed by the user is passed to the display information determination unit 15 through the pointing recognition unit 14 (S405 in FIG. 4).
[0105]
For example, assume that the pointing position is (600, 600).
[0106]
The display information determination unit 15 receives the ID of the object displayed in the current frame, its position and size from the related information holding unit 13 using the current frame number of the video information as a key (S305 in FIG. 4). Assume that the related information holding unit 13 holds data as shown in FIG. 2 and the frame number of the current video information is 1000. The related information holding unit 13 searches for data having a frame number of 1000. From the related information holding unit 13 to the display information determination unit 15, the position of the object 1 is (80, 80) and the size is (100, 200), the position of the object 2 is (300, 400), and the size is (200 , 100) (S305 in FIG. 4).
[0107]
Similar to the first embodiment of the present invention, the object 2 is determined as the most important object.
[0108]
Next, the importance for each object is determined.
[0109]
The increment of importance when the object is pointing is set to 1. The threshold for the number of frames when a point is continuously pointed on the screen is 900, and the rate of change is 0.002. Assume that the user is pointing somewhere on the screen continuously for 600 frames at this time. The increase / decrease amount of importance according to the number of objects included in the video is, for example, -1 / (11-N), where N is the number of objects included in the video. However, when N is 10 or more, N is calculated as 10.
[0110]
The threshold value for the number of frames when the user's face is continuously facing the screen is 900, and the rate of change is 0.001. It is assumed that the threshold of the number of frames is 900 and the rate of change is 0.001 when the user's face is continuously facing the screen and the screen is not pointing. It is assumed that the user has continuously faced the face in the screen direction for 900 frames or more at this time.
[0111]
In addition, the weights for addition are 0.2 for the object score and 1 for all others.
[0112]
(Object importance) = (weight of increase when object is the most important object) * (increase when object is the most important object) + (weight of object score) * (object score) + ( Weight of the increment when the object is pointed) * (Increment when the object is pointed) + (Weight of the number of frames pointing the screen continuously) * (Pointing the screen continuously Number of frames in the video) + (weight of the number of objects included in the video) * (numerical value determined from the number of objects included in the video) + (the user's face is continuously directed toward the screen) The weight when the user's face is continuously facing the screen) + (the continuous weight) -Even if the face of the user is facing the screen, the weight of the decrease when the screen is not pointing for a long time) * (Even if the user's face is continuously facing the screen, Therefore, the importance of the object is calculated as (importance of the object) = 1 * (increase when the object is the most important object) + 0.2 * ( Object score) + 1 * (increase when pointing) + 0.002 * (number of frames pointing the screen continuously) −1 / (11−number of objects included in video) + 1 * (Number of frames when the user's face is continuously facing the screen direction * rate of change)-1 * (the user's face is continuously facing the screen direction and the screen is pointed The number of frames * rate of change) and made of when not in Ingu.
[0113]
At this time, the importance of the object 1 is that the object 1 is not the most important object, the score of the object is 3.36, the object is not pointing, and the user points somewhere on the screen for 600 consecutive frames. Since the number of objects included in the video is two and the face faces the screen direction, 0 + 0.2 * 3.36 + 0 + 600 * 0.002-1 / (11-2) + 900 * 0.001 −0 = 2.66.
[0114]
The importance of the object 2 is that the object 2 is the most important object, the score of the object is 6.67, the object is not pointing, and the user points somewhere on the screen for 600 consecutive frames. Since the number of objects included in the video is two and the face faces the screen direction, 1 + 0.2 * 6.67 + 0 + 600 * 0.002-1 / (11-2) + 900 * 0.001- 0 = 4.32.
[0115]
Assuming that the degree of detail of the related information of the object is obtained by rounding down the decimal point of the importance of the object, the degree of detail of the related information of the object 1 is 2, and the degree of detail of the related information of the object 2 is 4.
Here, it is assumed that the screen is no longer pointing and the frame is over 900 frames with the face facing the screen. The number of objects in the video remains two.
[0116]
At this time, the importance of the object 1 is that the object 1 is not the most important object, the score of the object is 3.36, the object is not pointing, and the user is not pointing anywhere on the screen. Since the number of objects included in the video is two and the face faces the screen direction, 0 + 0.2 * 3.36 + 0 + 0-1 / (11-2) + 900 * 0.001-900 * 0.001 = 0.56.
[0117]
The importance of the object 2 is that the object 2 is the most important object, the object score is 6.67, the object is not pointed, and the user is not pointing anywhere on the screen, Since the number of objects included in the video is two and the face is facing the screen direction, 1 + 0.2 * 6.67 + 0 + 0-1 / (11-2) + 900 * 0.001-900 * 0.001 = 2 .22.
[0118]
Assuming that the degree of detail of the related information of the object is obtained by rounding down the decimal point of the importance of the object, the degree of detail of the related information of the object 1 is 0, and the degree of detail of the related information of the object 2 is 2.
[0119]
The display information determination unit 15 receives the related information according to the degree of detail for each object from the related information holding unit 13 (S305 in FIG. 4). The display information determination unit 15 passes the related information determined for each object together with the video information to the display unit 16 (S506 in FIG. 4), and the display unit 16 displays the related information of the object superimposed on the video information.
[0120]
In this way, it becomes possible to automatically adjust the related information of the object presented to the viewer.
[0121]
【The invention's effect】
The effect of the present invention is that man-hours for optimizing the information displayed by the program provider can be reduced, and the excess or deficiency of information displayed to the viewer can be reduced.
[0122]
The reason is that a plurality of pieces of information related to objects in the video are prepared for each degree of detail, and the degree of detail of information displayed by the system is determined from the viewer's state and operation history.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of a first exemplary embodiment of the present invention.
FIG. 2 is an explanatory diagram illustrating an example of object information and related information.
FIG. 3 is an explanatory diagram illustrating a display state in which related information is superimposed on an object.
FIG. 4 is a block diagram showing a configuration of a second exemplary embodiment of the present invention.
[Explanation of symbols]
11 Program recording medium
12 Receiver
13 Related information holding section
14 Pointing recognition unit
15 Display information determination unit
16 Display section
17 Face image recognition unit

Claims (10)

Receiving data broadcast program information including video information and related information with details including “ object information including position and size of objects present in the frame” and “ related information corresponding to details for each object ” ; A receiver that outputs video information, object information, and related information;
Object information from the receiving unit, a related information holding unit for holding related information,
Recognizing the pointing position on the screen designated by the user, the pointing recognition unit which outputs the pointing position information,
Input video information from the receiving unit, object information from the related information holding unit, related information, and pointing position information from the pointing recognition unit,
(A) If there is an object at the pointing position indicated by the pointing position information, that object is regarded as the most important object. (B) If there is no object at the pointing position indicated by the pointing position information, a score is calculated for each object. The object with the highest score is regarded as the most important object. (C) The score for each object is obtained by adding a weight to the size of the display area of the object, the closeness to the center of the screen, and the ratio of continuous display. And combined
Depending on the object's score, increase the importance,
Calculate the importance of the object as (weight of increase when the object is the most important object) * (increase when the object is the most important object) + (weight of the object score) * (score of the object) The value of the calculated importance of the object rounded down is used as the object detail, and the calculated object detail is applied to the related information to determine the object-related related information, and the video information and object- related A display information determination unit that outputs continuous information;
A display unit that superimposes and displays related information to be displayed corresponding to the object from the display information determination unit on the video information from the display information determination unit;
An information superimposing display device comprising: A face image recognition unit that generates face direction information indicating whether a user's face is facing the display unit and outputs the information to the display information determination unit;
Refer to the face direction information from the face image recognition unit,
The importance of an object is expressed as (weight increase when the object is the most important object) * (increase weight when the object is the most important object) + (object score weight) * (object score) + (predetermined time) Increased weight when the user's face is continuously facing the display unit) * (when the user's face is continuously facing the display unit for a certain period of time based on face direction information) The display information determination unit to calculate as
The information superimposing display device according to claim 1, further comprising : (A) If the object is the most important object, increase the importance,
(B) If the object is pointing, increase the importance by a certain amount,
(C) If somewhere on the screen is pointing continuously, increase the importance,
(D) Increase or decrease the importance according to the number of objects included in the video,
(E) Determine whether the object is the most important object, the score of the object, whether the object is pointed, how long the screen is pointing, and the value to be increased or decreased from the number of objects included in the video,
The importance of an object is expressed by (object score weight) * (object score) + (increase weight when the object is the most important object) * (increase amount when the object is the most important object) + (object Weight when the object is pointing) * (increment when the object is pointing) + (weight of the number of frames that are continuously pointing to the screen) * (pointing the screen continuously) The display information determining unit is calculated as (number of frames included) + (weight of the number of objects included in the video) * (numerical value determined from the number of objects included in the video). Item 1. The information superimposing display device according to Item 1. (A) If the object is the most important object, increase the importance,
(B) If the object is pointing, increase the importance by a certain amount,
(C) If somewhere on the screen is pointing continuously, increase the importance,
(D) Increase or decrease the importance according to the number of objects included in the video,
(E) Determine whether the object is the most important object, the score of the object, whether the object is pointed, how long the screen is pointing, and the value to be increased or decreased from the number of objects included in the video,
The importance of an object is expressed as (weight of object score) * (object score) + (weight of increase when the user's face is facing the display unit for a certain period of time) * (face According to the direction information, the increment when the user's face is facing the display unit continuously for a certain period of time) + (the weight of the increment when the object is the most important object) * (the object is the most important object) Increment) + (weight of increment when object is pointing) * (increment when object is pointing) + (weight of the number of frames pointing to the screen continuously) * (The number of frames that point to the screen continuously) + (weight of the number of objects included in the video) * (the number of objects included in the video) The display information determination unit information superimposing display device according to claim 2, wherein a calculating as a numerical value) determined. An information superimposed display method in an information superimposed display device comprising a receiving unit, a related information holding unit, a pointing recognition unit, a display information determination unit, and a display unit,
Data broadcast program information in which the receiving unit includes video information, “object information including the position and size of an object present in a frame”, and “related information with a level of detail for each object” Receiving video and outputting video information, object information, and related information;
The related information holding unit holds object information and related information from the receiving unit, and
A procedure for the pointing recognition unit to recognize a pointing position on a screen designated by a user and to output pointing position information;
The display information determination unit inputs video information from the reception unit, object information from the related information holding unit, related information, and pointing position information from the pointing recognition unit;
(A) If there is an object at the pointing position indicated by the pointing position information, that object is regarded as the most important object. (B) If there is no object at the pointing position indicated by the pointing position information, a score is calculated for each object. The object with the highest score is regarded as the most important object. (C) The score for each object is obtained by adding a weight to the size of the display area of the object, the closeness to the center of the screen, and the ratio of continuous display. The combined procedure,
Depending on the object's score, increase the importance,
Calculate the importance of the object as (weight of increase when the object is the most important object) * (increase when the object is the most important object) + (weight of the object score) * (score of the object) The value of the calculated importance of the object is rounded down to the object detail level, the calculated object detail level is applied to the related information, and the object-related related information is determined.
A procedure for outputting video information and object-related information;
The display unit is
A table corresponding to the object from the display information determination unit is added to the video information from the display information determination unit. To superimpose related information to be displayed,
An information superimposing display method comprising: An information superimposed display method in the information superimposed display device including a face image recognition unit,
A procedure in which the face image recognition unit creates face direction information indicating whether a user's face is facing the display unit and outputs the information to the display information determination unit;
The display information determination unit refers to face direction information from the face image recognition unit,
The importance of an object is expressed as (weight increase when the object is the most important object) * (increase weight when the object is the most important object) + (object score weight) * (object score) + (predetermined time) Increased weight when the user's face is continuously facing the display unit) * (when the user's face is continuously facing the display unit for a certain period of time based on face direction information) As an increase in
The information superimposed display method according to claim 5, further comprising: The display information determination unit (a) increases the importance if the object is the most important object,
(B) If the object is pointing, increase the importance by a certain amount,
(C) If somewhere on the screen is pointing continuously, increase the importance,
(D) Increase or decrease the importance according to the number of objects included in the video,
(E) Determine whether the object is the most important object, the score of the object, whether the object is pointed, how long the screen is pointing, and the value to be increased or decreased from the number of objects included in the video,
The importance of an object is expressed by (object score weight) * (object score) + (increase weight when the object is the most important object) * (increase amount when the object is the most important object) + (object Weight when the object is pointing) * (increment when the object is pointing) + (weight of the number of frames that are continuously pointing to the screen) * (pointing the screen continuously) 6. The method according to claim 5, further comprising a procedure of calculating: (number of frames included) + (weight of the number of objects included in the video) * (numerical value determined from the number of objects included in the video). Information superposition display method. The display information determination unit (a) increases the importance if the object is the most important object,
(B) If the object is pointing, increase the importance by a certain amount,
(C) If somewhere on the screen is pointing continuously, increase the importance,
(D) Increase or decrease the importance according to the number of objects included in the video,
(E) Determine whether the object is the most important object, the score of the object, whether the object is pointed, how long the screen is pointing, and the value to be increased or decreased from the number of objects included in the video,
The importance of an object is expressed as (weight of object score) * (object score) + (weight of increase when the user's face is facing the display unit for a certain period of time) * (face According to the direction information, the increment when the user's face is facing the display unit continuously for a certain period of time) + (the weight of the increment when the object is the most important object) * (the object is the most important object) Increment) + (weight of increment when object is pointing) * (increment when object is pointing) + (weight of the number of frames pointing to the screen continuously) * (The number of frames that point to the screen continuously) + (weight of the number of objects included in the video) * (the number of objects included in the video) Information superimposing display method according to claim 6, characterized in that it comprises the steps of calculating a numerical value) determined. An information superimposed display program in an information superimposed display device comprising a receiving unit, a related information holding unit, a pointing recognition unit, a display information determination unit, and a display unit,
Data broadcasting program information including video information, “object information including the position and size of an object existing in a frame”, and “related information with a detail level for each object” in the receiving unit And execute a procedure for outputting video information, object information, and related information,
Causing the related information holding unit to execute a procedure for holding object information and related information from the receiving unit;
Recognizing the pointing position on the screen designated by the user in the pointing recognition unit, and outputting the pointing position information;
The display information determination unit inputs video information from the reception unit, object information from the related information holding unit, related information, and pointing position information from the pointing recognition unit;
(A) If there is an object at the pointing position indicated by the pointing position information, that object is regarded as the most important object. (B) If there is no object at the pointing position indicated by the pointing position information, a score is calculated for each object. The object with the highest score is regarded as the most important object. (C) The score for each object is obtained by adding a weight to the size of the display area of the object, the closeness to the center of the screen, and the ratio of continuous display. The combined procedure,
Depending on the object's score, increase the importance,
Calculate the importance of the object as (weight of increase when the object is the most important object) * (increase when the object is the most important object) + (weight of the object score) * (score of the object) The value of the calculated importance of the object is rounded down to the object detail level, the calculated object detail level is applied to the related information, and the object-related related information is determined.
And a procedure for outputting video information and related information corresponding to the object,
In the display section,
Causing video information from the display information determination unit to superimpose and display related information to be displayed corresponding to the object from the display information determination unit;
An information superimposing display program characterized by that. An information superimposed display program in the information superimposed display device comprising a face image recognition unit,
Causing the face image recognition unit to create face direction information indicating whether a user's face is facing the display unit, and to execute a procedure of outputting to the display information determination unit;
The display information determination unit refers to face direction information from the face image recognition unit,
The importance of an object is expressed as (weight increase when the object is the most important object) * (increase weight when the object is the most important object) + (object score weight) * (object score) + (predetermined time) Increased weight when the user's face is continuously facing the display unit) * (when the user's face is continuously facing the display unit for a certain period of time based on face direction information) To calculate as
The information superimposition display program according to claim 9.

Images