JP4907483B2 - Video display device - Google Patents

Description

  The present invention is a video display device such as a TV, and in particular, based on user sensing information such as a user's position / orientation, line of sight, hand gesture, user's hand / arm movement, as well as conventional button input. The present invention relates to a video display device.

  The present invention also relates to a method for controlling an application service provided by a video display apparatus based on a combination of such operations on the premise of operation by a plurality of persons.

  Furthermore, the present invention relates to a system that creates a sense of realism as if the user is in the place by a full-scale display utilizing a large screen of a video display device and natural interaction.

  Along with the increase in screen size and thinning, not only watching TV programs and movies, but also using multiple information simultaneously, listing large amounts of information, communication using real-world presence, always presenting information The possibility of new usage methods such as bulletin boards, interior decorations such as wallpaper and picture frames is expanding.

  In addition, with the rise of the home network, it is also becoming realistic to share the user's operation history and status detected by each device via the network and comprehensively sense the user's context and situation (for example, patents) Reference 1).

  In this situation, in addition to watching conventional programs and movies, in order to make it easier to use the more complex functions required by the above new usage method, a new intuitive operation method Therefore, it is necessary to realize an autonomous operation method that reduces the user's explicit operation amount and burden by drawing the user's situation and intention based on the user sensing information.

In addition, in order to cope with usage that is not only passive viewing while taking advantage of the characteristics of the TV set in the shared space of the family such as the living room, an input method and operation system based on operation by multiple people are adopted. It is desirable.
JP 2004-246856 A

  The present invention is for solving the above-described problems, and the object of the present invention is to provide an intuitive operation by a plurality of persons by controlling an application for displaying the video corresponding to one or more viewing target users. An object of the present invention is to provide a video display device that can be operated.

  Furthermore, it is providing the method of controlling the service application provided by such a video display apparatus.

In order to solve the above problems, a video display device according to the present invention is a video display device operated by one or more users, and includes identification means for identifying one or more users, positions and operations of one or more users. Detecting means for detecting; display means for displaying video; and discriminating means for discriminating one or more users who are viewing targets of the displayed video based on the position and operation of the detected user among the identified one or more users. A control unit for controlling an application for displaying video; and a user database for storing user attribute information. The attribute information includes body feature information including at least body information, and a human relationship between one or more users. information is included, the control means, especially to control the application corresponding to said attribute information of one or more viewing target user, and displays the video To.
Here, the human relationship information between the one or more users is information on intimacy between users, and the control means determines whether information can be shared according to the degree of intimacy between the users. The application for displaying the video may be controlled.
In addition, the video display device further controls an application that performs a cooperative operation with a partner video display device installed in another location and connected to the network, and receives data for receiving operation information transmitted from the partner video display device. The scale conversion is performed based on the presentation screen size for the information on the object to be displayed in the actual size of the life-size video captured by the counterpart video display device and the operation information on the object among the operation information Life-size display conversion means, life-size information adding means for adding life-size information necessary for life-size display on the counterpart video display device to operation information based on user size information and user position information, Data transmitting means for transmitting operation information and life-size video to which the large information is added to the counterpart video display device Wherein the control means, together with the input information, processes the operation information after the scale conversion may control the application.
The video display device may further include a remote controller that obtains user explicit input information, and the control unit may control the application based on a plurality of relative positional relationships of the remote controller.
Further, the remote control is spherical, and the user can throw the remote control or roll it on the surface, and the control means displays the video based on the information on the trajectory or stop position of the remote control. You may control the application to do.

  The video display device of the present invention is characterized in that the control means corrects a display position and / or a size of a video to be viewed corresponding to at least a viewing position of one or more viewing target users.

  The video display device of the present invention includes a user database that stores user attribute information. The attribute information includes body feature information including at least height information, and the control means includes one or more viewing targets. An application for displaying the video is controlled corresponding to each attribute information of the user.

  The video display device according to the present invention further includes management means for managing each viewing state of the viewing target user in a standing position or a sitting position, and the body characteristic information includes the height and eye height in the standing position. The height and eye height information in the sitting position is included, and the control means controls an application for displaying the video based on the viewing state and body characteristic information of the viewing target user.

  In the video display device of the present invention, the body characteristic information includes dominant eye information and visual acuity information, and the control means displays the video based on the dominant eye information and visual acuity information of the viewing target user. It is characterized by controlling an application to be displayed.

  According to the video display device of the present invention, each of the viewing target users corresponds to one or more viewing target users, and the viewing target user is controlled by controlling an application that displays the video according to the attribute or situation of the viewing target user. Since multiple modals can be switched seamlessly, the convenience of TV compared to a PC is ensured, and a foundation for various TV enjoyment by multiple people that goes beyond viewing traditional programs and movies. Provided.

  The best mode for carrying out a video display device and a control method according to the present invention will be described below with reference to the accompanying drawings.

(Embodiment 1)
FIG. 1 is a diagram for explaining an example of an interface between an appearance of a video display device and related devices in the present embodiment. The video display device includes one or a plurality of user detection cameras in addition to an antenna for receiving a broadcast program. The video display device is controlled by the position and movement of the user detected by analysis of image information captured by the user detection camera.

  The video display device may be controlled based on the user's remote control operation such as the movement of the user's hand holding the gesture input remote control or the pressing of a button arranged on the remote control.

  Although not shown in FIG. 1, the plurality of speakers in the video display device may be arranged apart from each other such as an upper end and a lower end, and a left end and a right end of the video display device.

  If the speaker output is controlled according to the user's attribute, position, application state, etc., it is possible to give the user a greater sense of realism.

  In addition, the video display device is connected to other home appliances such as a digital still camera and a digital camcorder via a wireless network and a router / hub.

  The video display device can present the digital information / operation menu received from these devices on the screen, and can transmit the user's operation information for the menu presented on the screen to these devices.

  Furthermore, when the video display device is connected to a video display device installed at another location via a network, users in remote locations can share digital information such as photographs and videos via the video display device. it can.

  It should be noted that the video display device enables users to collaborate such that a slide show is composed of a plurality of photos.

  If the Internet is used as a means for network connection, distant users can share information and collaborate.

(Embodiment 2)
FIG. 2 is a diagram illustrating an example of a situation in which a plurality of users are located in a room having the video display device according to the present embodiment.

  FIG. 3 is a block diagram showing functions of the video display device, and FIG. 4 is a diagram for explaining the outline of the data structure of the user information DB (database) in FIG. FIG. 5 is a flowchart showing an outline of processing of the video display apparatus.

Hereinafter, the function of each block in FIG. 3 will be described.
The user detection camera 100 captures a user existing in front of the video display device. The user position detecting unit 101 extracts a user area from a plurality of videos captured by the user detection camera 100, and then uses a stereo vision principle to determine the relationship between the user and the video display device based on the correspondence between user areas in the plurality of videos. The relative position is calculated. After extracting the face area, the user identification unit 102 collates the face image registered in advance with the extracted face image, and outputs user identification information for identifying the user. The line-of-sight detection means 103 calculates the user's line-of-sight direction based on the position of the user's eye black eye region. After extracting the user's hand region, the hand position / shape detecting unit 104 collates with which shape the hand position and the hand shape match in advance, for example, “goo” “par” “right selection” Outputs hand shape information such as “select left”.

  The gesture input remote controller 105 includes a button input unit 1051 that outputs a button state indicating which of the arranged buttons is pressed by the user, and a motion sensor 1052 that detects the movement of the user's hand holding the gesture input remote controller 105. , Gesture identification means 1053. The motion sensor 1052 outputs hand movement information including movement and rotation. The gesture identification unit 1053 collates the output from the motion sensor with predetermined hand movement (gesture) information, and outputs gesture type information such as “shake”, “turn”, and “throw” and an operation amount. .

  The application control unit 106 includes a user position output by the user position detection unit 101, user identification information output by the user identification unit 102, a line-of-sight direction output by the line-of-sight detection unit 103, and a hand position output by the hand position / shape detection unit 104. The hand shape information, the button state output by the button input unit 1051, and the gesture type information output by the gesture identification unit 1053 are interpreted as input information. In addition to these input information, the application control means 106 controls the application based on input information from the user information DB 107A, the user state management means 107B, the modal control means 107C, and the viewing state determination means 107D.

  As shown in FIG. 4, the user information DB 107A stores basic attribute information, body feature information, human relationship information, and the like. The basic attribute information is, for example, name, sex, age, birthday, relationship, and the like. The user status management means 107B manages the user status for each user. The modal control means 107C selects a corresponding modal for each user, and determines a function to be executed next and a state to be transitioned based on the combination with the current state. The viewing state determination unit 107D determines, among the one or more identified users, a user who is a viewing target of the display video in consideration of at least the detected position or operation of the user.

  When updating the drawing content of the screen, the application control unit 106 outputs update information of the drawing content to the screen drawing unit 110. The screen drawing unit 110 presents the drawn screen content on the screen 111.

  The application control means 106 can select information to be presented to the user based on the basic attribute information. Thereby, the video display device can select and display news related to or interested in each user based on the age and occupation of each user, for example. Further, taking the user in FIG. 4 as an example, a message prompting her husband, Taro, to purchase a birthday present for Hanako, two weeks before December 21st, which is Matsushita's birthday. , Past gift histories, and suggested gift candidates. In addition, a congratulatory message to Hanako can be displayed on December 21st of the birthday.

  Also, the body feature information stored in the user information DB 107A is the height and eye height in the standing position, the height and eye height in the sitting position, the dominant hand, the dominant eye, the eyesight, the hearing, etc. Includes body shape and viewing ability.

  The application control means 106 can present information at a position and size that is easy for the user to see and a volume that is easy to hear based on the body feature information. For example, when the user recognizes that the user is standing, the video display device can display information centered on the eye height position of the user based on the information on the eye height when standing. Also, the video display device can increase the display size for users with poor vision. Fine adjustment of the display may be performed based on the dominant eye information.

  Also, if hand gesture with one hand is performed with a dominant hand, the display position of an icon or the like that prompts hand gesture can be determined based on the dominant hand information. That is, for a right-handed user, an icon or the like that prompts a hand gesture at a position on the screen corresponding to the right of the user's position, and for a left-handed user at a position on the screen corresponding to the left of the user's position. By displaying, a more natural interaction can be realized.

  Furthermore, the human relationship information stored in the user information DB 107A stores the familiarity between users registered in the database as 0.0 to 1.0 as shown in FIG. 4C, for example. The application control means 106 can control whether or not information can be shared among users based on the familiarity.

  As described above, the video display device of the present invention controls applications that display video corresponding to one or more viewing target users.

  Further, in the case of an application that performs a cooperative operation with a video display device installed at another location and connected to a network, the data receiving unit 108 receives operation information transmitted from the counterpart video display device. The life-size display conversion means 109 is dependent on the display size of the operation information (information of an object that is to be displayed in real size, such as a life-size video taken by the other-party video display device, and operation information for the object). Scale conversion is performed based on the presentation screen size. The application control unit 106 controls the application after processing the scale-converted operation information together with the input information.

  Further, the application control means 106 generates operation information for the counterpart video display device. In addition, the application control unit 106 uses the user size information acquired from the user information DB 107A based on the user identification information input from the user identification unit 102 and the user position information input from the user position detection unit 101 as life-size information. You may input into the addition means 112. FIG. When the input operation information is a life-size image taken by the user detection camera 100, the life-size information adding means 112 is based on the size information of the user and the user position information, and the life-size image on the counterpart image display device. Life-size information necessary for large display is added to the operation information. The data transmission means 113 transmits the operation information and the life-size video to which the life-size information is added to the counterpart video display device.

Here, the processing flow of the video display apparatus will be described with reference to the flowchart of FIG.
First, when the user detection camera 100 detects a face, the user identification unit 102 performs user identification by collating with body feature information stored in the user information DB 107A registered in advance (S201). For each identified user, the user position detection unit 101 calculates user position information, the line-of-sight detection unit 103 calculates line-of-sight direction information, and the hand position / shape detection unit 104 calculates hand position / hand shape information. (S202). Then, in view of at least the position or operation of the detected user among the one or more users identified in S201, the viewing state determination unit 107D determines the user who is the viewing target of the display video (S203).

  For example, taking the use scene shown in FIG. 2 as an example, since user A is standing and moving in the vicinity of the video display device, the user A is determined as a viewing target user. User B is determined as a viewing target user because he is sitting on the sofa and facing the video display device. Since the user C is standing and turning away from the video display device, the user C is determined not to be a viewing target user.

  When the number of users determined as viewing target users is plural (YES in S204), the modal control means 107C determines the form of collaboration work from the user position information and current application information (S205).

  The forms of collaboration work include a form in which a plurality of persons share the same work at the same time and a form in which a plurality of people perform separate work at the same time. In the former, there is a form in which one person takes the initiative and the other person gives advice, and multiple persons work in the same position.

  Then, based on the user position information, line-of-sight direction information, hand position / hand shape information calculated in S202, and the current application state, the modal control means 107C determines the operation modal for each user (S206). The operation modals mentioned here are the following four methods: “operation method by user position”, “operation method by gaze / face orientation”, “operation method by freehand gesture”, “operation method by gesture input remote control”. Either. When the operation modal is determined, an operation command is generated from the corresponding input information, for example, the hand shape information such as “goo”, “par”, “right selection”, “left selection” if the operation method is a freehand gesture. (S207).

  If the application requires life-size display (YES in S208), the scale information necessary for life-size display is acquired (S209), and life-size display information is generated (S210).

  Then, the application control unit 106 determines a function to be executed next from the operation command generated in S207 and the current application state (S211), and executes the function (S212).

(Embodiment 3)
FIG. 6 shows a user position calculation method based on the principle of stereo vision in the user position detection means 101 in this embodiment.

  As shown in FIG. 6A, two user detection cameras 100 are installed in parallel with the screen of the video display device at a distance B as a set, and the user position detection means 101 is photographed by each camera. The distance D between the user and the screen of the video display device is calculated based on the position shift of the corresponding user area in the image. Extraction of the region in which the user is photographed in the image photographed by each camera is, for example, stored in advance in the state where there is no user, and the difference from the image when the user appears It can be realized by seeking. It is also possible to obtain a user's face area by face image detection and face image collation, and use the face area as the user area.

FIG. 6B shows the principle of stereo vision for obtaining the distance D between the user and the camera installation surface (screen of the video display device) based on the corresponding user areas on the two images. Assuming that the user area corresponding to each of the images captured by the two cameras is a position measurement target, the images are projected onto the two images as shown in FIG. 6B. If the shift of the corresponding image on the image is Z, the distance D between the user and the video display device is calculated from the focal length f of the camera and the distance B between the optical axes of the camera.
D = f × B / Z
Is required. Further, the user position in the direction parallel to the screen of the video display device can be obtained based on the position of the user area in the image and the above distance D. The relative position of the user with respect to the video display device thus obtained is input from the user position detection unit 101 to the application control unit 106.

  Based on this user position information, the application control means 106 determines the display position on the screen of the video display device of information to be presented to the user, for example, so that the information shown in FIG. 7A and FIG. As shown, even if the user moves, it is possible to keep presenting information at a position that is always easy to see from the user. For example, as shown in FIG. 7A, if the user moves in front of the video display device, information can be displayed at a position on the screen close to the position of the user. Also, as shown in FIG. 7B, when the user approaches or moves away from the video display device, the application control means 106 reduces or enlarges the information display size and displays it in a size that is easy for the user to see. Can do. In particular, when the user approaches the video display device and can determine that he / she is interested in the displayed information, the information displayed by the application control means 106 is set to a more detailed information and displayed. Also good. Furthermore, as shown in FIG. 7C, information can be displayed at a height that is easy to see according to the height position of each user's face. Such a method of operating the video display device in accordance with the position of the user is called an operation method based on the user position.

(Embodiment 4)
8 and 9 show a gaze direction detection method in the gaze detection means 103 in the present embodiment.

  The line-of-sight direction is calculated based on the combination of the face direction and the direction of the black eye part in the eye. Therefore, first, the three-dimensional face direction of the person is estimated, then the direction of the black eye is estimated, and the gaze direction is calculated by integrating the two.

  As illustrated in FIG. 8A, the line-of-sight detection unit 103 first estimates the face orientation of the face from the image captured by the user detection camera 100. As a face direction estimation method, for example, it can be estimated by using the method described below with reference to FIGS. 8B, 9A, and 9B. FIG. 8B shows the overall flow. Areas of facial part feature points such as eyes, nose and mouth in the detected face area are prepared in advance for each of several face orientations. In the example of FIG. 9A, a face part feature point region is prepared in front of the face and ± 20 degrees to the left and right. A template image obtained by cutting out the area around each facial part feature point is prepared.

  First, the user detection camera 100 captures a user existing in front of the video display device (S401), and detects a face area from the captured image (S402). Next, a face part feature point area corresponding to each face orientation is applied to the detected face area (S403), and a region image of each face part feature point is cut out. The correlation between the clipped region image and the template image prepared in advance is calculated (S404), a weighted sum obtained by weighting the angle of each face by the correlation ratio is obtained, and this is set as the face orientation of the detected face (S405). . In the example of FIG. 9A, the correlation for the face orientation +20 degrees is 0.85, the correlation for the front orientation is 0.14, and the correlation for −20 degrees is 0.01, so the face orientation is 20 × 0.85 + 0. It is calculated as × 0.14 + −20 × 0.01 = 16.8 degrees.

  Here, each face part area is a target of correlation calculation, but the present invention is not limited to this. For example, the whole face area may be a target of correlation calculation. As another method, a method is known in which facial part feature points such as eyes, nose and mouth are detected from a face image, and the face orientation is calculated from the positional relationship of the facial part feature points. As a method of calculating the face direction from the positional relationship of the facial part feature points, rotate and enlarge the 3D model of the facial part feature points prepared in advance to best match the facial part feature points obtained from one camera. The method of calculating the face direction from the amount of rotation of the three-dimensional model obtained by reducing and matching, and the principle of stereo vision based on the images taken by two cameras as described in the third embodiment. Using the left and right cameras to calculate the three-dimensional position of each facial part feature point from the displacement of the facial part feature point position on the image, and to calculate the orientation of the face from the positional relationship of the obtained facial part feature points is there. For example, a method is known in which the normal direction of the plane stretched by the three-dimensional coordinate points of both eyes and mouth is the face direction.

  The line-of-sight detection means 103 estimates the direction of the black eye after the face orientation is determined. The black eye direction can be estimated by using the following method, for example. An outline of the estimation method will be described with reference to FIGS. 8B and 9B.

  In this method, gaze detection is performed in the sequence of gaze direction reference plane calculation, black eye center detection, and gaze direction calculation.

  First, regarding the calculation of the line-of-sight reference plane, the line-of-sight reference plane in the present method is a plane that serves as a reference when calculating the line-of-sight direction, and is the same as the left-right symmetrical plane of the face. In this method, the right and left symmetry plane of the face is calculated from the three-dimensional position of the eye using the fact that the eye is less affected by facial expressions and has fewer false detections than other facial parts such as the corner of the eye, the corner of the mouth, and the eyebrows. .

  The three-dimensional position of the eye is measured by detecting the eye of the two images taken by the stereo camera using the face detection module and the face part detection module, and measuring these in stereo. (S406). As shown in FIG. 9B, the line-of-sight direction reference plane is acquired as a vertical bisector of a line segment with the detected left and right eye heads as endpoints.

  Next, regarding the detection of the center of the black eye, what humans are seeing is that light entering from the pupil reaches the retina and is transmitted to the brain as an electrical signal. Therefore, when detecting the line-of-sight direction, it is only necessary to see the movement of the pupil. However, in the case of Japanese people, since the iris is black or brown, it is difficult to distinguish it from the pupil on the image. Therefore, here, since the center of the pupil and the center of the black eye (iris) substantially coincide with each other, the center of the black eye is detected as the line-of-sight direction feature. At the center of the black eye, first, the corner of the eye and the head of the eye are detected, and the region having the minimum luminance from the eye region including the corner of the eye and the head of the eye as shown in FIG. Next, a black-eye detection filter composed of regions 1 and 2 as shown in FIG. 9C-2 is set, and a circle center is searched so that the inter-region variance of the luminance of the pixels in regions 1 and 2 is maximized. This is the center of black eyes. Finally, as before, the three-dimensional position of the center of the black eye is acquired by stereo measurement (S407).

  Further, regarding the detection of the gaze direction, the gaze direction is detected using the calculated gaze direction reference plane and the three-dimensional position of the center of the black eye. The human eyeball diameter is known to have almost no individual difference in the case of an adult, and is about 24 mm in the case of a Japanese. Therefore, if the position of the center of the black eye when the reference direction (for example, the front) is pointed is known, it can be converted and calculated in the line-of-sight direction by obtaining the displacement from there to the current center position of the black eye. In the conventional method, the position of the center of the black eye when facing the reference direction is not known, so calibration was necessary.However, in this method, when facing the front, the midpoint of the center of the left and right black eyes is the face. The direction of the line of sight is calculated by measuring the distance between the midpoint of the center of the left and right black eyes and the line of sight direction reference plane (S408).

  In this method, the line-of-sight direction is acquired as a rotation angle θ in the left-right direction with respect to the front of the face. The rotation angle θ is obtained by the following equation.

R: Eyeball radius (12 mm)
d: Distance between line-of-sight reference plane and black eye midpoint

  The line-of-sight direction in the real space can be detected by matching the three-dimensional orientation of the face calculated based on the above procedure and the orientation of the black eyes in the face (S409).

  The application control unit 106 determines, for example, the display position of information to be presented to the user on the screen of the video display device based on the gaze direction information detected by the gaze detection unit 103 in the above procedure. As a result, the present video display device can continue to present information at a position that is always easy for the user to see even if the user's line of sight / face orientation moves. A method in which the user operates the video display device by changing the line of sight / face direction is referred to as an operation method based on the line of sight / face direction.

(Embodiment 5)
FIG. 10 shows a hand position / shape detection method in the hand position / shape detection means 104 according to the present embodiment.

  As shown in FIG. 10A, the position of the person is first detected from the image captured by the user detection camera 100, and the position / shape of the hand is detected around the person position. As a method for estimating the hand position / shape, for example, the following method can be used. Hereinafter, a description will be given with reference to FIG.

  First, as offline processing, the hand position / shape detection unit 104 prepares a large number of learning images of the hand to be detected (S501). The conditions such as the lighting environment and direction in the learning image are set so as to match the actual detection environment as much as possible. Next, an eigenspace that constitutes the principal component of the hand is created from the learning image prepared in S501 using principal component analysis (S502). A hand template image is prepared as a sample of the hand to be detected. The template image may be an average image of prepared hands or images of several hands such as goo and par. The created projection matrix onto the eigenspace and the hand template image are stored in the hand template database (S503).

Next, online processing for performing actual detection will be described.
First, the user detection camera 100 captures a user existing in front of the video display device (S504), and detects a face area from the captured image (S505).

  When a face area is detected in S505, a hand is detected around the area. An area similar to the prepared hand template is scanned around the face area using the hand template stored in the hand template database (S506). The determination of the peripheral area of the face may be a pre-set size range based on the face position, or by scanning an area close to the face in the peripheral area of the face based on the principle of stereo vision using two cameras. The search range may be reduced. In order to calculate the similarity for matching, the candidate region image of the extracted hand and the hand template image are first projected onto the eigenspace using the prepared projection matrix to the eigenspace. The method of comparing the distance between the two is performed. By comparing the distances in the space representing the main component of the hand, it is possible to perform detection with reduced influence of noise such as background. In the search area, an area that satisfies a predetermined threshold and has a distance closest to the hand template is defined as a hand position. Further, the shape of the hand template having the closest distance (for example, goo, par, etc.) is set as the detected hand shape (S507).

  If there is no region that satisfies the threshold in the search region, the detection is terminated as if the hand has not been taken.

  In this example, the template matching method is used for detecting the hand position / shape, but other methods such as a boosting method may be used.

  Based on the hand position / shape information, the application control means 106 presents, for example, a plurality of options and hand positions / shapes corresponding to the options to the user, and changes the position / shape of the user's hand. Accordingly, it can be determined that the corresponding option has been selected. A method in which the user operates the video display device by changing the shape and position of the hand in this way is called an operation method using a freehand gesture.

(Embodiment 6)
FIG. 11 shows an operation method by the gesture input remote controller 105 in the present embodiment.

  As shown in FIG. 11A, the user indicates a predetermined hand movement (gesture) such as shaking or turning while holding the gesture input remote controller 105 in his hand, or a desired position of the video display device. Thus, the video display device can be operated.

  FIG. 11B shows the configuration of the gesture input remote controller 105. The remote control includes therein a motion sensor 1052 that detects the movement of the user's hand holding the remote control. Further, the remote controller may be provided with buttons on the surface thereof as shown in FIG.

  The motion sensor 1052 is configured by any one of an acceleration sensor, an angular acceleration sensor (rate gyro), a geomagnetic sensor (electronic compass), or a combination of two or more. The acceleration sensor detects acceleration with respect to a predetermined axis. For example, as shown in FIG. 11B, the acceleration sensor detects acceleration with respect to each of the three axes orthogonal to the X axis, the Y axis, and the Z axis. It is. When the user moves his / her wrist and / or arm while holding the gesture input remote control 105 with his / her hand, and the position and / or posture of the gesture input remote control is changed, the gesture identifying means 1053 moves the predetermined hand movement (gesture ) And the identification result is input from the gesture identification unit 1053 to the application control unit 106.

  Here, an example is shown in which an acceleration sensor is used as the position and / or orientation detection means of the gesture input remote controller 105. However, for the same purpose, an angular acceleration sensor (rate gyro) and a geomagnetic sensor (electronic compass) are used. be able to.

  Based on the identified gesture, the application control means 106 controls the application such as scrolling the screen in response to the gesture of “turn” as shown in FIG.

  In addition, by using a geomagnetic sensor as the motion sensor 1052, when the user points to a desired position of the video display device, the corresponding position on the screen of the video display device is calculated so that a person skilled in the art can easily guess. Then, the calculation result can be input to the application control means 106.

  Based on the input of the corresponding position on the screen of the video display device, the application control means 106 focuses, for example, as shown in FIG. 12B, or activates the application corresponding to the position. Control.

  In this way, the user operates the video display device by pointing to a predetermined hand movement (gesture) or a desired position of the video display device while the user holds the gesture input remote control 105 with the hand. This is called an operation method by the input remote controller 105.

  The shape of the gesture input remote controller 105 is not limited to the longitudinal shape of a planar rectangle as shown in FIG. 11, but the planar shape may be a longitudinal shape such as an ellipse, and the sectional shape is not limited to a rectangle, but may be a circle or other polygonal shape. There may be. Furthermore, the shape of the remote controller may be a sphere as shown in FIG.

  In the spherical gesture input remote control, one or more spherical gesture input remote controls as shown in FIG. 13 are (a-1) rolled on the restrained surface, (a-2) grasped in the hand and holding the wrist. It is possible to manipulate objects (GUI objects, etc.) on the screen according to gestures by turning or moving to the center, or (a-3) pushing or rotating the ball floating in the air with both hands or one hand. It is. These operations can also be executed in a scene where a plurality of people operate a GUI for the same purpose in cooperation. As the structure of the spherical gesture input remote controller shown in FIG. 13A, the structure shown in FIG. In addition to the spherical outer shape, motion sensors (acceleration sensor, angular acceleration sensor (rate gyro), geomagnetic sensor (electronic compass), etc.), pressure sensor that senses the hand force when grasping, and information from the device side Display mechanism (LED, etc.), a mechanism for bidirectionally communicating signals of a spherical gesture input remote controller such as a vibration motor to the receiving side, a drive mechanism, and a power source. As shown in FIG. 13 (b-2), the structure of a spherical gesture input remote control that floats in the air is filled with helium gas or the like in a transparent or translucent elastic resin. A sensor group, a rotation mechanism, a communication mechanism, a display mechanism, and the like as shown in FIG. Display mechanisms such as LEDs and projectors also serve as a mechanism for casually projecting the state from the system side as feedback.

  An example of an operation system using these spherical gesture input remote controls is shown in FIG. 14A. The user holds the spherical gesture input remote control in the hand and rotates or moves the wrist, thereby For example, a three-dimensional character (avatar) or an object is operated like a marionette. In this case, the character is defined by a hierarchical structure such as a skeletal structure, and the translational movement amount (three degrees of freedom of X, Y, and Z) and the rotation amount (three degrees of freedom of X, Y, and Z) of the tip are included in the character. Designate the locus of the tip by moving the position of the spherical gesture input remote control. The position of each joint excluding the tip can be obtained by calculation using a technique such as inverse kinematics. Thereby, for example, in an application that displays a virtual space simulating a safari park, an interaction is realized in which food is presented to a full-sized animal (for example, a giraffe) displayed in the safari virtual space. That is, when the user holds the spherical gesture input remote control in his / her hand and puts his / her hand forward a predetermined distance or more in front of the user, the food is close to the full-size giraffe displayed in the safari virtual space. When the user moves his / her hand so that the food moves near the giraffe's mouth and the food is moved within a predetermined vicinity from the giraffe's mouth, the giraffe eats the interaction. It is. The actual size display utilizing the large screen of such a video display device and natural interaction can create a sense of presence as if it were in the place (safari park). It is also possible to grab the GUI on the screen by lightly grasping the spherical gesture input remote control in the hand, and further move and transform the GUI by rotating and moving the ball in the hand.

  It is also possible for the user to select or move the bubble-type GUI on the screen by grasping or turning the spherical gesture input remote control floating in the air. In this case, it is a feature that the user can feel a sense of unity between the action of grasping and turning the actual ball and the action of the GUI on the screen by the gesture. Furthermore, as another example of the operation using the spherical gesture input remote controller, as shown in FIG. 14B, by rolling or throwing the spherical gesture input remote controller toward the video display device, for example, information input by the user ( (Such as voice mail) is transmitted to a predetermined destination. This makes it possible to perform intuitive gestures, such as rolling a ball, that were previously operated according to the GUI menu, and there are users such as elderly people and children who have had difficulty with conventional GUI complicated operations. The spherical gesture remote controller can be used for easy operation. Further, the GUI object or content on the screen may be moved so as to transmit / receive information (such as voice mail) input by the user by shaking or throwing the spherical gesture input remote controller. Alternatively, by using the spherical gesture input remote controller by physically bringing the spherical gesture input remote controllers close to each other, bringing them into contact with each other, or exchanging them, and realizing the fusion and exchange of attributes given in advance to the ball, Intuitive operation with the user's feeling is realized. In this case, as a GUI on the screen, a spherical GUI such as a metaball or a content can be clearly indicated by fusing or separating contents like nuclear fusion.

  The present invention is a video display device that is controlled based on user sensing information such as a user position / orientation, line of sight, hand gesture, user hand / arm movement, etc., as well as conventional button input. For example, it can be used as a TV set in a shared space of a family such as a living room.

The figure explaining an example of the interface of the external appearance of a video display apparatus, and related apparatus The figure which shows an example of the condition where several users are located in the room | chamber interior which has an image | video display apparatus. Main function block diagram of video display device The figure explaining the outline of a data structure of user information DB Flow chart showing outline of processing of video display device The figure explaining the outline of the operation method by a user position, and its realization method The figure explaining the operation example of the operation method by a user position The figure explaining the outline of the operation method by gaze and face direction and the realization method The figure explaining the outline of the operation method by gaze and face direction and the realization method The figure explaining the outline of the operation method by freehand gesture and its realization method The figure explaining the outline of the operation method by the gesture input remote control and its realization method The figure explaining the operation example of operation by gesture input remote control The figure explaining the outline of the operation method by the spherical gesture input remote control and its realization method The figure explaining the example of operation of operation by a spherical gesture input remote control

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 User detection camera 101 User position detection means 102 User identification means 103 Line-of-sight detection means 104 Hand position / shape detection means 105 Gesture input remote control 106 Application control means 107A User information DB
107B User state management means 107C Modal control means 107D Viewing state determination means 108 Data reception means 109 Life size display conversion means 110 Screen drawing means 111 Screen 112 Life size information addition means 113 Data transmission means

Claims (5)

A video display device operated by one or more users,
Identifying means for identifying one or more users;
Detection means for detecting the position and movement of one or more users;
Display means for displaying video;
A discriminating means for discriminating one or more users who are viewing targets of the displayed video based on the detected position and operation of the detected user among the one or more identified users;
Control means for controlling an application for displaying video ;
A user database that stores user attribute information is provided.
The attribute information includes body feature information including at least body information, and human relationship information between one or more users,
The video display apparatus, wherein the control unit controls an application for displaying the video corresponding to the attribute information of one or more viewing target users. The human relationship information between the one or more users is information on intimacy between users,
The video display apparatus according to claim 1, wherein the control unit determines the degree of information sharing according to the degree of closeness between the users and controls an application that displays the video. The video display device further includes:
Control the application that works in cooperation with the other party's video display device installed in another location and connected to the network,
Data receiving means for receiving operation information transmitted from the counterpart video display device;
Of the operation information, a life-size display that performs scale conversion based on the presentation screen size for information on an object that is to be presented in a full-size image captured by the counterpart video display device and operation information for the object Conversion means;
Life-size information adding means for adding life-size information necessary for life-size display on the counterpart video display device to operation information based on user size information and user position information;
Data transmission means for transmitting operation information and life-size video to which the life-size information is added to the counterpart video display device;
Wherein the control unit, together with the input information, processes the operation information after the scale conversion, and controlling the application, the image display apparatus according to claim 1. The video display device further includes a remote control for obtaining explicit input information of the user,
The video display apparatus according to claim 3 , wherein the control unit controls an application based on a plurality of relative positional relationships of the remote controller . The remote control is spherical, and the user can throw the remote control or roll on the surface,
5. The video display apparatus according to claim 4 , wherein the control unit controls an application for displaying the video based on information on a trajectory of the remote controller or information on a stop position .