JP2014191629A - Display device and screen control method in the same device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow a user to perform seamless operation that does not interfere stereoscopic vision in 3-D display, and requires only visual line input and eliminates the need of auxiliary operation.SOLUTION: A display device (portable information terminal 10) includes a display unit 13 that performs 3-D screen display, and a visual line measurement unit 12 that measures a visual line position to a screen of the display unit 13, and further includes a control unit 11 that executes applications. The control unit 11 detects the movement of a visual line with respect to a depth direction of the screen on the basis of information output from the visual line measurement unit 12, and displays a related screen overlapped on the display unit 13 to detect input to the applications.

Description

  The present invention relates to a display device that performs 3D video display and a screen control method in the device.

  With the increase in functionality of mobile information terminals and the diversification of applications, many icons have been placed on the screen. Since the number of icons has increased, screen switching methods such as page turning for managing the display have become common.

  In recent years, 3D liquid crystal has been mounted on portable information terminals, and it has become possible to display a large number of icons in a three-dimensional space. The 3D liquid crystal is a display device capable of stereoscopic display without requiring dedicated glasses by controlling the traveling direction of light so that different images can be seen by the left and right eyes. For example, by adding the right-eye image and the left-eye image and displaying the image on a TFT (Thin Film Transistor) liquid crystal, an image corresponding to each eye can be seen.

  By the way, in addition to a mouse and keyboard frequently used as a man-machine interface, or a touch panel which is currently mainstream, a line-of-sight input interface that can be operated using a user's line of sight is known. According to the line-of-sight input, it is possible to determine which position on the screen the user is viewing from the line-of-sight position of the user. For example, Patent Document 1 discloses a technique for determining a screen based on line-of-sight information. However, the technique disclosed in Patent Document 1 is an operation method on a 2D screen and cannot be applied on a 3D screen.

JP-A-9-204287

  As described above, a portable information terminal requires a method for efficiently operating a large number of icons displayed in a three-dimensional space by mounting a 3D liquid crystal. However, a touch panel leads to hiding the 3D liquid crystal screen, resulting in a stereoscopic view. There was a problem of inhibiting. On the other hand, if the line-of-sight input does not hide the screen, the operation can be performed on the 3D screen without obstructing the stereoscopic view.

  However, there is no means for transmitting the user's decision making such as a mouse click operation for line-of-sight input. For this reason, an auxiliary operation using a button, a method for fixing the line-of-sight position for a certain time, or a method using wink or the like It substitutes with. Accordingly, there is a demand for the appearance of a seamless operation method that eliminates the need for an auxiliary operation only by line-of-sight input without obstructing stereoscopic vision even in the case of 3D display.

  The present invention has been made in order to solve the above-described problem, and even in the case of 3D display, a seamless operation that does not require an auxiliary operation with only a line-of-sight input without obstructing stereoscopic vision is possible. An object is to provide a device and a screen control method in the device.

In order to solve the above-described problem, a display device according to an aspect of the present invention is a display device including a display unit that performs 3D screen display, and a line-of-sight measurement unit that measures a line-of-sight position with respect to the screen of the display unit. A control unit that executes an application, and the control unit detects line-of-sight movement in the depth direction of the screen based on information output from the line-of-sight measurement unit, and superimposes a related screen on the display unit. Displaying and detecting an input to the application.

  In the display device according to another aspect of the present invention, the control unit determines detection of an input to the application when detecting a movement of a threshold value or more in the depth direction.

  In the display device according to another aspect of the present invention, the control unit changes and controls the focal length by adjusting the distance between the left-eye image and the right-eye image for each user based on information registered in advance. To do.

  In the display device according to another aspect of the present invention, when the line of sight is in the foreground screen, the control unit uses a background image in the background of the display unit as a blurred background image at a position where nothing is displayed in the foreground. When the line of sight is moved in the depth direction, the front screen is made transparent, and the control is changed from normal to blurred display to the normal display.

  In the display device according to another aspect of the present invention, the control unit displays a guide video focused on an arbitrary position on the back screen when moving the line of sight to the back screen displayed on the display unit. It is characterized by displaying.

  In the display device according to another aspect of the present invention, the control unit recognizes a cancel operation when the viewpoint is moved to an arbitrary position where nothing is displayed on the front screen.

  In the display device according to another aspect of the present invention, the control unit performs enlargement / reduction display by following the content displayed on the screen of the display unit as the viewpoint moves in the depth direction. .

  In addition, an image control method according to an aspect of the present invention includes a display device that includes a display unit that performs 3D screen display, a line-of-sight measurement unit that measures a line-of-sight position of the display unit with respect to the screen, and a control unit that executes an application. A screen control method for detecting an input to the application, the step of detecting line-of-sight movement in the depth direction of the screen based on information output from the line-of-sight measurement unit, and a screen related to the line-of-sight movement And a step of detecting an input to the application by superimposing the display on the display unit.

  According to the display device and the screen control method in the device according to one embodiment of the present invention, even in the case of 3D display, it is possible to perform a seamless operation that does not require an auxiliary operation only by a line-of-sight input without obstructing stereoscopic vision. .

It is a block diagram which shows the structure of the portable information terminal which concerns on embodiment of this invention. It is a figure which shows the screen operation image of the portable information terminal which concerns on embodiment of this invention. It is a flowchart which shows the flow of the determination process by the focal distance change of the viewpoint in the portable information terminal which concerns on embodiment of this invention. It is a figure which shows the 3D display image of the portable information terminal which concerns on embodiment of this invention. It is a figure which shows another screen operation image of the portable information terminal which concerns on embodiment of this invention. FIG. 6 is a diagram illustrating a screen operation image of the portable information terminal following FIG. 5. FIG. 7 is a diagram illustrating a screen operation image of the portable information terminal following FIG. 6. FIG. 8 is a diagram illustrating a screen operation image of the portable information terminal following FIG. 7.

  Hereinafter, an embodiment for carrying out the present invention by grasping the attached drawings (hereinafter simply referred to as the present embodiment) will be described in detail.

(Configuration of the embodiment)
FIG. 1 is a block diagram showing a configuration of a portable information terminal 10 as a display device according to the present embodiment. As shown in FIG. 1, a portable information terminal 10 according to the present embodiment includes a control unit 11, a line-of-sight measurement unit 12, and a display unit 13, all of which have a plurality of lines for address, data, and control. It is connected to a system bus 14 composed of books.

  The control unit 11 includes, for example, a microprocessor including a memory, and executes, for example, an application by sequentially reading and executing a program recorded in the memory. In addition, the control unit 11 detects line-of-sight movement in the depth direction of the screen based on the information output from the line-of-sight measurement unit 12, superimposes the related screen on the display unit 13, and superimposes it on the application. Has a function to detect input.

  When the control unit 11 detects a movement in the depth direction that is greater than or equal to the threshold value, the control unit 11 determines the detection of the input to the application. The control unit 11 can change and control the focal length by adjusting the distance between the left-eye video and the right-eye video of the user based on information registered in advance for each user. In addition, when the user's line of sight is on the front screen in the screen configuration superimposed on the display unit 13, the control unit 11 displays a blurred background image on the back screen of the display unit 13 and displays nothing in front. When the line of sight is moved in the depth direction at a position where the screen is not placed, the screen on the near side may be made transparent and the display on the back may be changed from the blurred state to the normal display.

  When the control unit 11 moves the line of sight to the back screen displayed on the display unit 13, the control unit 11 may display a guide video focused on an arbitrary position on the back screen. Further, the control unit 11 may recognize the cancel operation when the line of sight is moved in the depth direction at an arbitrary position where nothing is displayed on the front screen. Further, the control unit 11 may perform enlargement / reduction display by following the content displayed on the screen of the display unit 13 as the viewpoint moves in the depth direction.

  The line-of-sight measurement unit 12 is constituted by, for example, an infrared camera, tracks the movement angle (rotation, angle) of the eyeball by irradiating the eyeball with infrared rays and detects the reflected light, and cooperates with the control unit 11. It has a function of identifying the area that the user is watching.

  The display unit 13 is configured by, for example, 3D liquid crystal. The 3D liquid crystal performs stereoscopic display using a parallax barrier, and is configured by, for example, a combination of a TFT liquid crystal display and a switch liquid crystal. By controlling the switch liquid crystal, an optical parallax barrier is generated, the light is separated so that different light can reach the left and right eyes, and an image obtained by adding the right eye image and the left eye image is used as a 3D display image. To display. Thus, the user can see an image corresponding to each eye.

(Operation of the embodiment)
Hereinafter, the operation of the portable information terminal 10 according to the present embodiment shown in FIG. 1 will be described in detail with reference to FIGS.

  First, the screen operation image shown in FIG. 2 will be described. As a screen control method by line-of-sight input on a 3D screen, first, some coordinates are determined at specific positions in the depth direction. In the example shown in FIG. 2, there are two locations A and B. Here, it is assumed that the screen when the user performs browsing or the like is displayed at “screen position B”. When a user performs browsing, a determination operation such as “OK” or “NG” or a link is necessary. However, as a feature of line-of-sight input, the determination is made by the user consciously moving the viewpoint in the depth direction. There is a way to do it.

  Specifically, when the user is looking at the button A displayed at the screen position A, the viewpoint is within the display frame range (XY coordinates) of the button A, and the focal length (Z-axis) of the screen position A Coordinates). Since this viewpoint can move and change only the Z axis within the XY axis coordinate frame (change the focal length), it exceeds the threshold in the Z axis direction (movement distance in the Z axis direction). At this point, it is considered that the button A has been pressed. Then, by displaying the operation screen after button determination at the screen position B, it becomes unnecessary to return the viewpoint distance (Z-axis) to the screen position A again, so that the user can perform a smooth and seamless screen operation.

  In the case of a link instead of a button, the same processing can be performed by setting a certain frame so as to surround the link display. FIG. 3 is a flowchart showing a flow of determination processing based on the focal length of the starting point by the control unit 11.

  In FIG. 3, the control unit 11 first determines whether the user is looking inside the button or the link frame based on information output from the line-of-sight measurement unit 12 (step S <b> 101). If the viewpoint is within the display frame range (XY coordinates) of the button or the display frame range (XY coordinates) of the link (“Yes” in step S101), and the movement of the line of sight in the Z-axis direction by the user is detected (Step S102 “Yes”) Since this viewpoint can move and change only the Z-axis (change the focal length) within the XY-axis coordinate frame, the control unit 11 performs the adjustment in the Z-axis direction. It is determined whether or not the movement is greater than or equal to a threshold value (movement distance in the Z-axis direction) (step S103). When the threshold value is exceeded (Step S103 “Yes”), the control unit 11 confirms the movement of the line of sight in the button frame (Step S104 “Yes”) and the Z axis direction that is the determination direction (Step S103 “Yes”). S105 “Yes”), it is recognized that the button or link has been determined (step S106).

  Subsequently, the control unit 11 displays the screen after the determination on the display unit 13 (step S107), returns to the button or link frame determination process in step S101, and repeats the determination process by changing the focal length of the viewpoint.

  The viewpoint is not within the button display frame range (XY coordinates) or the link display frame range (XY coordinates) (step S101 “No”), and the line of sight is not moved in the Z-axis direction (step S102 “No”). "), Not moving beyond the threshold value (step S103" No "), not within the button frame (step S104" No "), not moving in the Z-axis direction as the determination direction (step S105" No ") In this case also, the process returns to the button or link frame determination process in step S101, and the determination process by changing the focal length of the viewpoint is repeated.

  FIG. 4 shows a 3D display image of the portable information terminal 10 according to the present embodiment. Usually, when a user performs stereoscopic viewing, there is a distance (z-axis coordinate) at which stereoscopic viewing can be comfortably performed for each user. In addition, 3D video is realized by, for example, transmitting stereoscopically different images to the left and right eyes as shown in FIG. 4, and the distance in the Z-axis direction of the stereoscopic video is determined by the left and right eyes. It is determined by the distance A between the images to be sent.

In other words, in order to display a stereoscopic image at a distance, when the stereoscopic image is displayed in front of the physical screen of the display unit 13 (that is, when the stereoscopic image appears to pop out as shown in FIG. 4), the distance A When the stereoscopic image is displayed at the back of the physical screen of the display unit 13 (that is, when the stereoscopic image is visible at the back), the distance A is increased. At the same time, it becomes possible by reducing the image size. Therefore, by adjusting only the distance A without changing the size of the video, it is possible to change only the focal length without changing the size of the stereoscopic video recognized by the user. By using this, while the user is browsing, by adjusting the distance A little by little to a distance that the user can comfortably, it is possible to move the image display to an easy focal length without giving the user a sense of incongruity Become.

  By applying this to the screen after determination based on the focal length, the image displayed at the screen position B is moved to a comfortable vertex distance (tentatively the screen position A) so that the user does not feel uncomfortable. In addition, when browsing for a long time at the screen position A, such as Web browsing or 2D video viewing, it is possible to stretch the eyes by gradually changing the distance A, thereby preventing eye fatigue. It becomes possible. The determination operation by moving the line of sight in the Z-axis direction can be performed in either the back direction or the near direction of the Z axis, and the screen position C can be displayed by moving the focal length in front of the screen position A. Is possible.

  At this time, in order to display in an easy-to-understand manner in which direction the user moves the focus in the Z-axis, the button image may be made translucent and a guide image focused on the screen position B may be displayed. .

(Effect of embodiment)
As described above, according to the portable information terminal 10 according to the present embodiment, the control unit 11 detects line-of-sight movement in the depth direction of the screen based on information output from the line-of-sight measurement unit 12, and displays related screens. An input to the application is detected by overlaying on the display unit 13. Therefore, even in the case of 3D display, it is possible to realize a seamless operation that does not require an auxiliary operation only by a line-of-sight input without obstructing stereoscopic vision. That is, in line-of-sight input, it is determined which position on the screen the user is looking at from the line-of-sight position of the user. Unlike an input device such as a touch panel or a mouse, not only the XY axis (plane) but also the Z axis (depth) These coordinates can also be discriminated from the focal position. Therefore, when the user intentionally moves the viewpoint in the depth direction as a means for displaying the user's intention, an operation with only a line-of-sight input that does not require an auxiliary operation becomes possible.

  For example, it is assumed that data of a specific icon A arranged in a three-dimensional space is displayed, and the user is in a state where the viewpoint is aligned with the icon A. When the user edits the data of the icon A with an editor or the like, the editor screen is opened with an intention display such as a double click with a mouse or the like. However, in the portable information terminal 10 according to the present embodiment, the user fixes the line of sight to the icon A. By shifting the line of sight in the Z-axis direction from the state in which the XY axes are fixed, an intention display corresponding to double-clicking is performed, and a screen related to the Z-axis coordinate position shifted by the user is displayed. Specifically, it is possible to display a selection screen such as an editor or directly open the editor. If this method is used, it becomes possible to easily call different functions only by eye-gaze input.

In the portable information terminal 10 according to the present embodiment, when the viewpoint is shifted, the control unit 11 displays the screen on the display unit 13 in a superimposed manner (superimpose). . In this case, if you want to return from the front screen to the back screen, it may be difficult to move the viewpoint if the back screen is hidden by the front screen. The unit 11 displays the front screen as a background image and displays it as a screen behind the focus. When the position of the viewpoint is shifted in the Z-axis direction of the back screen at the position where the icon is displayed on the front screen, the front screen is made transparent and the back screen is changed from blur display to normal display. . Also, if the viewpoint is shifted in the Z-axis direction of the back screen at a position where nothing is displayed on the front screen, it is recognized as a cancel operation.

  Here, a more specific screen operation image is illustrated with reference to FIGS. First, as shown in FIG. 5, it is assumed that screens A, B, and C overlap from the front to the back, and the user aligns the viewpoint with the Z-axis position of screen B. At that time, the control unit 11 makes the screen A transparent (or blurs with semi-transparency), and the screen C is displayed as a background image of the screen B on the display unit 13 as an image that is out of focus. Next, when the user moves the viewpoint focusing on the position of the screen A from the state where the viewpoint is focused on the specific text file (file b) displayed on the screen B, the control unit 11 displays the display unit. 13 displays the screen A and displays an icon (button) such as an editor of the screen A. Here, in addition to the editor icon, for example, a deletion icon for deleting the file, a browsing icon for simply browsing the file, and the like are displayed.

  Then, when the user aligns the viewpoint with the editor icon on the screen A (FIG. 6) and shifts the viewpoint to the position of the screen B, the control unit 11 displays a selection screen of “YES or No” on the screen B of the display unit 13. Display (FIG. 7). At this time, instead of the screen B, the selection screen may be displayed at a position in front of the screen A, or the selection screen may be displayed on the screen C. In the selection screen shown in FIG. 7, when the user selects YES, that is, when the viewpoint is set to “YES” on the selection screen of screen B and the line of sight is shifted as it is, the control unit 11 displays the screen of the display unit 13. In A, an editor screen for editing the file b is displayed. In FIG. 6, when returning to the screen B without starting an editor or the like on the screen A, the focus is returned to the position of the screen B at a position where no icon is displayed on the screen A.

  By the way, it is known that eye fatigue is likely to occur due to stereoscopic viewing, and points (depth) at which it is difficult to feel fatigue while browsing a 3D screen for each user are known. When the user browses text and image information in the 3D space, eye fatigue can be reduced by adjusting the depth coordinate (Z coordinate) displayed for each user. This can be realized by the control unit 11 changing and controlling the focal length by adjusting the distance between the left-eye video and the right-eye video for each user based on information registered in advance for each user.

  Therefore, a user who is browsing text and screen information while inputting his / her line of sight moves his / her viewpoint to a point (Z coordinate) where stereoscopic viewing is easy, and follows the Z coordinate to follow the text and image information. By doing so, stereoscopic viewing at a point with less user fatigue becomes possible. When moving characters and image information in the Z-axis direction, enlargement / reduction display is performed to prevent the user from feeling uncomfortable so that the size of the character does not change for the user. That is, for example, when the balloon image is moved from the front to the back, the image is inflated so that the size visible to the user does not change.

  Contrary to the above-mentioned conscious viewpoint movement of the user, the image information is gradually moved in the Z-axis direction so that the apparent size does not change when a specific image is viewed for a long time in a stereoscopic view. By moving back and forth, it becomes possible to shift the viewpoint without giving the user a sense of incongruity, thus preventing eye fatigue. Further, when the viewpoint is moved in the Z-axis direction, the control unit 11 performs an enlargement / reduction process on the image, so that operations such as pinch-in and pinch-out can be performed by moving the viewpoint.

  In addition, although only the portable information terminal 10 was illustrated as a display apparatus which concerns on this embodiment, if it has the display part 13 which can display 3D, such as a game machine, a smart phone, a tablet, PC, viewpoint measurement, such as an infrared camera It can be applied to any electronic device simply by adding the unit 12.

As mentioned above, although preferred embodiment of this invention was explained in full detail, it cannot be overemphasized that the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above embodiment. Further, it is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  DESCRIPTION OF SYMBOLS 10 ... Display apparatus (portable information terminal), 11 ... Control part, 12 ... Gaze measurement part, 13 ... Display part, 14 ... System bus

Claims (8)

A display device comprising: a display unit that performs 3D screen display; and a line-of-sight measurement unit that measures a line-of-sight position with respect to the screen of the display unit,
A control unit for executing the application;
The controller is
A display that detects gaze movement in the depth direction of the screen based on information output from the line-of-sight measurement unit and detects an input to the application by overlaying related screens on the display unit. apparatus. The controller is
The display device according to claim 1, wherein detection of an input to the application is determined when movement of a threshold value or more in the depth direction is detected. The controller is
3. The display device according to claim 1, wherein the focal length is changed and controlled by adjusting a distance between the left-eye image and the right-eye image for each user based on information registered in advance. The controller is
When the line of sight is on the front screen, the back screen of the display unit is a blurred background image, and when the line of sight is moved in the depth direction, the front screen becomes transparent and the normal screen is displayed from the blurred state. The display device according to claim 1, wherein the change control is performed. The controller is
5. The display device according to claim 4, wherein when a line of sight is moved to a back screen displayed on the display unit, a guide video focused on an arbitrary position on the back screen is displayed. The controller is
The display device according to claim 1, wherein when the line of sight moves in the depth direction at an arbitrary position where nothing is displayed on the front screen, the display device is recognized as a cancel operation. The controller is
The display device according to claim 1, wherein enlargement / reduction display is performed by following the content displayed on the screen of the display unit as the viewpoint moves in the depth direction. In a display device including a display unit that performs 3D screen display, a line-of-sight measurement unit that measures a line-of-sight position with respect to the screen of the display unit, and a control unit that executes an application, There,
Detecting line-of-sight movement in the depth direction of the screen based on information output from the line-of-sight measurement unit;
Detecting an input to the application by overlaying a screen related to the line-of-sight movement on the display unit;
A screen control method for a display device, comprising: