JP2022187199A - Image display system and image display method - Google Patents

Abstract

To provide an image display system capable of easily estimating the sight line of a user, for realizing foveated rendering.SOLUTION: An image display system 2 comprises: an attachment part 8 which is attached to the head of a user 12; a display part 10 which is arranged at the attachment part 8, and displays an image 14 to the user 12; a sensor 34 which detects movement of the attachment part 8; a sensor 18 which detects the movement of a controller 6; and a processor 46 which controls display of the image 14, and draws the image 14 including a high resolution area 54, and a medium resolution area 56 in which the resolution is lower than that of the high resolution area 54, and moves a pointer 16 indicating a position on the image 14, on the basis of the detection result of the sensor 18. The processor 46 estimates the sight line of the user 12 on the basis of each detection result of the sensors 18, 34, and controls the position of the high resolution area 54 so that the estimated sight line of the user 12 overlaps with the high resolution area 54.SELECTED DRAWING: Figure 7

Description

The present disclosure relates to an image display system and an image display method.

A head-mounted image display device called a head-mounted display (HMD) is known (see Patent Document 1, for example). This type of image display device has a mounting section and a display section. The wearing part has, for example, a spectacle shape and is worn on the user's head. The display unit is arranged on the wearing unit and displays a virtual reality (VR) image to the user, for example.

In the image display device described above, a technique called foveated rendering is used in order to greatly reduce the load of image rendering processing. Foveated rendering renders images at high resolution in the central vision area (also called the foveal area) and at low resolution in the peripheral vision area. In order to realize this foveated rendering, the mounted portion of the image display device is equipped with a dedicated device for detecting the line of sight of the user.

WO2018/122902

The present disclosure provides an image display system and image display method capable of estimating a user's line of sight in a simple manner in order to achieve foveated rendering.

An image display system according to the present disclosure includes a mounting unit that is worn on the head of a user, a display unit that is arranged on the mounting unit and displays an image to the user, a first sensor that detects movement of the mounting unit, an operation a second sensor for detecting body movement; a processor for controlling display of an image; and a processor that draws an image containing a region of and moves a pointer that indicates a position on the image based on the detection result of the second sensor, the processor is provided with the first sensor and the second sensor The user's line of sight is estimated based on the respective detection results, and the position of the first area is controlled so that the estimated user's line of sight is superimposed on the first area.

According to the image display system and the like according to the present disclosure, it is possible to estimate the line of sight of the user by a simple method in order to realize foveated rendering.

1 is a diagram showing an overview of an image display system according to an embodiment; FIG. 1 is a block diagram showing the configuration of an image display system according to an embodiment; FIG. 3 is a block diagram showing the configuration of a controller according to the embodiment; FIG. 1 is a block diagram showing the configuration of a head mounted display according to an embodiment; FIG. 4 is a flow chart showing the flow of operations of the image display system according to the embodiment. 6 is a diagram for explaining step S103 in the flowchart of FIG. 5; FIG. 6 is a diagram for explaining step S107 in the flowchart of FIG. 5; FIG. 6 is a diagram for explaining step S107 in the flowchart of FIG. 5; FIG. 6 is a diagram for explaining step S107 in the flowchart of FIG. 5; FIG. 6 is a diagram for explaining step S110 in the flowchart of FIG. 5; FIG. 6 is a diagram for explaining step S110 in the flowchart of FIG. 5; FIG. 6 is a diagram for explaining step S110 in the flowchart of FIG. 5; FIG. 6 is a diagram for explaining step S110 in the flowchart of FIG. 5; FIG. FIG. 11 is a block diagram showing the configuration of an image display system according to a modified example of the embodiment; It is a block diagram which shows the structure of the head mounted display based on the modification of embodiment.

Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially the same configurations may be omitted. This is to avoid unnecessary verbosity in the following description and to facilitate understanding by those skilled in the art.

It is noted that the inventors provide the accompanying drawings and the following description for a full understanding of the present disclosure by those skilled in the art and are not intended to limit the claimed subject matter thereby. No.

(Embodiment)
Embodiments will be described below with reference to FIGS. 1 to 15. FIG.

[1. Overview of Image Display System]
First, an overview of an image display system 2 according to an embodiment will be described with reference to FIG. FIG. 1 is a diagram showing an overview of an image display system 2 according to an embodiment. FIG. 2 is a block diagram showing the configuration of the image display system 2 according to the embodiment.

In FIG. 1, the horizontal direction is the X-axis direction, the front-rear direction is the Y-axis direction, and the vertical direction is the Z-axis direction.

As shown in FIGS. 1 and 2, an image display system 2 according to the embodiment includes a head-mounted display (hereinafter sometimes referred to as "HMD") 4 and a controller 6 (an example of an operating body). . The image display system 2 is a stand-alone image display system that does not use equipment other than the head-mounted display 4 and the controller 6 .

The head-mounted display 4 is, for example, an eyeglass-type or goggle-type wearable terminal. The head mounted display 4 has a mounting section 8 and a display section 10 . The mounting unit 8 is a housing mounted on the head of the user 12 so as to cover both eyes of the user 12 . The display unit 10 is arranged on the mounting unit 8 and positioned so as to face both eyes of the user 12 .

The display unit 10 is, for example, a liquid crystal display, and displays images to the user 12 . The display unit 10 displays, for example, an image 14 as shown in (b) of FIG. 6, which will be described later. In the example shown in FIG. 6B, the image 14 is a three-dimensional virtual reality (VR) image showing a landscape, and includes an object 13 representing a tree, an object 15 representing a hut, and an object 17 representing a stump. contains. When the user 12 moves the head on which the mounting section 8 is worn, the angle of the image 14 changes following the movement of the mounting section 8 . A pointer 16 pointing to a position on the image 14 is superimposed on the image 14 . The pointer 16 is used, for example, when performing an operation of selecting an icon or the like displayed on the image 14 . In addition, in FIG. 6(b), the pointer 16 is illustrated by a black circle.

The controller 6 is a controller for operating the display of the image 14 displayed on the display unit 10 and is connected to the head mounted display 4 so as to be capable of wireless communication. The controller 6 is arranged separately from the mounting portion 8 of the head-mounted display 4, and is held and operated by the user 12 by hand. For example, when the user 12 moves the hand holding the controller 6 , the pointer 16 follows the movement of the controller 6 and moves on the image 14 .

[2. Controller configuration]
The configuration of the controller 6 according to the embodiment will be described with reference to FIG. FIG. 3 is a block diagram showing the configuration of the controller 6 according to the embodiment.

As shown in FIG. 3, the controller 6 includes a sensor 18 (an example of a second sensor), a communication unit 20, haptics 22, a memory 24, an input unit 26, a processor 28, a battery 30, a USB (Universal Serial Bus) connector 32 .

The sensor 18 is, for example, an IMU (Inertial Measurement Unit) sensor, and includes an acceleration sensor in three axial directions of the X, Y and Z axes and a gyro sensor around the three axes of the X, Y and Z axes. The sensor 18 detects movements of the controller 6 in 6 DoF (Degree of Freedom). 6DoF is the degrees of freedom of movement, including movement in three axial directions, X, Y, and Z, and rotation (pitch, roll, and yaw) about three axes, X, Y, and Z. . The sensor 18 outputs a detection signal (an example of a detection result) indicating the detected movement of the controller 6 to the processor 28 .

In this embodiment, the sensor 18 includes an acceleration sensor and a gyro sensor, but is not limited to this, and may include various sensors such as a pressure sensor and a magnetic sensor.

The communication unit 20 is a communication adapter for wirelessly communicating with the head mounted display 4 by Bluetooth (registered trademark), for example. The communication unit 20 transmits detection signals from the sensor 18 to the head mounted display 4 according to instructions from the processor 28 .

The haptics 22 are actuators that give a tactile sensation to the hand of the user 12 holding the controller 6 .

Memory 24 stores a program for controlling controller 6 .

The input unit 26 is, for example, a button or the like arranged on the controller 6 and receives an operation by the user 12 . The input unit 26 outputs to the processor 28 an input signal indicating the received operation of the user 12 .

The processor 28 reads the program stored in the memory 24 and controls the controller 6 by executing the read program. Specifically, the processor 28 instructs the communication unit 20 to transmit the detection signal of the sensor 18 to the head mounted display 4 .

The battery 30 is, for example, a rechargeable secondary battery, and supplies power to each part of the controller 6 .

The USB connector 32 is a connector that can be connected to various devices. For example, when a power supply device is connected to the USB connector 32, power from the power supply device is supplied to the battery 30 via the USB connector 32, and the battery 30 is charged.

[3. Configuration of head-mounted display]
The configuration of the head mounted display 4 according to the embodiment will be described with reference to FIG. FIG. 4 is a block diagram showing the configuration of the head mounted display 4 according to the embodiment.

As shown in FIG. 4, the head mounted display 4 includes a display unit 10, a sensor 34 (an example of a first sensor), a communication unit 36, a haptic 38, a memory 40, a camera 42, an input unit 44 , a processor 46 , a battery 48 and a USB connector 50 . Since the display unit 10 has already been described, a description thereof will be omitted here.

The sensor 34 is, for example, an IMU sensor, and includes an acceleration sensor in the X-, Y-, and Z-axes, and a gyro sensor around the X-, Y-, and Z-axes. The sensor 34 detects 6 DoF movements of the mounting portion 8 (see FIG. 1) of the head mounted display 4 . 6DoF is the degrees of freedom of movement, including movement in three axial directions, X, Y, and Z, and rotation (pitch, roll, and yaw) about three axes, X, Y, and Z. . The sensor 34 outputs a detection signal (an example of a detection result) indicating the detected movement of the mounting portion 8 of the head mounted display 4 to the processor 46 .

In this embodiment, the sensor 34 includes an acceleration sensor and a gyro sensor, but is not limited to this, and may include various sensors such as a pressure sensor and a magnetic sensor.

The communication unit 36 is a communication adapter for wirelessly communicating with the controller 6 by Bluetooth (registered trademark), for example. The communication unit 36 receives the detection signal of the sensor 18 transmitted from the controller 6 and outputs the received detection signal of the sensor 18 to the processor 46 .

The haptics 38 are actuators for giving a tactile sensation to the head of the user 12 wearing the wearing section 8 .

Memory 40 stores a program for controlling head mounted display 4 .

The camera 42 is arranged on the mounting portion 8 of the head mounted display 4, and photographs the front of the user 12, for example. In addition, the image captured by the camera 42 may be displayed on the display unit 10 . Also, by analyzing the video captured by the camera 42 with the processor 46, it can be converted to 6DoF. In this case, camera 42 may be replaced with sensor 34 .

The input unit 44 is, for example, a button or the like arranged on the mounting unit 8 of the head-mounted display 4 , and receives operations from the user 12 . The input unit 44 outputs to the processor 46 an input signal indicating the received operation of the user 12 .

The processor 46 reads the program stored in the memory 40 and controls the head mounted display 4 by executing the read program. Specifically, the processor 46 has an image processing section 52 for controlling display of the image 14 on the display section 10 .

The image processing unit 52 includes a high-resolution area 54 (an example of a first area) and a high-resolution area 54 arranged around the high-resolution area 54 and having a higher resolution than the high-resolution area 54, as shown in FIG. and a low resolution area 58 (an example of a second area) arranged around the intermediate resolution area 56 and having a resolution lower than that of the intermediate resolution area 56 (an example of a second area). Render the containing image 14 . In this embodiment, the high-resolution area 54 is a circular area, and the medium-resolution area 56 is an annular area having the same center as the high-resolution area 54 . In FIG. 6B, the high resolution area 54 is the dark gray area, the medium resolution area 56 is the light gray area, and the low resolution area 58 is the white area. At this time, it is preferable that the area of the high-resolution area 54 is made slightly wider in response to fine movements of the user's 12 line of sight. When the resolution of the high-resolution area 54 is 100%, the resolution of the medium-resolution area 56 is, for example, 60%, and the resolution of the low-resolution area is, for example, 20%.

The image processing unit 52 also changes the angle of the image 14 based on the detection signal of the sensor 34 to follow the movement of the mounting unit 8 of the head mounted display 4 . Further, the image processing unit 52 moves the pointer 16 on the image 14 by following the movement of the controller 6 based on the detection signal of the sensor 18 transmitted from the controller 6 .

Further, the image processing unit 52 estimates the line of sight of the user 12 based on the detection signal of the sensor 34 and the detection signal of the sensor 18 of the controller 6 , and performs a high resolution image so that the estimated line of sight of the user 12 is superimposed on the high resolution area 54 . Controls the position of the resolution area 54 . This provides foveated rendering in which the image 14 is rendered at high resolution in the central vision area and at low resolution in the peripheral vision area.

The battery 48 is, for example, a rechargeable secondary battery, and supplies power to each part of the head mounted display 4 .

The USB connector 50 is a connector that can be connected to various devices. For example, when a power supply device is connected to the USB connector 50, power from the power supply device is supplied to the battery 48 via the USB connector 50, and the battery 48 is charged.

[4. Operation of Image Display System]
The operation of the image display system 2 according to the embodiment will be described with reference to FIGS. 5 to 13. FIG. FIG. 5 is a flow chart showing the operation flow of the image display system 2 according to the embodiment. FIG. 6 is a diagram for explaining step S103 in the flowchart of FIG. 7 to 9 are diagrams for explaining step S107 in the flowchart of FIG. 10 to 13 are diagrams for explaining step S110 in the flowchart of FIG.

6 to 13 are top views showing the positional relationship between the head mounted display 4 and the controller 6. FIG. Each (b) of FIGS. 6 to 13 shows the display of the head mounted display 4 when the positional relationship between the head mounted display 4 and the controller 6 is the same as (a) of each of FIGS. 6 to 13. 1 is an example of an image 14 displayed on the unit 10; In addition, in (b) of each of FIGS. 6 to 13, the intersection point of the dashed cross lines represents the center of the image 14 (that is, the center of the display surface of the display unit 10).

First, the user 12 wears the mounting portion 8 of the head-mounted display 4 on the head and holds the controller 6 by hand. In this state, as shown in FIG. 5, calibration is first performed in the head mounted display 4 (S101), and then calibration is performed in the controller 6 (S102).

Thereby, the processor 46 of the head mounted display 4 determines the initial position of the center (front) of each of the head mounted display 4 and the controller 6 (S103). Hereinafter, in this initial position, as shown in FIG. 6A, the center of each of the head mounted display 4 and the controller 6 is oriented in the same direction (plus direction of the Y axis). At this time, as shown in FIG. 6B, the center of the high-resolution area 54 coincides with the center of the image 14, and the pointer 16 is superimposed on the center of the object 15 of the image 14. . At this time, the center of the object 15 in the image 14 coincides with the center of the image 14 .

The image processing unit 52 of the processor 46 of the head mounted display 4 acquires the rotation angle data indicating the rotation angle of the controller 6 included in the detection signal of the sensor 18 of the controller 6 (S104). Next, the image processing unit 52 of the processor 46 acquires rotation angle data indicating the rotation angle of the mounting unit 8 of the head mounted display 4, which is included in the detection signal of the sensor 34 of the head mounted display 4 (S105).

In the present embodiment, "rotation" means rotation around the Z-axis, and "rotation angle" means the rotation angle in rotation around the Z-axis. In the top views shown in (a) of each of FIGS. 6 to 13, the rotation angle of each of the head mounted display 4 and the controller 6 at the initial position is assumed to be 0°, and clockwise from the initial position ((a) of FIG. 6). ) in the direction indicated by the arrow P) (hereinafter also referred to as “rightward”), the rotation angle is positive, and the counterclockwise direction from the initial position (the direction indicated by the arrow Q in FIG. 6(a)) ( Hereinafter, the rotation angle when rotated to the left is assumed to be negative.

The image processing unit 52 of the processor 46 determines whether the absolute value of the difference between the rotation angle of the controller 6 and the rotation angle of the mounting unit 8 of the head-mounted display 4 is larger than the first threshold and smaller than the second threshold. It is determined whether or not (S106). That is, it is determined whether or not the absolute value of the difference satisfies the following equation (1).

First threshold<|Rotation angle of controller 6−Rotation angle of HMD 4|<Second threshold (Formula 1)

In Equation 1, the first threshold is, for example, 5°. Also, the second threshold is a value larger than the first threshold, eg, 45°.

If the absolute value of the difference satisfies the above equation 1, that is, if the absolute value of the difference is larger than the first threshold and smaller than the second threshold (YES in S106), the image processing of the processor 46 Unit 52 assumes that user 12's line of sight is at pointer 16 (ie, user 12 is following the movement of pointer 16 with their eyes). The image processing unit 52 of the processor 46 determines the center position of the high resolution area 54 as the position of the pointer 16 (S107). Then, the image processing unit 52 of the processor 46 executes foveated rendering for controlling the position of the high resolution area 54 so that the pointer 16 is superimposed on the high resolution area 54 (S108), and draws the image 14 ( S109).

Here, an example in which the absolute value of the difference is larger than the first threshold and smaller than the second threshold will be described with reference to FIGS. 7 to 9. FIG.

In the example shown in FIG. 7A, the rotation angle of the mounting portion 8 of the head mounted display 4 is 0°, and the rotation angle of the controller 6 is 20°. That is, the user 12 does not rotate the mounting portion 8 of the head mounted display 4 from the initial position, but rotates the controller 6 rightward from the initial position by 20°. At this time, the absolute value of the difference is |20°−0°|=20°, which satisfies the above equation (1). As shown in FIG. 7B, the angle of the image 14 does not change from the initial angle (the angle shown in FIG. 6B), and the pointer 16 follows the rotation of the controller 6 to move the image. 14 to the right from the center of the object 15 by a predetermined distance (a distance corresponding to a rotation angle of 20° of the controller 6). In this case, the image processing unit 52 of the processor 46 estimates that the line of sight of the user 12 is at the position of the pointer 16 and determines the center position of the high resolution area 54 to be the position of the pointer 16 .

In the example shown in FIG. 8(a), the rotation angle of the mounting portion 8 of the head mounted display 4 is -20°, and the rotation angle of the controller 6 is 0°. That is, the user 12 does not rotate the controller 6 from the initial position, but rotates the mounting section 8 of the head mounted display 4 leftward by 20° from the initial position. At this time, the absolute value of the difference is |0°−(−20°)|=20°, which satisfies the above equation (1). Note that, as shown in FIG. 8B, the angle of the image 14 changes from the initial angle to the left by a predetermined angle (the angle corresponding to the rotation angle of the mounting portion 8 of the head mounted display 4 -20°). . On the other hand, since the positional relationship between the pointer 16 and the object 15 in the image 14 does not change, the pointer 16 shifts to the right from the center of the image 14 together with the object 15 as the angle of the image 14 changes. In this case, the image processing unit 52 of the processor 46 estimates that the line of sight of the user 12 is at the position of the pointer 16 and determines the center position of the high resolution area 54 to be the position of the pointer 16 .

In the example shown in FIG. 9A, the rotation angle of the mounting portion 8 of the head mounted display 4 is -45°, and the rotation angle of the controller 6 is -20°. That is, the user 12 rotates the mounting portion 8 of the head-mounted display 4 to the left from the initial position by 45 degrees, and rotates the controller 6 to the left from the initial position by 20 degrees. At this time, the absolute value of the difference is |−20°−(−45°)|=25°, which satisfies Equation 1 above. As shown in FIG. 9B, the angle of the image 14 changes from the initial angle to the left by a predetermined angle (the angle corresponding to the rotation angle of the mounting portion 8 of the head mounted display 4 -45°). . On the other hand, the pointer 16 follows the rotation of the controller 6 and moves a predetermined distance to the left of the object 15 in the image 14 from the center of the object 15 (distance corresponding to the rotation angle of the controller 6 -20°). do. As a result, the pointer 16 is shifted rightward from the center of the image 14 . In this case, the image processing unit 52 of the processor 46 estimates that the line of sight of the user 12 is at the position of the pointer 16 and determines the center position of the high resolution area 54 to be the position of the pointer 16 .

Returning to the flowchart of FIG. 5, in step S106, if the absolute value of the difference does not satisfy the above equation 1, that is, the absolute value of the difference is less than or equal to the first threshold, or more than or equal to the second threshold. In that case (NO in S106), the image processing unit 52 of the processor 46 estimates that the line of sight of the user 12 is at the center of the image 14 (that is, the user 12 is gazing at the center of the image 14). Then, the image processing unit 52 of the processor 46 determines the center position of the high resolution area 54 as the center of the image 14 (S110). Then, the image processing unit 52 of the processor 46 executes foveated rendering for controlling the position of the high resolution area 54 so that the center of the image 14 overlaps the high resolution area 54 (S108), and draws the image 14. (S109).

Now, with reference to FIGS. 10 to 13, examples of cases where the absolute value of the difference is less than or equal to the first threshold or greater than or equal to the second threshold will be described.

In the example shown in FIG. 10(a), the rotation angle of the mounting portion 8 of the head mounted display 4 is 0°, and the rotation angle of the controller 6 is 0°. That is, the user 12 has not rotated either the mounting section 8 of the head mounted display 4 or the controller 6 from the initial position. At this time, the absolute value of the difference is |0°−0°|=0°, which does not satisfy Equation 1 above. Note that, as shown in FIG. 10B, the angle of the image 14 does not change from the initial angle, and the pointer 16 does not move from the center of the object 15 of the image 14 with respect to the object 15 . In this case, the image processing portion 52 of the processor 46 assumes that the line of sight of the user 12 is at the center of the image 14 and determines the center position of the high resolution area 54 to be the center of the image 14 .

In the example shown in FIG. 11(a), the rotation angle of the mounting portion 8 of the head mounted display 4 is -45°, and the rotation angle of the controller 6 is 0°. That is, the user 12 does not rotate the controller 6 from the initial position, but rotates the mounting section 8 of the head mounted display 4 leftward from the initial position by 45°. At this time, the absolute value of the difference is |0°−(−45°)|=45°, which does not satisfy Equation 1 above. Note that, as shown in FIG. 11B, the angle of the image 14 is changed from the initial angle to the left by a predetermined angle (the angle corresponding to the rotation angle of the mounting portion 8 of the head mounted display 4 -45°). . On the other hand, since the positional relationship between the pointer 16 and the object 15 in the image 14 does not change, the pointer 16 shifts to the right from the center of the image 14 together with the object 15 as the angle of the image 14 changes. In this case, the image processing portion 52 of the processor 46 assumes that the line of sight of the user 12 is at the center of the image 14 and determines the center position of the high resolution area 54 to be the center of the image 14 .

In the example shown in FIG. 12(a), the rotation angle of the mounting portion 8 of the head mounted display 4 is -45°, and the rotation angle of the controller 6 is 20°. That is, the user 12 rotates the mounting portion 8 of the head-mounted display 4 to the left from the initial position by 45 degrees, and rotates the controller 6 to the right from the initial position by 20 degrees. At this time, the absolute value of the difference is |20°−(−45°)|=65°, which does not satisfy Equation 1 above. Note that, as shown in FIG. 12B, the angle of the image 14 is changed from the initial angle to the left by a predetermined angle (the angle corresponding to the rotation angle of the mounting portion 8 of the head mounted display 4 -45°). . On the other hand, the pointer 16 follows the rotation of the controller 6 and moves to the right of the object 15 in the image 14 from the center of the object 15 by a predetermined distance (distance corresponding to the rotation angle of 20 degrees of the controller 6). . In this case, the image processing portion 52 of the processor 46 assumes that the line of sight of the user 12 is at the center of the image 14 and determines the center position of the high resolution area 54 to be the center of the image 14 .

In the example shown in FIG. 13(a), the rotation angle of the mounting portion 8 of the head mounted display 4 is 0°, and the rotation angle of the controller 6 is -2°. That is, the user 12 does not rotate the mounting portion 8 of the head mounted display 4 from the initial position, but rotates the controller 6 leftward from the initial position by 2 degrees. At this time, the absolute value of the difference is |2°−0°|=2°, which does not satisfy the above equation (1). Incidentally, as shown in FIG. 13(b), the angle of the image 14 does not change from the initial angle. On the other hand, the pointer 16 follows the rotation of the controller 6 and moves to the left of the object 15 in the image 14 from the center of the object 15 by a predetermined distance (distance corresponding to the rotation angle of 2° of the controller 6). . In this case, the image processing portion 52 of the processor 46 assumes that the line of sight of the user 12 is at the center of the image 14 and determines the center position of the high resolution area 54 to be the center of the image 14 .

Returning to the flowchart of FIG. 5, after step S109, if the display of the image 14 is to be continued (YES in S111), the process returns to step S104 described above. On the other hand, if the display of the image 14 is not to be continued (NO in S111), the flow chart of FIG. 5 ends.

[5. effect]
In this embodiment, the image display system 2 includes a mounting unit 8 mounted on the head of the user 12 , a display unit 10 arranged on the mounting unit 8 and displaying an image 14 to the user 12 , and A sensor 34 for detecting movement, a sensor 18 for detecting movement of the operating body (controller 6), and a processor 46 for controlling display of the image 14, which are arranged in a high resolution area 54 and around the high resolution area 54. and a middle resolution region 56 (low resolution region 58) having a resolution lower than that of the high resolution region 54, and a pointer 16 pointing to a position on the image 14 based on the detection result of the sensor 18. and a processor 46 for moving. The processor 46 estimates the line of sight of the user 12 based on the detection results of the sensors 34 and 18, and controls the position of the high resolution area 54 so that the estimated line of sight of the user 12 is superimposed on the high resolution area 54. .

According to this, the processor 46 estimates the line of sight of the user 12 by a simple method in order to realize foveated rendering using the respective detection results of the movement of the mounting unit 8 and the movement of the operating body. be able to. As a result, since it is not necessary to mount a dedicated device for detecting the line of sight of the user 12 on the mounting unit 8, the size and power consumption of the head mounted display 4 can be reduced.

Moreover, in the present embodiment, the sensor 34 detects the rotation angle of the mounting portion 8 . A sensor 18 detects the rotation angle of the operating body. The processor 46 estimates the line of sight of the user 12 based on the difference between the rotation angle of the operating body and the rotation angle of the mounting section 8 .

According to this, the processor 46 can easily estimate the line of sight of the user 12 based on the difference between the rotation angle of the operating body and the rotation angle of the mounting section 8 .

In addition, in the present embodiment, the processor 46 determines (i) if the difference is greater than a first threshold and less than a second threshold that is greater than the first threshold, the line of sight of the user 12 is at the position of the pointer 16, the position of the high resolution region 54 is controlled so that the pointer 16 is superimposed on the high resolution region 54, and (ii) the difference is less than or equal to the first threshold, or If it is greater than or equal to the threshold of 2, it is assumed that the line of sight of the user 12 is at the center of the image 14 and the position of the high resolution area 54 is controlled so that the center of the image 14 overlaps the high resolution area 54 .

According to this, the processor 46 can easily estimate that the line of sight of the user 12 is at the position of the pointer 16 or at the center of the image 14 based on the difference between the rotation angle of the operating body and the rotation angle of the mounting section 8. can be done.

Further, in the present embodiment, the operating body is the controller 6 which is arranged separately from the mounting section 8 and operated by the user 12 .

According to this, the processor 46 utilizes the detection results of the movement of the mounting part 8 and the movement of the controller 6 to estimate the line of sight of the user 12 in a simple manner in order to achieve foveated rendering. be able to.

In addition, in the present embodiment, the image display method is as follows: (a) movement of the wearing unit 8 that is worn on the head of the user 12 and that has a display unit 10 that displays the image 14 to the user 12; (b) detecting the movement of the operating body (controller 6); (d) moving the pointer 16 pointing to a position on the image 14 based on the detection result of (b) above; and (e) estimating the line of sight of the user 12 based on the detection result of (a) above and the detection result of (b) above, and superimposing the estimated line of sight of the user 12 on the high resolution area 54 . and controlling the position of 54.

According to this, the line of sight of the user 12 can be estimated by a simple method in order to realize foveated rendering, as described above.

[6. Modification]
A configuration of an image display system 2A according to a modification of the embodiment will be described with reference to FIG. FIG. 14 is a block diagram showing the configuration of an image display system 2A according to a modification of the embodiment.

As shown in FIG. 14, an image display system 2A according to the modified example of the embodiment includes a head mounted display 4A, a controller 6, and a terminal device 60. As shown in FIG. The image display system 2A is a tethered image display system using a terminal device 60 in addition to the head mounted display 4 and the controller 6 .

The terminal device 60 is, for example, a personal computer, a smartphone, or the like, and is connected to the head mounted display 4A for wired communication. Note that the terminal device 60 may be connected to the head mounted display 4A so as to be capable of wireless communication.

Next, a configuration of a head mounted display 4A according to a modified example of the embodiment will be described with reference to FIG. FIG. 15 is a block diagram showing the configuration of a head mounted display 4A according to a modification of the embodiment.

As shown in FIG. 15, in a head mounted display 4A according to the modification of the embodiment, the processor 46A does not have the image processing unit 52 described above, and the processor 62 of the terminal device 60 has the image processing unit 52 described above. has the function of The terminal device 60 is connected to the USB connector 50 of the head mounted display 4A. The processor 46A of the head mounted display 4A transmits and receives various data to and from the terminal device 60 via the USB connector 50. FIG. Specifically, the processor 46A transmits the detection signal of the sensor 34 and the detection signal of the sensor 18 (see FIG. 3) of the controller 6 to the terminal device 60. FIG. The processor 46A also receives data representing the image 14 transmitted from the terminal device 60 .

In this modification, the processor 62 of the terminal device 60 has the function of the image processing unit 52 described above, so the processing load on the processor 46A of the head mounted display 4A can be reduced.

(other modifications, etc.)
As described above, the above embodiment has been described as an example of the technology disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can be applied to embodiments in which modifications, replacements, additions, omissions, etc. are made as appropriate. Further, it is also possible to combine the constituent elements described in the above embodiments to form a new embodiment.

Therefore, other embodiments will be exemplified below.

In the above embodiment, the image processing unit 52 of the processor 46 estimates the line of sight of the user 12 based on the detection signal of the sensor 34 and the detection signal of the sensor 18 of the controller 6, and the estimated line of sight of the user 12 is a high-resolution area. Although the position of the high resolution area 54 is controlled so as to be superimposed on 54, it is not limited to this. For example, the image processing unit 52 of the processor 46 may control the position and/or range of the high resolution area 54 such that the estimated line of sight of the user 12 overlaps the high resolution area 54 .

Further, in the above embodiment, the image processing unit 52 of the processor 46 estimates the line of sight of the user 12 based on the respective rotation angles of the head mounted display 4 and the controller 6, but is not limited to this. And the line of sight of the user 12 may be estimated based on each rotation angle and/or amount of movement of the controller 6 (movement distance and movement direction in the three axial directions of the X-axis, Y-axis, and Z-axis).

Further, in the above embodiment, the case where the operating body is one controller 6 has been described, but the present invention is not limited to this, and the operating body may be a plurality of controllers. For example, the operating body may be a right-hand controller gripped by the right hand of the user 12 and a left-hand controller gripped by the left hand of the user 12 . In this case, the position of the pointer 16 may be an intermediate position between the indicated coordinates of the right-hand controller and the left-hand controller. Alternatively, the position of the pointer 16 may be the indicated coordinate position of the last valid controller of the right-hand controller and the left-hand controller.

Further, in the above embodiment, the case where the operating body is the controller 6 that is held and operated by hand has been described. Alternatively, the hand of the user 12 may be used as the operating body. In this case, a hand tracking function can be realized by recognizing the hand of the user 12 with the camera 42 of the head mounted display 4 (4A).

Further, in the above-described embodiment, the controller 6 is of a type in which the operation body is gripped by hand and operated, but is not limited to this, and may be, for example, a bracelet-type, ring-type, or glove-type controller.

In the above embodiments, each component may be implemented by dedicated hardware or by executing a software program suitable for each component. Each component may be realized by reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory by a program execution unit such as a CPU or processor.

Also, part or all of the functions of the image display system 2 (2A) according to the above embodiment may be implemented by a processor such as a CPU executing a program.

As described above, the embodiment has been described as an example of the technique of the present disclosure. To that end, the accompanying drawings and detailed description have been provided.

Therefore, among the components described in the attached drawings and detailed description, there are not only components essential for solving the problem, but also components not essential for solving the problem in order to illustrate the above technology. can also be included. Therefore, it should not be immediately recognized that those non-essential components are essential just because they are described in the attached drawings and detailed description.

In addition, the above-described embodiments are intended to illustrate the technology of the present disclosure, and various modifications, replacements, additions, omissions, etc. can be made within the scope of the claims or equivalents thereof.

The present disclosure is applicable as, for example, an image display system including a head-mounted display and a controller.

2, 2A Image display system 4, 4A Head-mounted display 6 Controller 8 Mounting unit 10 Display unit 12 User 13, 15, 17 Object 14 Image 16 Pointer 18, 34 Sensor 20, 36 Communication unit 22, 38 Haptics 24, 40 Memory 26 , 44 input units 28, 46, 46A, 62 processors 30, 48 batteries 32, 50 USB connector 42 camera 52 image processing unit 54 high resolution area 56 medium resolution area 58 low resolution area 60 terminal device

Claims (5)

a wearing part worn on a user's head;
a display unit arranged on the mounting unit and configured to display an image to the user;
a first sensor that detects movement of the mounting part;
a second sensor that detects the movement of the operating body;
A processor for controlling display of the image, the image including a first area and a second area arranged around the first area and having a resolution lower than that of the first area. a processor that draws and moves a pointer pointing to a position on the image based on the detection result of the second sensor;
The processor estimates a line of sight of the user based on detection results of each of the first sensor and the second sensor, and estimates the line of sight of the user so that the estimated line of sight of the user overlaps the first area. An image display system that controls the position of one region. The first sensor detects a rotation angle of the mounting part,
The second sensor detects a rotation angle of the operating body,
The image display system according to claim 1, wherein the processor estimates the line of sight of the user based on the difference between the rotation angle of the operating body and the rotation angle of the mounting section. The processor determines that (i) if the difference is greater than a first threshold and less than a second threshold that is greater than the first threshold, then the user's line of sight is at the location of the pointer; and controlling the position of the first area so that the pointer overlaps the first area, and (ii) the difference is equal to or less than the first threshold, or the second threshold In the above case, it is estimated that the line of sight of the user is at the center of the image, and the position of the first area is controlled so that the center of the image overlaps with the first area. The image display system described in . The image display system according to any one of claims 1 to 3, wherein the operating body is a controller arranged separately from the mounting section and operated by the user. (a) detecting movement of a wearable part that is worn on the head of a user and has a display that displays an image to the user;
(b) detecting movement of the operating body;
(c) rendering said image comprising a first region and a second region disposed around said first region and having a lower resolution than said first region;
(d) moving a pointer indicating a position on the image based on the detection result of (b);
(e) estimating the line of sight of the user based on the detection result of (a) and the detection result of (b); and controlling the position of the region.

Images