JP6935921B2 - Terminal devices and programs - Google Patents

Description

The present invention relates to terminal devices and programs.

There is a head-mounted display (HMD: Head Mounted Display) that can be worn on the user's head and displayed on a display arranged in front of the user's eyes to display an image in a virtual space. For example, in a game using an HMD, in addition to a display worn on the head, there is a game in which a device such as a controller that is held and operated by a hand is used in combination to play the game. However, since the user cannot observe the hand because the field of view to the real space is covered by the HMD, an erroneous operation may occur when the finger is once separated from the controller. Therefore, there is an HMD that detects a movement such as a tilt of the head and performs a display / non-display operation of options (for example, Patent Document 1).

Further, in recent years, as the performance of smartphones and the like has improved, some smartphones and the like are used as HMDs. For example, by attaching a smartphone to the head using a simple attachment, it can be used as an HMD. In this case, when the smartphone is attached to the head, the input device such as the touch panel provided on the smartphone cannot be used. Therefore, for example, it is necessary to realize the operation for the options displayed on the HMD by detecting the movement of the head using a sensor such as a gyro built in the smartphone.

Japanese Patent No. 5767386

When operating the options displayed on the HMD using the movement of the head, the direction of the head is set within a specific range (within the range of the determination area of the options), and in that state, a predetermined time ( For example, there is an operation method in which the selection is confirmed when 3 seconds have passed. However, in such an operation method, when it is necessary to select from a large number of options, the range of the judgment area of each option displayed on the HMD becomes small, so that the head is unstable. When the operation was performed by movement, erroneous input such as unintentionally deviating from the range of the judgment area occurred, and the operability was poor.

One of the objects of the present invention is to provide a terminal device and a program for improving operability when performing a selection operation on an option displayed on an HMD.

Another object of the present invention is to provide a terminal device and a program capable of exerting the effects described in the embodiments described later.

In order to solve the above-mentioned problems, one aspect of the present invention is at least a head-mounted display system capable of displaying a view image representing a view from a virtual viewpoint in a virtual space as a stereoscopic image using binocular parallax. A terminal device that can be used as a part and has a detection unit that detects information about the direction of the terminal device and a selection object that has a determination area that determines whether or not there is a selection from the user is arranged in the virtual space. The object is provided with an object arranging unit and a display control unit for displaying the selected object arranged in the view direction from the virtual viewpoint in the virtual space on the display unit according to the detection result of the detection unit. The arrangement unit is a terminal device that changes the determination area of the selected object based on the positional relationship between the line-of-sight position corresponding to the line-of-sight direction from the virtual viewpoint in the virtual space and the determination area of the selected object.

Further, one aspect of the present invention is a program for making a computer function as the above-mentioned terminal device.

The external view which shows an example of the HMD system which concerns on 1st Embodiment. The figure which shows an example of the hardware configuration of the terminal apparatus which concerns on 1st Embodiment. The figure which shows the definition of the direction of the virtual space which concerns on 1st Embodiment. Explanatory drawing of a field of view direction and a line-of-sight position which concerns on 1st Embodiment. The figure which shows an example of the selection menu screen. The figure which shows an example of the selection menu screen which the line-of-sight position overlaps with the selection object. The figure which shows an example of the selection menu screen which changed the judgment area. The figure which shows the selection menu screen example when the selection of a selection object is confirmed. The figure which shows an example of the selection menu screen when the number of selection objects is large. The figure which shows the selection menu screen example which the line-of-sight position overlaps with the selection object. The block diagram which shows an example of the functional structure of the terminal apparatus which concerns on 1st Embodiment. The flowchart which shows an example of the object arrangement processing which concerns on 1st Embodiment. The flowchart which shows an example of the determination process of the selection object which concerns on 1st Embodiment. The figure which shows an example of the selection menu screen where the line-of-sight position and the judgment area of a selection object overlap. The figure which shows an example of the selection menu screen in which the line-of-sight position and the judgment area of a selection object do not overlap. The flowchart which shows an example of the determination process of the selection object which concerns on 1st Embodiment. The figure which shows an example of the selection menu screen which concerns on 2nd Embodiment. The figure which shows an example of the selection menu screen when the user's field of view changes in the pitch direction. The figure which shows an example of the selection menu screen when the user's field of view changes in the yaw direction. The figure which shows an example of the selection menu screen when the user's field of view changes to the roll direction. The flowchart which shows an example of the object arrangement processing which concerns on 2nd Embodiment. The figure which shows an example of the selection menu screen which concerns on 3rd Embodiment. The flowchart which shows an example of the object arrangement processing which concerns on 3rd Embodiment.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[First Embodiment]
The first embodiment of the present invention will be described.
FIG. 1 is an external view showing an example of an HMD (Head Mounted Display) system according to the present embodiment. The HMD is attached to the user's head and can display a visual field image representing the visual field from a virtual viewpoint in the virtual space. For example, the HMD can display a stereoscopic image using the binocular parallax between the right eye and the left eye. In addition, the HMD is equipped with a sensor that detects the movement and tilt of the HMD such as a gyro, detects changes in the movement and tilt of the head of the user who is wearing it, and virtualizes it on the display according to the change. Display a view image in the view direction of space. For example, the visual field image projected on the HMD changes to a rightward visual field image in the virtual space when the user's head is turned to the right, and changes to an upward visual field image in the virtual space when the user's head is turned upward. You can give the user an immersive feeling as if you were there.

The illustrated HMD system 1 has a configuration in which the terminal device 10 can be used as an HMD by attaching the terminal device 10 provided with the display unit 12 to the attachment 2. The attachment 2 can be attached to the terminal device 10 so as to cover the front view of the user while being attached to the user's head, and is a lens for the right eye for visually recognizing the display unit 12 of the attached terminal device 10. It includes a 3R and a left eye lens 3L, and a strap 5 for attaching the HMD system 1 to the user's head. By attaching the HMD system 1 to the head, the user can visually recognize the field of view image displayed on the terminal device 10 through the right eye lens 3R and the left eye lens 3L.

[Hardware configuration of terminal device 10]
The terminal device 10 is a portable computer device that can be used as at least a part of the HMD system 1, and is a mobile phone such as a smartphone or a feature phone, a personal digital assistant (PDA), a tablet PC, or a home game. Machines, etc. can be applied. In the present embodiment, the terminal device 10 will be described as a smartphone.

FIG. 2 is a diagram showing an example of the hardware configuration of the terminal device 10. The terminal device 10 includes, for example, a display unit 12, a sensor 13, a timer 14, a storage unit 15, a communication unit 16, and a CPU (Central Processing Unit) 17.

The display unit 12 is a display for displaying information such as an image or text, and includes, for example, a liquid crystal display panel, an organic EL (Electroluminescence) display panel, and the like. For example, the display unit 12 displays a stereoscopic image (right eye image and left eye image) using binocular parallax as a visual field image from a virtual viewpoint in the virtual space. In addition, the display unit 12 displays various objects arranged in the virtual space together with the field of view image. The various objects are, for example, user interfaces such as a selection object (choice) having a determination area for determining whether or not a user has selected, and an object for making it possible to visually recognize the line-of-sight position corresponding to the line-of-sight direction from a virtual viewpoint. It is an object to be displayed for UI).

The sensor 13 is a sensor that detects information regarding the direction of the terminal device 10. For example, the sensor 13 is a gyro sensor that detects an angle, an angular velocity, an angular acceleration, or the like of an object. The sensor 13 may be a sensor that detects a change in direction, or may be a sensor that detects the direction itself. For example, the sensor 13 is not limited to the gyro sensor, but may be an acceleration sensor, an inclination sensor, a geomagnetic sensor, or the like.

The timer 14 has a timekeeping function for measuring time.

The storage unit 15 includes, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), an EEPROM (Electrically Erasable Programmory), a ROM (Read-One Memory), a ROM (Read-One Memory), and a ROM (Read-One Memory). Stores virtual space data (image data), data of objects placed in the virtual space, game programs using the virtual space, and the like.

The communication unit 16 communicates with another device via the network NW.

The CPU 17 functions as a control center that controls each unit included in the terminal device 10. For example, the CPU 17 functions as a control unit that controls each unit of the terminal device 10 by executing various programs stored in the storage unit 15.

Each of the above-described configurations is communicably connected to each other via a bus. Further, the terminal device 10 may be configured to include a hardware configuration such as a speaker (not shown), an audio output terminal, a camera, a GPS (Global Positioning System) receiving module, an operation button for receiving a user's operation input, or a touch panel. ..

For example, the terminal device 10 executes a game program using the HMD. This game is played while the user is wearing the HMD system 1 and changes the visual field range of the visual field image (game screen) displayed on the display unit 12 by changing the direction of the head or the like. The terminal device 10 detects information regarding the direction of the terminal device 10 based on the detection result of the sensor 13, and causes the display unit 12 to display a field of view image (game screen) in the field of view based on the detection result. Further, when the terminal device 10 accepts the input of the selection operation by the user, the terminal device 10 arranges a selection object (choice) having a determination area for determining the presence or absence of selection from the user in the virtual space, and displays the view image together with the display unit 12. Display it.

[Definition in virtual space]
FIG. 3 is a diagram showing a definition of the direction of the virtual space according to the present embodiment. In the present embodiment, the vertical direction in which the user stands upright is defined as the Z axis, the axis orthogonal to the Z axis and the direction connecting the user and the display unit 12 is defined as the X axis, and is orthogonal to the Z axis and the X axis. Let the axis be the Y axis.

Here, a change in the rotation direction about the Z axis is also referred to as a change in the yaw direction (horizontal direction), and a change in the rotation direction about the Y axis is also referred to as a change in the pitch direction (vertical direction). , The change in the rotation direction about the X axis is also referred to as the change in the roll direction. For example, the sensor 13 described above detects the angular velocity or the angular acceleration in the rotation direction (yaw direction, pitch direction, and roll direction) of each axis. The change in the yaw direction may be referred to as a change in the left-right direction, and a change in the pitch direction may be referred to as a change in the up-down direction.

FIG. 4 is an explanatory diagram of a field of view direction and a line of sight position according to the present embodiment. In this figure, assuming that the virtual viewpoint K (user's virtual viewpoint) in the virtual space is the intersection (origin) of the X-axis, the Y-axis, and the Z-axis, and the user's line-of-sight direction is the X-axis direction, the view from the virtual viewpoint is set. The range of the view image in the direction (that is, the view) is the yaw angle α (internal angle between the broken line a and the broken line b and the internal angle between the broken line c and the broken line d) and the pitch angle centered on the line-of-sight direction (X-axis direction). It is a range determined by β (internal angle between the broken line a and the broken line d and the internal angle between the broken line b and the broken line c). Here, the yaw angle α and the pitch angle β are angles set in advance as the angle of view of the visual field image of the virtual space to be displayed on the HMD system 1. When the user's head changes in the pitch direction or the yaw direction, the line-of-sight direction changes from the X-axis direction to the pitch direction or the yaw direction according to the change, and the line-of-sight direction also changes in the pitch direction or the yaw direction. Further, when the user's head changes in the roll direction, the line-of-sight direction remains in the X-axis direction and the line-of-sight direction rotates in the roll direction.

In addition, various objects are arranged in the virtual space as needed. For example, various objects are arranged on the same plane (same layer) orthogonal to the line-of-sight direction in the virtual space. For example, assuming that the surface on which the object is arranged in the virtual space is a surface L1 as shown in the figure, the line-of-sight position P1 which is an object for making the line-of-sight position visible is a broken line indicating a range in the view direction on this surface L1. It is arranged at the center of the range (view) surrounded by the quadrangle with the intersections a, b, c, and d having a1, b1, c1, and d1 as the vertices (that is, the intersection of the plane L1 and the X axis). The position where the line-of-sight position P1 is arranged is not limited to the center of the range (field of view) surrounded by the quadrangle with the intersections a1, b1, c1, and d1 as the vertices on the surface L1, and is not limited to the center of other positions (for example, from the center). It may be placed at a position slightly below). Further, a selection object (choice) having a determination area for determining the presence / absence of selection from the user is arranged on the surface L1 in the virtual space. The determination region may be a surface (two-dimensional shape) or a three-dimensional shape (three-dimensional shape). For example, a selection object (choice) arranged in a range (field of view) surrounded by a quadrangle having the intersections a1, b1, c1, and d1 of the surface L1 and the broken lines a, b, c, and d1 as vertices, and a line-of-sight position P1. Will be displayed on the display unit 12 together with the field of view image.

When the user's head changes in the pitch direction or the yaw direction, the line-of-sight direction and the line-of-sight direction change in the pitch direction or the yaw direction according to the change. Move so that it is displayed in the center). Further, when the user's head changes in the roll direction, the line-of-sight position P1 rotates according to the change in the roll direction while maintaining the arranged position. On the other hand, the selected object does not move from the arranged position even if the field of view changes in the pitch direction, the yaw direction, or the roll direction. That is, when the field of view changes, the selected objects placed in the changed field of view are displayed.

When displaying a stereoscopic image using binocular parallax, there are a visual field direction and a visual field direction corresponding to each virtual viewpoint for the right eye and a left eye, and the visual field image and various objects in the respective visual field directions and the visual field directions. An image for the right eye and an image for the left eye including the above are displayed, but in the present embodiment, the image for the right eye and the image for the left eye will be described without distinction for the sake of simplicity.

[Display example and operation example of selected object]
Next, a display example and an operation example of the selected object will be described. In the following, a screen in which a plurality of selected objects can be selected and displayed on the display unit 12 is also referred to as a “selection menu screen”. The selection menu screen is a screen in which a selection object also arranged in the virtual space and a line-of-sight position P1 corresponding to the line-of-sight direction are displayed using a field-of-view image in the field-of-view direction in the virtual space as a background image.

FIG. 5 is a diagram showing an example of the selection menu screen. On the illustrated selection menu screen G10, the line-of-sight position P1 which is an object arranged at the line-of-sight position corresponding to the line-of-sight direction in the virtual space is displayed in the center of the screen. In addition, two columns in which four selected objects arranged in the virtual space are arranged in the Z-axis direction are displayed side by side with a predetermined interval on the left and right. In the left column, four selection objects of option A, option C, option E, and option G are displayed side by side at predetermined intervals in the Z-axis direction. In the right column, four selection objects of option B, option D, option F, and option H are displayed side by side at predetermined intervals in the Z-axis direction. Here, it is assumed that the display area of the selected object is a determination area for determining the presence / absence of selection. The display area of the selected object and the determination area for determining the presence / absence of selection may be different areas. Since the line-of-sight position P1 does not overlap with the determination area of any of the selected objects in the line-of-sight direction, none of the selected objects (choices) is selected.

When the direction of the user's head changes while the selection menu screen G10 shown in FIG. 5 is displayed, the relative position between the selected object and the line-of-sight position P1 changes. The line-of-sight position P1 is fixed and displayed in the center of the screen because it follows and moves so as to maintain the line-of-sight position in response to a change in the direction of the head. On the other hand, since the viewing direction of the selected object changes according to the change in the direction of the head, the selected object moves on the screen according to the change in the viewing direction.

FIG. 6 is a diagram showing an example of a selection menu screen in which the line-of-sight position P1 overlaps the selection object. In the illustrated selection menu screen G11, the line-of-sight position P1 and the option C have an overlapping positional relationship in the line-of-sight direction. When the line-of-sight position P1 overlaps the determination area of the selected object, the determination area of the selected object (here, option C) changes.

FIG. 7 is a diagram showing an example of a selection menu screen in which the determination area has changed. In the illustrated selection menu screen G12, the determination area of the selection object (here, option C) in which the line-of-sight position P1 overlaps the determination area is expanded. As a result, it becomes easy to maintain the line-of-sight position P1 in the determination area of the selected object to be selected, and it becomes easy to prevent an erroneous operation of accidentally moving the line-of-sight position P1 out of the determination area.

The judgment area of the selected object may be expanded by expanding the judgment area to be arranged in the virtual space, or by moving the position to be arranged while keeping the size of the judgment area closer to the user (virtual viewpoint K). It may be displayed in a larger size than the object. Further, here, the display area is expanded in the same manner as the determination area of option C, but only the determination area of option C may be expanded and the display area may not be expanded.

Further, the display color of the selected object (here, option C) in which the line-of-sight position P1 overlaps the determination area may be changed. For example, the display color of the selected object whose line-of-sight position P1 does not overlap the determination area is blue, and the display color of the selected object whose line-of-sight position P1 overlaps the determination area (here, option C) is changed from blue to red. You may. This makes it easier to see what is selected. In this way, when expanding the determination area of the selected object, the display mode of the selected object may be changed.

Further, the elapsed time, which is an object for making it possible to visually recognize the elapsed time since the line-of-sight position P1 overlaps the determination area in the vicinity of the selected object (here, option C) in which the line-of-sight position P1 overlaps the determination area. The gauge T1 is arranged and displayed. For example, the elapsed time gauge T1 is a circular gauge, and the bars increase in a circular shape with the passage of time after the line-of-sight position P1 overlaps the determination region, and the bars increase in a circular shape when a predetermined time or more elapses. The bar stretches around. Here, the above-mentioned predetermined time is a determination time preset as a threshold value for determining whether or not the selected object is selected, and the elapsed time after the line-of-sight position P1 overlaps the determination area is the predetermined time or more. When becomes, it is determined that the selected object in the determination area is selected. On the other hand, when the elapsed time from the line-of-sight position P1 overlapping the determination area is less than a predetermined time, it is determined that the selected object in the determination area has not been selected yet. For example, if the line-of-sight position P1 deviates from the determination area before the bar increasing in a circle becomes one round, it is determined that the selected object in the determination area is not selected, and the measurement time (elapsed time). Is reset.

On the other hand, when the circularly increasing bar of the elapsed time gauge T1 reaches one round, the selection of the selected object (here, option C) in which the line-of-sight position P1 overlaps the determination area is confirmed.
FIG. 8 is a diagram showing an example of a selection menu screen when the selection of the selected object is confirmed. In the illustrated selection menu screen G13, the circularly increasing bar of the elapsed time gauge T1 extends for one round, and the selection object (here, option C) in which the line-of-sight position P1 overlaps the determination area is selected. It will be confirmed. At this time, other selected objects arranged in the virtual space may be erased or may be displayed inconspicuously by increasing the transmittance. This results in a relatively highlighted display of the selected selected objects.

The above-mentioned expansion of the determination area of the selected object is an expansion within a range that does not overlap with the determination area of another selected object in the line-of-sight direction. Further, the expansion of the determination area of the selected object may be an expansion within a range that is spaced apart from the determination area of another selected object in a direction orthogonal to the line-of-sight direction. Therefore, as the number of selected objects increases, it becomes difficult to expand the determination area of the selected objects. Therefore, when expanding the determination area of the selected object, the terminal device 10 may change the position of the selected object arranged elsewhere.

FIG. 9 is a diagram showing an example of a selection menu screen when the number of selected objects is large. In the illustrated selection menu screen G14, two columns in which five selection objects arranged in the virtual space are arranged in the Z-axis direction are displayed side by side with a predetermined interval on the left and right. In the left column, five selection objects of option A, option C, option E, option G, and option I are displayed side by side at predetermined intervals in the Z-axis direction. In the right column, five selection objects of option B, option D, option F, option H, and option J are displayed side by side at predetermined intervals in the Z-axis direction. Further, the line-of-sight position P1 arranged in the virtual space is displayed in the center of the screen. The line-of-sight position P1 has a positional relationship that does not overlap with the determination area of any of the selected objects in the line-of-sight direction.

FIG. 10 is a diagram showing an example of the selection menu screen in which the line-of-sight position P1 overlaps the selection object in the selection menu screen shown in FIG. On the illustrated selection menu screen G15, the line-of-sight position P1 and the option E have an overlapping positional relationship in the line-of-sight direction. When the line-of-sight position P1 overlaps the determination area of the selected object, the determination area of the selected object (here, option E) is expanded, but the arrangement interval with the selected object adjacent in the Z-axis direction is narrow. Therefore, when the determination area of the selected object (here, option E) is expanded, the selected objects (here, options A and C) on the upper side of the option E and the lower side are secured so as to secure the expanded area. Each of the selected objects (here, options G and I) moves away from option E.

Other information may be displayed in the enlarged display area of the selected object. For example, when the selected object is the selected object of the purchased item, the item name is usually displayed in the selected object, but when selected, the purchase price or the like may be displayed in addition to the item name.

[Functional configuration of terminal device 10]
Next, the functional configuration of the terminal device 10 will be described with reference to FIG. FIG. 11 is a block diagram showing an example of the functional configuration of the terminal device 10 according to the present embodiment. The terminal device 10 includes a control unit 110 as a functional configuration realized by the CPU 17 executing a program stored in the storage unit 15. The control unit 110 includes a detection unit 111, a determination unit 112, an object arrangement unit 113, and a display control unit 114.

The detection unit 111 detects information regarding the direction of the terminal device 10 based on the detection result of the sensor 13. For example, the detection unit 111 determines the change in the visual field direction or the visual field direction of the user who wears the HMD system 1 to which the terminal device 10 is attached, based on the information regarding the direction detected by the sensor 13 or the change in the direction. It is detected according to the direction in which 10 is facing. The detection unit 111 may detect information regarding the rate of change in the visual field direction.

The determination unit 112 determines whether or not the selected object is selected based on the positional relationship between the user's line-of-sight position P1 and the determination area of the selected object. For example, the determination unit 112 measures the elapsed time from the positional relationship in which the line-of-sight position P1 and the determination area of the selected object do not overlap in the line-of-sight direction to the overlapping positional relationship, and selects based on the measured elapsed time. Determine the presence or absence. For example, the determination unit 112 determines that the selected object has been determined (selected) when a predetermined time or more has elapsed since the line-of-sight position P1 and the determination area of the selected object overlap. On the other hand, the determination unit 112 states that the selected object has not yet been selected (no selection) when the elapsed time from the positional relationship in which the line-of-sight position P1 and the determination area of the selected object overlap is less than a predetermined time. judge.

The object arrangement unit 113 arranges a selection object having a determination area for determining whether or not there is a selection from the user in the virtual space. Further, the object arranging unit 113 determines the line-of-sight direction from the virtual viewpoint based on the detection result of the detection unit 111, and sets an object (for example, the line-of-sight position P1) for making the line-of-sight position corresponding to the line-of-sight direction visible. Place in virtual space. Further, the object arranging unit 113 arranges an object (for example, an elapsed time gauge T1) for making the elapsed time visible in the virtual space. For example, the object arranging unit 113 may arrange the above-mentioned various objects on the same surface (same layer) in the virtual space, or may arrange them separately on a plurality of surfaces (plural layers). The object arranging unit 113 obtains time information such as elapsed time by timing using a timer 14.

The display control unit 114 causes the display unit 12 to display a field of view image in the field of view from a virtual viewpoint in the virtual space according to the detection result of the detection unit 111. Further, the display control unit 114 displays various objects (selected objects, line-of-sight position, elapsed time gauge, etc.) arranged in the visual field direction from the virtual viewpoint in the virtual space according to the detection result of the detection unit 111. Display on 12. The field of view image data in each direction in the virtual space is stored in the storage unit 15 as virtual space data. The display control unit 114 acquires a visual field image corresponding to the visual field direction from the virtual space data and displays it on the display unit 12 as a visual field image.

[Details of object placement processing]
Next, the object placement process according to the present embodiment will be described in detail.
The object arrangement unit 113 changes the determination area of the selected object based on the positional relationship between the line-of-sight position P1 corresponding to the line-of-sight direction from the virtual viewpoint in the virtual space and the determination area of the selected object. For example, the object arranging unit 113 widens the determination area of the selected object when the line-of-sight position P1 and the determination area of the selected object are overlapped from the positional relationship that does not overlap in the line-of-sight direction. Here, when the determination area of the selected object is widened, the object arranging unit 113 widens the determination area of another selected object within a range that does not overlap with the determination area in the line-of-sight direction.

When the determination area of the selected object is widened, the object arranging unit 113 widens the determination area of the selected object within a range that is spaced apart from the determination area of the other selected object in a direction orthogonal to the line-of-sight direction. Further, the object arranging unit 113 may change the position of another selected object when the determination area of the selected object is widened. Further, the object arranging unit 113 may change the display mode of the selected object when the determination area of the selected object is widened. Further, the object arranging unit 113 waits for a predetermined time from the positional relationship in which the line-of-sight position P1 and the determination area of the selected object do not overlap in the line-of-sight direction to overlap, and then the determination area of the selected object or the determination area. The display area corresponding to may be widened.

Next, the operation of the object placement process according to the present embodiment will be described with reference to FIG. FIG. 12 is a flowchart showing an example of the object arrangement process according to the present embodiment. It is assumed that the visual field image in the visual field direction and the visual field position P1 are displayed on the display unit 12. First, when the trigger for displaying the selection menu is received, the control unit 110 arranges a plurality of selection objects in the virtual space (step S100). As a result, the display unit 12 displays a selection menu screen (see, for example, FIG. 5) including the field of view image and the plurality of selection objects. Here, the trigger of the selection menu display is that the game is started, a predetermined time has passed since the game was started, a specific action has been performed in the game, and so on. It was set when it was necessary to confirm the intention.

The control unit 110 determines whether or not the line-of-sight position P1 and the determination area of the selected object overlap in the line-of-sight direction (step S102). When it is determined (YES) that the line-of-sight position P1 and the determination area of the selected object overlap in the line-of-sight direction (for example, see FIG. 6), the control unit 110 sets the determination area of the selected object to be enlarged (for example). , FIG. 7), that is, the determination area is widened (step S104). At this time, the control unit 110 arranges the elapsed time gauge T1 in the vicinity of the selection object (for example, option C) whose line-of-sight position P1 overlaps the determination area.

On the other hand, when it is determined that the line-of-sight position P1 and the determination area of the selected object do not overlap in the line-of-sight direction (for example, see FIG. 5) (NO), the control unit 110 sets the determination area of the selected object to the initial value. That is, the determination area is not changed (step S106).

Next, the control unit 110 determines whether or not the trigger for turning off the display of the selection menu has been accepted (step S108). Here, the trigger for turning off the display of the selection menu is that when any of the selection objects is selected, the option to turn off the display of the selection menu is selected when a predetermined time has elapsed after displaying the selection object. If, etc. When it is determined that the trigger for turning off the display of the selection menu has been accepted (YES), the control unit 110 erases the selected objects arranged in the virtual space (step S110), and ends the object arrangement process.

On the other hand, when it is determined that the trigger for turning off the display of the selection menu is not accepted (NO), the control unit 110 detects the angular velocity based on the detection result of the sensor 13 (step S112). Then, the control unit 110 determines whether or not there is a change in the viewing direction based on the detected angular velocity (step S114). When it is determined that there is no change in the viewing direction (NO), since there is no change in the relative position between the line-of-sight position P1 and the determination area of the selected object, the control unit 110 returns to the process of step S108 and displays the selection menu. Determines if an off trigger has been accepted. On the other hand, when it is determined that there is a change in the viewing direction (YES), the relative position between the line-of-sight position P1 and the determination area of the selected object changes. Therefore, the control unit 110 returns to the process of step S102 and returns to the line-of-sight position. It is determined whether or not P1 and the determination area of the selected object overlap in the line-of-sight direction. If it continues to overlap, the judgment area of the selected object is left set to enlarge, and if it changes from the non-overlapping state to the overlapping state, the judgment area of the selected object is changed from the initial value to the enlargement. , When the overlapping state is changed to the non-overlapping state, the judgment area of the selected object is changed from the expansion to the initial value.

[Operation of selection / non-selection judgment processing]
Next, with reference to FIG. 13, the operation of the determination process for determining whether or not the selected object arranged in the object arrangement process according to the present embodiment is selected will be described. FIG. 13 is a flowchart showing an example of a determination process for determining whether or not the selected object according to the present embodiment is selected. First, when the trigger for displaying the selection menu is received, the control unit 110 arranges a plurality of selection objects in the virtual space, so that the selection menu screen (for example, see FIG. 5) is displayed on the display unit 12 (step). S200). Further, the control unit 110 resets the measurement time (step S202).

Next, the control unit 110 determines whether or not the line-of-sight position P1 and the determination area of the selected object overlap in the line-of-sight direction (step S204). When it is determined that the line-of-sight position P1 and the determination area of the selected object do not overlap in the line-of-sight direction (see, for example, FIG. 5) (NO), the control unit 110 returns the process to step S202. On the other hand, when it is determined (YES) that the line-of-sight position P1 and the determination area of the selected object overlap in the line-of-sight direction (for example, see FIG. 6), the control unit 110 starts timing the elapsed time and clocks the elapsed time. An elapsed time gauge T1 in which a bar corresponding to the elapsed elapsed time is displayed is arranged in the virtual space in the vicinity of the selected objects determined to overlap (step S206). As a result, the elapsed time gauge T1 is displayed in the vicinity of the selected object (see, for example, FIG. 7).

Next, the control unit 110 determines whether or not a predetermined time or more has elapsed since the line-of-sight position P1 and the determination area of the selected object overlap (step S208). If it is determined that the predetermined time or more has not elapsed (NO), the control unit 110 returns to the process of step S204 and continues to measure the elapsed time since the overlap. On the other hand, when it is determined that a predetermined time or more has elapsed since the line-of-sight position P1 and the determination area of the selected object overlap (YES), the control unit 110 determines that the selected object has been selected (step S210), and determines. End the process.

[Other examples of selection / non-selection determination processing]
In the above-mentioned determination process for determining the selection / non-selection of the selected object, an example of determining the selection / non-selection based on the elapsed time since the line-of-sight position P1 and the determination area of the selected object overlap has been described. Another example will be described for the timekeeping specification of.
FIG. 14 is a diagram showing an example of a selection menu screen in which the line-of-sight position P1 and the determination area of the selected object overlap. In the illustrated selection menu screen G16, the line-of-sight position P1 and the selection object (here, option B) overlap, and an elapsed time gauge T1 indicating the elapsed time since the overlap is displayed. This elapsed time is a state in which the predetermined time for determining the selection has not been reached. Here, when the user turns to the left, the selected object moves to the right relative to the line-of-sight position P1.

FIG. 15 is a diagram showing an example of a selection menu screen in which the line-of-sight position P1 and the determination area of the selected object do not overlap because the selected object moves to the right relative to the line-of-sight position P1. .. The illustrated selection menu screen G17 shows that the line-of-sight position P1 and the determination area of the selection object (here, option B), which overlapped in FIG. 14, have changed to a state in which they do not overlap. When the line-of-sight position P1 and the selected object do not overlap, the measurement time is reset to 0 in the determination process shown in FIG. 13, but the time counting of the elapsed time may be stopped without resetting. For example, in the example shown in FIG. 15, the time counting of the elapsed time is stopped, and the bar of the elapsed time gauge T1 indicates the elapsed time (when the line-of-sight position P1 and the determination area of the selected object overlap). It shows that it has stopped. Further, when the line-of-sight position P1 and the selected object determination area change from the overlapping state to the non-overlapping state, the time may advance slower than the time of the elapsed time of the overlapping state, or the time may be set back. You may let me. As described above, the determination unit 112 may perform different timing depending on whether the line-of-sight position P1 and the determination area of the selected object are in a positional relationship that does not overlap or overlap in the line-of-sight direction.

In this case, even if the line-of-sight position P1 once deviates from the judgment area of the selected object, if the line-of-sight position P1 overlaps by moving the line of sight to the judgment area of the same selected object again, the time counting of the elapsed time is restarted from the continuation. Even if the line of sight is accidentally removed from the determination area for a moment, it is not necessary to retime the elapsed time from 0.

When the line-of-sight position P1 deviates from the determination area of the selected object and a predetermined time (for example, 10 seconds) elapses, it may be considered that the user does not intend to continue the selection and the measurement time may be reset. .. For example, the determination unit 112 determines the elapsed time measured after the line-of-sight position P1 and the determination area of the selected object overlap each other so that the line-of-sight position P1 and the determination area of the selected object do not overlap. When the time elapses, the measurement time (elapsed time) may be reset.

Further, when the line-of-sight position P1 moves to the determination area of another selected object (options A, C, etc.), the timed measurement time (elapsed time) is reset, and the timed time is started from 0. .. For example, the determination unit 112 has a positional relationship in which the line-of-sight position P1 and the determination area of another selected object overlap each other for the elapsed time measured after the line-of-sight position P1 and the determination area of the selected object overlap. If so, reset the measurement time (elapsed time).

Next, with reference to FIG. 16, the operation of the determination process for determining whether or not the selected object is selected based on the timekeeping specification of the elapsed time described above will be described. FIG. 16 is a flowchart showing an example of a determination process for determining whether or not the selected object according to the present embodiment is selected. The illustrated steps S300, S302, S304. S306. Each process of S308 and S310 is performed in steps S200, S202 and S204 shown in FIG. S206. Corresponding to each process of S208 and S210, only the subsequent process when NO is determined in the process of step S308 is different from the determination process shown in FIG. Here, a process different from the determination process shown in FIG. 13 will be described, and the description of the same process will be omitted.

In step S308, when it is determined that a predetermined time or more has not elapsed since the line-of-sight position P1 and the determination area of the selected object overlap (NO), the control unit 110 determines that the line-of-sight position P1 and the determination area of the selected object are aligned with each other. It is determined whether or not they overlap in the line-of-sight direction (step S312). If it is determined that the line-of-sight position P1 and the determination area of the selected object do not overlap (NO), the control unit 130 temporarily stops the time counting and display (elapsed time gauge T1 bar) of the elapsed time (step S314). .. At this time, instead of pausing the time and display of the elapsed time, the time and display may be performed so that the passage of time progresses slowly, or the time and display may be displayed so that the passage of time recedes. May be good.

Next, the control unit 110 determines whether or not a predetermined time or more has elapsed since the line-of-sight position P1 and the determination area of the selected object do not overlap (step S316). If it is determined that the predetermined time or more has elapsed (YES), the control unit 110 considers that the user has no intention of continuing the selection, returns the process to step S302, and resets the measurement time.

On the other hand, when it is determined that a predetermined time or more has not elapsed since the line-of-sight position P1 and the determination area of the selected object do not overlap (NO), the control unit 110 returns the process to step S312 and returns the process to the line-of-sight position. It is determined whether or not P1 and the determination area of the selected object overlap in the line-of-sight direction. Then, when it is determined that the line-of-sight position P1 and the determination area of the selected object overlap in the line-of-sight direction (YES), the control unit 110 determines that the selected object in which the line-of-sight position P1 overlaps last time is the selected object. It is determined whether or not there is (step S318). If it is determined that the selected objects overlapped each other last time, the control unit 110 returns the process to step S308, and resumes the time counting and display of the elapsed time. In step S314, the control unit 110 sets the time and display so that the passage of time advances slowly, or sets the time and display so that the passage of time recedes, the original time and display. Return to. On the other hand, when it is determined that the selected object whose line-of-sight position P1 overlaps is another selected object (NO), the control unit 110 returns the process to step S302 to reset the measurement time.

[Summary of the first embodiment]
As described above, the terminal device 10 according to the present embodiment is at least the HMD system 1 capable of displaying a visual field image representing the visual field from a virtual viewpoint in the virtual space as a stereoscopic image using binocular parallax. It is a terminal device that can be used as a part. For example, the terminal device 10 includes a detection unit 111, an object arrangement unit 113, and a display control unit 114. The detection unit 111 detects information regarding the direction of the terminal device 10. The object arrangement unit 113 arranges a selection object having a determination area for determining whether or not there is a selection from the user in the virtual space. The display control unit 114 causes the display unit 12 to display the selected objects arranged in the visual field direction from the virtual viewpoint in the virtual space according to the detection result of the detection unit 111. Then, the object arranging unit 113 changes the determination area of the selected object based on the positional relationship between the line-of-sight position P1 corresponding to the line-of-sight direction from the virtual viewpoint in the virtual space and the determination area of the selected object.

Here, the change of the determination area may be a change in the size of the determination area, for example, a change in the area (area) or a change in the volume, or a change in the position where the determination area is arranged. good. Further, the change of the determination area may be a change of the shape of the determination area. Further, the above-mentioned positional relationship may be a two-dimensional planar positional relationship projected in the line-of-sight direction, or may be a three-dimensional positional relationship when the object is a three-dimensional object (three-dimensional shape). You may.

As a result, the terminal device 10 can change the size (apparent size) of the determination area according to the positional relationship between the line-of-sight position P1 and the determination area of the selected object. It is possible to improve the operability when performing. The object arranging unit 113 may be easier to see because it is selected by changing the color, brightness, or the like of the determination area.

For example, on the selection menu screen, there are cases where it is not desired to interfere with the display of the field of view image as the background as much as possible, or when there are many selection items, it is necessary to reduce the determination area of one option. In such a case, if the determination area is small, there is a difficulty in operability such that the line-of-sight position is unintentionally out of the determination area due to an erroneous operation within the period for determining the presence or absence of selection. Therefore, for example, the object arranging unit 113 widens the determination area of the selected object when the line-of-sight position P1 and the determination area of the selected object are overlapped from the positional relationship that does not overlap in the line-of-sight direction. As a result, the terminal device 10 expands the determination area when the line-of-sight position P1 enters the determination area of the selected object, so that it is possible to suppress the occurrence of an erroneous operation in which the line-of-sight position P1 unintentionally deviates from the determination area. ..

For example, when the determination area of the selected object is widened, the object arranging unit 113 widens the determination area of another selected object within a range that does not overlap with the determination area in the line-of-sight direction. As a result, when the user intentionally changes to the selection of another selected object, the terminal device 10 is erroneously operated so that the line-of-sight position P1 deviates from the currently overlapping determination area so that the selection can be made. Occurrence can be suppressed. Further, since the terminal device 10 can maintain the independence of the determination area of each selected object, it is possible to prevent an erroneous operation when it is desired to select any one of the selected objects.

Further, when the determination area of the selected object is widened, the object arranging unit 113 may be widened within a range in which a predetermined interval is provided in a direction orthogonal to the line-of-sight direction with the determination area of another selected object. As a result, when the user intentionally changes the selection to another selected object, the terminal device 10 can immediately remove the line-of-sight position P1 from the currently overlapping determination area.

Further, the object arranging unit 113 may change the position of another selected object when the determination area of the selected object is widened. As a result, the terminal device 10 can secure an area for expanding the determination area by moving the other selected objects even when there are a large number of selected objects.

When the determination area of the selected object is widened, the object arranging unit 113 changes the display mode of the selected object. For example, the object placement unit 113 may display other information on the enlarged selection object. For example, when the selected object is a selection of purchased items, the item name is displayed on the selected object of the normal size before the line-of-sight positions P1 overlap, and when the line-of-sight positions P1 overlap and expand, the item name is displayed. In addition, the purchase price and the like may be displayed. As a result, the terminal device 10 can display the information related to the selection decision on the selection object, so that the user can prevent the user from making an erroneous selection. Further, the object arranging unit 113 may change the color of the selected object being enlarged to a color different from that before the enlargement. In this case, the terminal device 10 can improve the operability by improving the visibility of the selected object to be selected.

Further, the object arranging unit 113 waits for a predetermined time from the positional relationship in which the line-of-sight position P1 and the determination area of the selected object do not overlap in the line-of-sight direction to overlap, and then the determination area of the selected object or the determination area. The display area corresponding to may be widened. That is, even when the line-of-sight position P1 enters the determination area of the selected object, the terminal device 10 does not immediately expand the determination area of the selected object, but instead expands the determination area for a predetermined time (for example, 0.2 to 0. A time lag of about 3 seconds) may be provided. As a result, the terminal device 10 can prevent the determination area from expanding when the line-of-sight position P1 passes over the determination area in a short time even though there is no intention to select, so that the terminal device 10 can visually recognize when performing the selection operation. The sex can be improved. For example, if the selected object is enlarged each time the line-of-sight position P1 passes, the visibility becomes annoying and poor. It should be noted that the time lag is provided only in the display area of the selected object, and the time lag may not be provided in the determination area. Further, a time lag may be provided in the same manner for both the display area of the selected object and the determination area. Further, by advancing input speed to determine regions of line-of-sight position P1, it may be a time lag is set. For example, advances provided a time lag when input speed is above the predetermined value, advance input speed may not be provided a time lag in the case of less than the predetermined value.

Further, the object arranging unit 113 determines the line-of-sight direction from the virtual viewpoint based on the detection result of the detection unit 111, and arranges an object for making the line-of-sight position P1 visible in the virtual space. As a result, the terminal device 10 can determine, for example, the line-of-sight position P1 corresponding to the line-of-sight direction in the field-of-view image (for example, the center of the field-of-view image) according to the direction of the head of the user wearing the HMD system 1. It is possible to improve the operability by making it easier for the user to recognize the positional relationship between the line-of-sight position P1 and the selected object.

In the present embodiment, an example of detecting the line-of-sight direction based on the direction of the terminal device 10 has been described, but the present invention is not limited to this, and for example, the line-of-sight direction is detected by detecting the direction of the user's eyeball. (Eye tracking) may be used. In that case, since the line-of-sight direction is based on detection different from the movement of the head, the line-of-sight position P1 moves in the field-of-view image, not in a fixed position in the field-of-view image.

Further, the terminal device 10 includes a determination unit 112 that determines whether or not the selected object is selected based on the positional relationship between the line-of-sight position P1 and the determination area of the selected object. As a result, the terminal device 10 can provide an operation UI for selecting an option in the HMD system 1 by using the line of sight of the user.

For example, the determination unit 112 measures the elapsed time from the positional relationship in which the line-of-sight position P1 and the determination area of the selected object do not overlap in the line-of-sight direction to the overlapping positional relationship, and selects based on the measured elapsed time. Determine the presence or absence. As a result, the terminal device 10 determines that the selection is made on the condition that a predetermined time elapses after the line-of-sight position P1 overlaps the determination area of the selected object. It is possible to prevent the position P1 from being selected only by passing through it.

Further, the object arranging unit 113 arranges an object in the virtual space so that the elapsed time can be visually recognized. As a result, the terminal device 10 can present the timing at which the selection is confirmed so that the user can easily recognize it.

Further, the determination unit 112 performs different timing depending on whether the line-of-sight position P1 and the determination area of the selected object are in a positional relationship that does not overlap or overlap in the line-of-sight direction. For example, the determination unit 112 performs timing when the line-of-sight position P1 and the determination area of the selected object have an overlapping positional relationship in the line-of-sight direction, and stops timing (does not reset to 0) when the positional relationship does not overlap. Keep the elapsed time with). In addition, when the positional relationship does not overlap, the time may be measured so that the passage of time progresses slowly or the passage of time recedes without stopping the timing. As a result, even if the line-of-sight position P1 is out of the determination area due to an erroneous operation, if the line-of-sight position P1 is moved to the same determination area again, the terminal device 10 does not retime from 0, but the elapsed time from the continuation. Since the timing is restarted, the operability can be improved.

In addition, the determination unit 112 measures the elapsed time measured after the line-of-sight position P1 and the determination area of the selected object overlap in the line-of-sight direction when the line-of-sight position P1 and the determination area of another selected object are in the line-of-sight direction. If there is an overlapping positional relationship, reset it. As a result, when the line-of-sight position P1 moves to the determination area of another selected object, the terminal device 10 considers that the user has no intention of continuing the selection and returns the timed elapsed time to 0, which is the same. Even if the line-of-sight position P1 is unintentionally moved to the determination area, the measurement of the elapsed time is not restarted from the continuation, and it is possible to prevent an erroneous operation from occurring.

Further, the determination unit 112 does not overlap the line-of-sight position P1 and the determination area of the selected object in the line-of-sight direction with respect to the elapsed time measured since the line-of-sight position P1 and the determination area of the selected object overlap in the line-of-sight direction. It may be reset when a predetermined time elapses in the positional relationship. As a result, when the line-of-sight position P1 deviates from the determination area and a predetermined time (for example, 10 seconds) elapses, the terminal device 10 considers that the user has no intention of continuing the selection, and measures the elapsed time. Is returned to 0, so even if the line-of-sight position P1 is unintentionally moved to the same determination area, the measurement of the elapsed time is not restarted from the continuation, and it is possible to prevent an erroneous operation from occurring.

Further, the determination unit 112 may change the measuring speed depending on which position in the determination area of the selected object the line-of-sight position P1 overlaps in the line-of-sight direction. For example, the closer the determination unit 112 is to the center of the determination area, the faster the timing may be. In this case, the closer to the center of the determination area, the faster the bar of the elapsed time gauge T1 extends by one round, and the shorter the period for which it is determined that the bar has been selected as a result. As a result, the terminal device 10 can also provide the user with a method of shortening the period until the selection is determined while providing the determination time for preventing the malfunction. Instead of changing the timing speed, the determination time for determining (determining) that the clock has been selected may be changed. For example, the closer the determination unit 112 is to the center of the determination area, the shorter the determination time may be.

Further, after the line-of-sight position P1 is entered in the determination region, a region may be provided in which the speed of timing (the speed at which the bar of the elapsed time gauge T1 extends) becomes faster. The region may be a region that does not require elapsed time (ie, a region whose selection is determined immediately). In that case, the area that does not require this elapsed time is set to a position in the determination area that does not include the line-of-sight position P1 so that the line-of-sight position P1 is not unintentionally entered by the user.

Further, the display control unit 114 causes the display unit 12 to display a field of view image in the field of view from a virtual viewpoint in the virtual space according to the detection result of the detection unit 111. As a result, the terminal device 10 can provide a visual field image in the visual field direction in the virtual space so that the user can see it according to the movement of the head of the user who is equipped with the HDM system 1. Further, since the terminal device 10 can simultaneously display the field of view image in the virtual space and the selected object, the user can make a selection while referring to the game information in the virtual space.

[Second Embodiment]
Next, a second embodiment of the present invention will be described.
Since the basic configurations of the HMD system 1 and the terminal device 10 according to the present embodiment are the same as the configurations shown in FIGS. 1, 2, and 11, the processing characteristic of the present embodiment will be described. In the present embodiment, when the selection operation is performed by changing the viewing direction on the selection menu screen, a process of limiting the change in the relative position of the selected object with respect to the line-of-sight position in a predetermined direction will be described.

FIG. 17 is a diagram showing an example of a selection menu screen according to the present embodiment. On the illustrated selection menu screen G20, the line-of-sight position P1 is displayed in the center of the screen, and four selection objects (option A, option B, option C,) arranged side by side in the Z-axis direction in the virtual space. And option D) are displayed. For example, when the user changes the view direction (line-of-sight direction) to the pitch direction (vertical direction) which is the arrangement direction (Z-axis direction) of the selected objects in order to select option A, the view direction (line-of-sight direction) changes. Correspondingly, the field view image in the virtual space changes in the pitch direction (vertical direction). Further, the four selected objects arranged in the virtual space change in the pitch direction (vertical direction) relative to the line-of-sight position P1 by changing the visual field image in the pitch direction (vertical direction).

FIG. 18 is a diagram showing an example of a selection menu screen when the user's visual field direction (line-of-sight direction) changes to the pitch direction (upward direction). In the illustrated selection menu screen G21, when the user changes the visual field direction (line-of-sight direction) upward, the selected object changes downward relative to the line-of-sight position P1, and the line-of-sight position P1 is the option A. It shows the state where it overlaps the judgment area.

On the other hand, in response to a change in the user's visual field direction (line-of-sight direction) other than the arrangement direction of the selected objects (here, other than the pitch direction), the selected object is moved according to the change with respect to the line-of-sight position P1. The change in relative position may be restricted.
FIG. 19 is a diagram showing an example of a selection menu screen when the user's visual field direction (line-of-sight direction) changes to the yaw direction (right direction) in the selection menu screen shown in FIG. In the illustrated selection menu screen G22, the visual field image is changed to the right because the user changes the visual field direction (line-of-sight direction) to the right, but the change in the relative position of the selected object with respect to the line-of-sight position P1 is small. The broken line M1 indicates the position of the selected object (the position at the center of the determination area) before the movement. On the other hand, the broken line M2 indicates the position of the selected object (the position at the center of the determination area) after the movement. That is, the object arranging unit 113 moves the selected object from the position centered on the broken line M1 to the position centered on the broken line M2 by changing the user's visual field direction (line-of-sight direction) in the yaw direction (right direction). ..

Further, since the object arranging unit 113 also changes the user's visual field direction (line-of-sight direction) in the roll direction in a direction other than the arrangement direction of the selected objects, the object arranging unit 113 rotates and moves the selected objects according to the change. Let me. FIG. 20 is a diagram showing an example of a selection menu screen when the user's visual field direction (line-of-sight direction) changes to the roll direction. The illustrated selection menu screen G23 shows the arrangement of the selection objects when the user's visual field direction (line-of-sight direction) changes by an angle θ in the roll direction (here, counterclockwise about the X-axis). The broken line M1 indicates the position of the selected object (the position at the center of the determination area) before the rotation movement. On the other hand, the broken line M2 indicates the position of the selected object (the position at the center of the determination area) after the rotational movement. That is, when the user's line-of-sight direction (line-of-sight direction) changes by an angle θ in the roll direction (here, counterclockwise about the X-axis), the object arrangement unit 113 is in the rotation direction between the line-of-sight position P1 and the selected object. The selected object is rotated by an angle θ in the roll direction (here, counterclockwise around the X axis) so that the relative position does not change.

The angle at which the user's visual field direction (line-of-sight direction) changes and the angle at which the selected object is rotated may or may not be matched. For example, the angle for rotating and moving the selected object may be an angle obtained by multiplying the angle at which the user's view direction (line-of-sight direction) has changed by a predetermined ratio, or an angle obtained by adding or subtracting a predetermined angle. Further, an upper limit may be provided for the angle at which the selected object is rotated and moved.

In this way, when the object arranging unit 113 determines that the visual field direction (line-of-sight direction) from the virtual viewpoint in the virtual space has changed based on the detection result of the detection unit 111, the object arrangement unit 113 has the visual field direction (line-of-sight direction). Based on the change, the selected objects are moved while restricting the movement of the selected objects in the alignment direction. For example, the object arranging unit 113 moves the selected object according to the change in the visual field direction (line-of-sight direction) so that the relative position between the line-of-sight position P1 and the selected object does not change in a direction other than the arrangement direction of the selected objects. ..

The object arranging unit 113 may move the selected object according to a change in the visual field direction (line-of-sight direction) so that the line-of-sight position P1 falls within a predetermined range in a direction other than the arrangement direction of the selected objects. Here, the predetermined range is, for example, the width of the determination area of the selected objects in the direction orthogonal to the arrangement direction of the selected objects (for example, the width indicated by W in FIG. 19). Further, the predetermined range may be, for example, a range of angles in the rotation direction about the line-of-sight direction.

Next, the operation of the object placement process according to the present embodiment will be described with reference to FIG. FIG. 21 is a flowchart showing an example of the object arrangement process according to the present embodiment. First, when the trigger for displaying the selection menu is received, the control unit 110 arranges a plurality of selected objects side by side in a predetermined direction (for example, the Z-axis direction) in the virtual space (step S400). As a result, the display unit 12 displays a selection menu screen (see, for example, FIG. 17) including the field of view image and the plurality of selection objects.

The control unit 110 detects the angular velocity based on the detection result of the sensor 13 (step S402). Then, the control unit 110 determines whether or not there is a change in the viewing direction (line-of-sight direction) based on the detected angular velocity (step S404).

When it is determined that there is a change in the viewing direction (line-of-sight direction) (YES), the control unit 110 limits the movement in a predetermined direction (arrangement direction of selected objects, for example, the Z-axis direction), and the viewing direction. The selected object is moved based on the change in (line-of-sight direction) (see FIGS. 18 and 19), and the process proceeds to step S408 (step S406). On the other hand, when it is determined that there is no change in the visual field direction (line-of-sight direction) (NO), the control unit 110 proceeds to the process of step S408.

Next, the control unit 110 determines whether or not the trigger for turning off the display of the selection menu has been accepted (step S408). If it is determined that the trigger for turning off the display of the selection menu is not accepted (NO), the control unit 110 returns the process to step S402. On the other hand, when it is determined that the trigger for turning off the display of the selection menu has been accepted (YES), the control unit 110 erases the selected objects arranged in the virtual space (step S410), and ends the object arrangement process. ..

[Summary of the second embodiment]
As described above, the terminal device 10 according to the present embodiment is at least one of the HMD systems 1 capable of displaying a visual field image representing the visual field from a virtual viewpoint in the virtual space as a stereoscopic image using binocular parallax. It is a terminal device that can be used as a unit. For example, the terminal device 10 includes a detection unit 111, an object arrangement unit 113, and a display control unit 114. The detection unit 111 detects information regarding the direction of the terminal device 10. The object arranging unit 113 arranges a plurality of selected objects having a determination area for determining the presence / absence of selection from the user side by side in a predetermined direction in the virtual space. The display control unit 114 causes the display unit 12 to display the selected objects arranged in the visual field direction from the virtual viewpoint in the virtual space according to the detection result of the detection unit 111. Then, when it is determined that the visual field direction from the virtual viewpoint in the virtual space has changed based on the detection result of the detection unit 111, the object arranging unit 113 moves in a predetermined direction based on the change in the visual field direction. Move the selected object while limiting.

As a result, the terminal device 10 can limit the relative position between the line-of-sight position P1 and the determination area of the selected object so as not to change in a direction unnecessary for selection. Operability can be improved.

Further, the object arranging unit 113 determines the line-of-sight direction from the virtual viewpoint based on the detection result of the detection unit 111. Further, the object arranging unit 113 arranges an object (for example, the line-of-sight position P1) for making the line-of-sight position corresponding to the line-of-sight direction from the virtual viewpoint visible in the virtual space. As a result, the terminal device 10 can determine, for example, the line-of-sight position P1 corresponding to the line-of-sight direction in the field-of-view image (for example, the center of the field-of-view image) according to the direction of the head of the user wearing the HMD system 1. It is possible to improve the operability by making it easier for the user to recognize the positional relationship between the line-of-sight position P1 and the selected object.

For example, the object arranging unit 113 arranges an object (for example, the line-of-sight position P1) for making the line-of-sight position visible and a selected object on the same surface in the virtual space. The same plane is, for example, a plane orthogonal to the line-of-sight direction in the virtual space. The above-mentioned identical surface is not limited to exactly the same surface, and may be substantially the same surface. Further, the same plane is not limited to a plane orthogonal to the line-of-sight direction, and may be a plane in any direction in the virtual space, or may be a plane or a curved surface. As a result, the terminal device 10 can realize the selection operation UI of the options based on the relative position (positional relationship) between the line-of-sight position P1 and the selected object. Specifically, the terminal device 10 arranges the line-of-sight position P1 and the selected object on the same surface in the virtual space, so that the line-of-sight position P1 and the selected object have the same focal length in stereoscopic vision using binocular parallax. Therefore, it is possible to make it easier to visually recognize the positional relationship between the two.

When it is determined that the viewing direction from the virtual viewpoint in the virtual space has changed based on the detection result of the detecting unit 111, the object arranging unit 113 selects the line-of-sight position P1 in a direction other than the arrangement direction of the selected objects. The selected object is moved according to the change in the visual field direction so that the line-of-sight position is within the width in the direction orthogonal to the arrangement direction of the determination area of the object. As a result, the terminal device 10 prevents the line-of-sight position P1 from deviating from the width of the determination area of the selected object except for the direction necessary for selecting the selected object (that is, the direction unnecessary for the operation). Since it is restricted, it is possible to facilitate the selection operation for the options. In addition, if the selection object is restricted in this way, even if the selection object is made smaller (narrowing the width), the effect on the operation is small. It can also be reduced.

Further, the object arranging unit 113 may change the line-of-sight position P1 beyond the width of the selected object in the sense of leaving a cancel area for canceling the selection (remove the line-of-sight position P1 outside the determination area). ..

When it is determined that the viewing direction from the virtual viewpoint in the virtual space has changed based on the detection result of the detecting unit 111, the object arranging unit 113 sets the line-of-sight position P1 and the selected object in directions other than the arrangement direction of the selected objects. The selected object is moved according to the change in the field of view so that the relative position with the object does not change. As a result, the terminal device 10 is relative to the line-of-sight position P1 and the determination area of the selected object except for the direction in which the line-of-sight position P1 required for selecting the selected object is moved (that is, the direction unnecessary for the operation). Since the position is restricted so that it does not change, it is possible to facilitate the selection operation for the options. In addition, if the selection object is restricted in this way, even if the selection object is made smaller (narrowing the width), the effect on the operation is small. It can also be reduced.

Further, the terminal device 10 includes a determination unit 112 that determines whether or not the selected object is selected based on the relative position (positional relationship) between the line-of-sight position P1 and the determination area of the selected object. As a result, the terminal device 10 can provide an operation UI for selecting an option in the HMD system 1 by using the line of sight of the user.

When the object arranging unit 113 determines that the field of view from the virtual viewpoint in the virtual space has changed in the rotation direction about the line-of-sight direction based on the detection result of the detection unit 111, the line-of-sight position P1 and the selected object Rotate the selected object in the direction of rotation so that the relative position in the direction of rotation does not change. As a result, even when the user tilts his / her head in the roll direction, the terminal device 10 maintains the relative relationship between the vertical / horizontal directions with respect to the user's head and the arrangement direction of the selected objects. It can be made easier to do.

Note that the object arranging unit 113 rotates the selected object in the rotation direction even when it is determined that the field of view from the virtual viewpoint in the virtual space has changed in the rotation direction about the line-of-sight direction based on the detection result of the detection unit 111. It is not necessary to rotate it. In this case, on the selection menu screen, the selected object is displayed diagonally by the amount of change in the rotation direction together with the field of view image as the background.

Further, the display control unit 114 causes the display unit 12 to display a field of view image in the field of view from the virtual viewpoint in the virtual space according to the detection result of the detection unit 111, whereby the terminal device 10 can display the HDM system 1 It is possible to provide a visual field image in the visual field direction in the virtual space so that the user can see it according to the movement of the head of the user wearing the. Further, since the terminal device 10 can simultaneously display the field of view image in the virtual space and the selected object, the user can make a selection while referring to the game information in the virtual space.

[Third Embodiment]
Next, a third embodiment of the present invention will be described.
Since the basic configurations of the HMD system 1 and the terminal device 10 according to the present embodiment are the same as the configurations shown in FIGS. 1, 2, and 11, the processing characteristic of the present embodiment will be described. In the present embodiment, when a plurality of columns in which a plurality of selected objects are lined up are arranged, the selected objects to be determined whether or not to be selected are switched for each column based on the movement of the user's head. An example will be described.

FIG. 22 is a diagram showing an example of a selection menu screen in which the selection object to be determined can be switched for each column. On the illustrated selection menu screen G25, two columns in which five selection objects are arranged in the Z-axis direction (the left column arranged on the broken line M1 and the right column arranged on the broken line M2) are left and right (Y-axis direction). They are displayed side by side with a predetermined interval. In the left column, five selection objects of option A, option C, option E, option G, and option I are displayed side by side at predetermined intervals in the Z-axis direction. In the right column, five selection objects of option B, option D, option F, option H, and option J are displayed side by side at predetermined intervals in the Z-axis direction. In this selection menu screen G25, the selection target is switched for each column. For example, when the column on the left side (A) is the selection target, any one of the five selection objects of option A, option C, option E, option G, and option I can be selected. On the other hand, when the column on the right side (B) is the selection target, any one of the five selection objects of option B, option D, option F, option H, and option J can be selected.

For example, when the movement of the user's head is in the rotation direction (roll direction) about the X axis, the columns to be selected are switched. Specifically, when the movement of the user's head is rotated to the right, the column to be selected is switched from the left column to the column to the right, and when the movement of the user's head is rotated to the left, the column to be selected is selected. The column switches from the right column to the left column. If the movement of the user's head is rotated regardless of the direction of rotation of the movement of the user's head, the column to be selected is switched.

Further, the columns to be selected may be switched based on the information on the rotation direction and the rotation speed as well as the rotation direction. For example, if the rotation speed is greater than or equal to a predetermined value, the viewing direction rotates and the columns to be selected are switched, and if the rotation speed is less than the predetermined value, the columns to be selected are switched only by rotating the viewing direction. You may not have it.

FIG. 23 is a flowchart showing an example of the object arrangement process according to the present embodiment. First, upon receiving the trigger for displaying the selection menu, the control unit 110 arranges a plurality of selected objects in a plurality of columns in a predetermined direction (for example, in the Z-axis direction) in the virtual space (step S500). As a result, the display unit 12 displays a selection menu screen (see, for example, FIG. 22) including the field of view image and the plurality of selection objects in the left and right rows.

The control unit 110 detects the angular velocity based on the detection result of the sensor 13 (step S502). Then, the control unit 110 determines whether or not there is a change in the view direction (line-of-sight direction) (change in the rotation direction about the X-axis) based on the detected angular velocity (step S504).

When it is determined that there is a change in the rotation direction about the X axis (YES), the control unit 110 determines whether or not the change speed in the rotation direction is equal to or greater than a predetermined value (step S506). Then, when it is determined that the change speed in the rotation direction is equal to or greater than a predetermined value (YES), the control unit 110 switches the row of the selected objects to be determined (step S508). On the other hand, when it is determined that the change speed in the rotation direction is not equal to or more than a predetermined value (NO), the control unit 110 does not switch the column of the selected object to be determined.

Next, the control unit 110 determines whether or not the trigger for turning off the display of the selection menu has been accepted (step S510). If it is determined that the trigger for turning off the display of the selection menu is not accepted (NO), the control unit 110 returns the process to step S502. On the other hand, when it is determined that the trigger for turning off the display of the selection menu has been accepted (YES), the control unit 110 erases the selected objects arranged in the virtual space (step S512), and ends the object arrangement process. ..

[Summary of the third embodiment]
As described above, in the terminal device 10 according to the present embodiment, the object arranging unit 113 arranges a plurality of columns in which a plurality of selected objects are lined up in the virtual space. The determination unit 112 determines whether or not the selected object is selected for each column, and switches the determination column based on the detection result of the detection unit 111. As a result, when the terminal device 10 selects an option for each of a plurality of columns, the terminal device 10 only changes the direction of the head of the user while restricting the movement of the selected object in a predetermined direction based on the change in the viewing direction. , Since the column to be selected can be switched, the operability at the time of selecting the option can be improved.

For example, when the determination unit 112 determines that the field of view from the virtual viewpoint has changed in the rotation direction about the line-of-sight direction from the virtual viewpoint, the determination unit 112 switches the determination column according to the changed rotation direction. As a result, when the terminal device 10 selects an option for each of a plurality of columns, the user can switch the selected column to the column in the corresponding direction simply by changing the direction of the head to the roll direction. It is possible to improve the operability at the time of the selection operation.

The detection unit 111 detects the information regarding the rate of change in the direction as the information regarding the direction of the terminal device 10. When the detection unit 111 detects information on the rate of change equal to or higher than a predetermined value, the determination unit 112 switches the determination column. As a result, when the terminal device 10 selects an option for each of a plurality of columns, the user can switch the column to be selected by changing the direction of the head at a speed equal to or higher than a predetermined value. It is possible to improve the operability when performing the selection operation.

In addition, the determination unit 112 can be used both in the case where the field of view from the virtual viewpoint changes in the rotation direction about the line-of-sight direction from the virtual viewpoint and in the case where the detection unit 111 detects information on the change speed of a predetermined value or more. The column to be judged based on may be switched, or the column to be judged based on either one may be switched. For example, the determination unit 112 may switch the determination column to the right or to the right only when the field of view from the virtual viewpoint changes to the right in the rotation direction about the line-of-sight direction from the virtual viewpoint. When a change and a change rate of a predetermined value or more are detected, the determination column may be switched to the right side. Further, the determination unit 112 sets the determination column on the right side or the left side (for example, in the case of two columns, from one to the other, 3) when a change speed of a predetermined value or more is detected regardless of the change in the rotation direction. If it is more than a column, it may be switched to a preset but side column).

[Modification example]
Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configuration is not limited to the above-described embodiments, and includes designs and the like within a range that does not deviate from the gist of the present invention. For example, the configurations described in the first to third embodiments described above can be arbitrarily combined. For example, in the second and third embodiments, after the line-of-sight position P1 overlaps the determination region, the elapsed time is timed, the elapsed time gauge T1 is displayed, and the elapsed time is elapsed, as in the first embodiment. Whether or not to select is determined (determined) based on the time.

In the above embodiment, the example in which the arrangement direction of the selected objects is the Z-axis direction has been described, but the arrangement direction of the selected objects is not limited to the Z-axis direction, and may be, for example, the Y-axis direction. , The direction may have a Y-axis component and a Z-axis component. Further, when the selected objects are arranged in three dimensions, the arrangement direction of the selected objects may be a direction having an X-axis component, a Y-axis component, and a Z-axis component. Further, in the example shown in FIG. 22, a plurality of columns in which a plurality of selected objects are arranged in the Z-axis direction are arranged in the left-right (Y-axis direction), but a plurality of columns in which a plurality of selected objects are arranged in the Y-axis direction are arranged. Is an example in which the columns are arranged vertically (Z-axis direction), and the columns may be switched vertically (Z-axis direction).

In the above embodiment, the configuration of the HMD system 1 that can be used as an HMD by attaching the terminal device 10 to the attachment 2 has been described as an example, but the present invention is not limited to this, and for example, the terminal device 10 is an attachment. It may be a form that can be worn on the head by itself without requiring 2, that is, an HMD. Further, in the above embodiment, as an HMD (HMD system) in which the terminal device 10 is composed of at least a part thereof, a configuration example that can be attached to the user's head has been described, but the HMD can be completely attached to the head. It is not limited to the configuration of a different shape. For example, the terminal device 10 that can be used as an HMD maintains a predetermined positional relationship between the user's eyes and the display unit 12 by being held in front of the user regardless of the presence or absence of an attachment, and is a virtual space. It may be a device (for example, a shape like binoculars) that makes the visual field image and various objects visible inside.

Further, in the above embodiment, an example of a selection object as an option displayed on the selection menu screen has been described, but the present invention is not limited to this. For example, the selected object may be an object arranged as a character, an item, or the like having a determination area in the virtual space in the game executed by the terminal device 10. As an example, the process relating to the selected object described in the first embodiment may be applied to the process for a plurality of enemy objects (selected objects) flying in the virtual space of the game. For example, an attack is emitted based on the user's line-of-sight position, and when the irradiation of the enemy object (selected object) (the positional relationship where the judgment area and the line-of-sight position overlap) elapses for a predetermined time, the enemy object is shot down (selected object). It may be applied in a process such as (with selection).

Further, the program for realizing the function of the control unit 110 described above is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read by the computer system and executed to serve as the control unit 110. May be processed. Here, "loading and executing a program recorded on a recording medium into a computer system" includes installing the program in the computer system. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. Further, the "computer system" may include a plurality of computer devices connected via a network including a communication line such as the Internet, WAN, LAN, and a dedicated line. Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, or a storage device such as a hard disk built in a computer system. As described above, the recording medium in which the program is stored may be a non-transient recording medium such as a CD-ROM. The recording medium also includes an internal or external recording medium that can be accessed from the distribution server to distribute the program. The code of the program stored in the recording medium of the distribution server may be different from the code of the program in a format that can be executed by the terminal device. That is, the format stored in the distribution server does not matter as long as it can be downloaded from the distribution server and installed in a form that can be executed by the terminal device. The program may be divided into a plurality of parts, downloaded at different timings, and then combined by the terminal device, or the distribution server for distributing each of the divided programs may be different. Furthermore, a "computer-readable recording medium" is a volatile memory (RAM) inside a computer system that serves as a server or client when a program is transmitted via a network, and holds the program for a certain period of time. It shall also include things. Further, the above program may be for realizing a part of the above-mentioned functions. Further, it may be a so-called difference file (difference program) that can realize the above-mentioned function in combination with a program already recorded in the computer system.

Further, the one-line-of-sight position portion or all of the functions of the control unit 110 described above may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each of the above-mentioned functions may be made into a processor individually, or a part or all of them may be integrated into a processor. Further, the method of making an integrated circuit is not limited to the LSI, and may be realized by a dedicated circuit or a general-purpose processor. Further, when an integrated circuit technology that replaces an LSI appears due to advances in semiconductor technology, an integrated circuit based on this technology may be used.

[Additional Notes]
From the above description, the present invention can be grasped as follows, for example. Reference numerals in the accompanying drawings are added in parentheses for convenience in order to facilitate understanding of the present invention, but the present invention is not limited to the illustrated mode.

(Appendix 1) The terminal device (10) according to one aspect of the present invention is a head-mounted display system capable of displaying a view image representing a view from a virtual viewpoint in a virtual space as a stereoscopic image using binocular parallax. A terminal device that can be used as at least a part of the above, and has a detection unit (111, S112, S402, S502) that detects information about the direction of the terminal device, and a determination area that determines whether or not there is a selection from the user. Object placement units (113, S100, S200, S300, S400, S500) that place selected objects in the virtual space, and the viewing direction from the virtual viewpoint in the virtual space according to the detection result of the detection unit. The object arranging unit includes a display control unit (114, S100, S200, S300, S400, S500) for displaying the selected object arranged in the above on the display unit, and the object arranging unit is from the virtual viewpoint in the virtual space. The determination area of the selected object is changed based on the positional relationship between the line-of-sight position corresponding to the line-of-sight direction and the determination area of the selected object (S104).

According to the configuration of Appendix 1, the terminal device changes the size (apparent size) of the determination area according to the positional relationship between the line-of-sight position corresponding to the line-of-sight direction from the virtual viewpoint and the determination area of the selected object. Therefore, it is possible to improve the operability when performing a selection operation for an option.

(Appendix 2) Further, one aspect of the present invention is the terminal device according to Appendix 1, in which the object arranging unit has a positional relationship in which the line-of-sight position and the determination area of the selected object do not overlap in the line-of-sight direction. When the positional relationship overlaps with the above, the determination area of the selected object is widened.

According to the configuration of Appendix 2, the terminal device expands the determination area when the line-of-sight position corresponding to the line-of-sight direction from the virtual viewpoint enters the determination area of the selected object, so that the line-of-sight position is unintentionally determined by the user. It is possible to suppress the occurrence of erroneous operations that deviate from the area.

(Appendix 3) Further, one aspect of the present invention is the terminal device according to the appendix 2, and when the object arranging unit widens the determination area of the selected object, it can be used as a determination area of another selected object. Widen within a range that does not overlap in the line-of-sight direction.

According to the configuration of Appendix 3, when the user intentionally changes to the selection of another selected object, the terminal device deviates from the determination area where the line-of-sight positions overlap so that the selection can be made. The occurrence of erroneous operation can be suppressed. Further, since the terminal device can maintain the independence of the determination area of each selected object, it is possible to prevent an erroneous operation when it is desired to select any one of the selected objects.

(Appendix 4) Further, one aspect of the present invention is the terminal device according to the appendix 3, and when the object arranging unit widens the determination area of the selected object, it is combined with the determination area of the other selected object. , Widen within a range with a predetermined interval in the direction orthogonal to the line-of-sight direction.

According to the configuration of Appendix 4, when the user intentionally changes the selection to another selected object, the terminal device can immediately remove the line-of-sight position from the currently overlapping determination area.

(Supplementary Note 5) Further, one aspect of the present invention is the terminal device according to any one of Supplementary note 2 to Supplementary note 4, and the object arranging unit may be used when the determination area of the selected object is widened. The position of the selected object that has been placed is changed.

According to the configuration of Appendix 5, the terminal device can secure an area for expanding the determination area by moving other selected objects even when there are a large number of selected objects.

(Supplementary Note 6) Further, one aspect of the present invention is the terminal device according to any one of Supplementary note 2 to Supplementary note 5, and the object arranging unit is the selection when the determination area of the selection object is widened. Change the display mode of the object.

According to the configuration of Appendix 6, since the terminal device can display the information related to the selection decision on the expanding selection object on the selection object, it is possible to prevent the user from making an erroneous selection. Further, since the terminal device can change the color of the selected object being enlarged to a color different from that before the enlargement, the operability can be improved by improving the visibility of the selected object to be selected.

(Supplementary Note 7) Further, one aspect of the present invention is the terminal device according to any one of Supplementary note 2 to Supplementary note 6, wherein the object arranging unit has the line-of-sight position and the determination area of the selected object. After a predetermined time has elapsed from the positional relationship that does not overlap in the line-of-sight direction to the overlapping positional relationship, the determination area of the selected object or the display area corresponding to the determination area is widened.

According to the configuration of Appendix 7, the terminal device can prevent the determination area from expanding when the line-of-sight position passes over the determination area in a short time even though there is no intention to select, so that the selection operation is performed. The visibility at the time can be improved.

(Appendix 8) Further, one aspect of the present invention is the terminal device according to any one of Supplementary notes 1 to 7, wherein the object arranging unit determines the line-of-sight direction from the virtual viewpoint of the detecting unit. An object that is determined based on the detection result and makes the line-of-sight position visible is arranged in the virtual space.

According to the configuration of Appendix 8, the terminal device determines, for example, the line-of-sight position corresponding to the line-of-sight direction in the view image (for example, the center of the view image) according to the direction of the head of the user equipped with the HMD system. It is possible to improve the operability by making it easier for the user to recognize the positional relationship between the line-of-sight position and the selected object.

(Supplementary note 9) Further, one aspect of the present invention is the terminal device according to any one of Supplementary notes 1 to 8, based on the positional relationship between the line-of-sight position and the determination area of the selected object. A determination unit (112, S204 to S210, S304 to S310) for determining whether or not a selected object is selected is provided.

According to the configuration of Appendix 9, the terminal device can be an operation UI that allows the user to select an option by using the line of sight of the user in the HMD system.

(Appendix 10) Further, one aspect of the present invention is the terminal device according to the appendix 9, and the determination unit is based on a positional relationship in which the line-of-sight position and the determination area of the selected object do not overlap in the line-of-sight direction. The elapsed time from the overlapping positional relationship is measured, and the presence or absence of selection is determined based on the measured elapsed time (S204 to S210, S304 to S310).

According to the configuration of Appendix 10, the terminal device determines that the selection is made on the condition that a predetermined time elapses after the line-of-sight position overlaps the determination area of the selected object. It is possible to prevent selection by simply passing the line-of-sight position over the top.

(Appendix 11) Further, one aspect of the present invention is the terminal device according to Appendix 10, in which the object arranging unit arranges an object for making the elapsed time visible in the virtual space.

According to the configuration of Appendix 11, the terminal device can present the timing at which the selection is confirmed so that the user can easily recognize it.

(Appendix 12) Further, one aspect of the present invention is the terminal device according to the appendix 10 or the appendix 11, and in the determination unit, the line-of-sight position and the determination area of the selected object do not overlap in the line-of-sight direction. Timekeeping is performed differently depending on whether there is a positional relationship or an overlapping positional relationship (S314).

According to the configuration of Appendix 12, even if the line-of-sight position deviates from the judgment area due to an erroneous operation, if the line-of-sight position is moved to the same judgment area again, the time is not restarted from 0, but the time is elapsed from the continuation. Since the time timing is restarted, the operability can be improved.

(Supplementary Note 13) Further, one aspect of the present invention is the terminal device according to any one of Supplementary note 10 to Supplementary note 12, and in the determination unit, the line-of-sight position and the determination area of the selected object are the line-of-sight. The elapsed time measured after the positional relationship overlaps in the direction is reset when the line-of-sight position and the determination area of the other selected object overlap in the line-of-sight direction (S318).

According to the configuration of Appendix 13, when the line-of-sight position shifts to the determination area of another selected object, the terminal device considers that the user has no intention of continuing the selection and returns the timed elapsed time to 0. Therefore, even if the line-of-sight position is unintentionally moved to the same determination area, the measurement of the elapsed time is not restarted from the continuation, and it is possible to prevent an erroneous operation from occurring.

(Supplementary note 14) Further, one aspect of the present invention is the terminal device according to any one of Supplementary note 10 to Supplementary note 12, and in the determination unit, the line-of-sight position and the determination area of the selected object are the line-of-sight. The elapsed time measured after the positional relationship overlaps in the direction is reset when a predetermined time elapses when the line-of-sight position and the determination area of the selected object do not overlap in the line-of-sight direction (S316). ..

According to the configuration of Appendix 14, when the line-of-sight position deviates from the determination area and a predetermined time (for example, 10 seconds) elapses, the terminal device considers that the user has no intention of continuing the selection and clocks the time. Since the elapsed time is returned to 0, even if the line-of-sight position is unintentionally moved to the same determination area, the measurement of the elapsed time is not restarted from the continuation, and it is possible to prevent an erroneous operation from occurring. ..

(Supplementary Note 15) Further, one aspect of the present invention is the terminal device according to any one of Supplementary note 10 to Supplementary note 12, wherein the determination unit has a determination area in which the line-of-sight position is the selection object in the line-of-sight direction. The speed to be measured is changed depending on which position in the position it overlaps with.

According to the configuration of Appendix 15, the terminal device can also provide the user with a method of shortening the period until the determination is made while providing the determination time for preventing the malfunction.

(Supplementary note 16) Further, one aspect of the present invention is the terminal device according to any one of Supplementary notes 1 to 15, and the display control unit is in the virtual space according to the detection result of the detection unit. The field of view image in the field of view from the virtual viewpoint is displayed on the display unit.

According to the configuration of Appendix 16, the terminal device can provide a visual field image in the visual field direction in the virtual space so that the user can see it according to the movement of the head of the user who is equipped with the HDM system. Further, since the terminal device can simultaneously display the field of view image in the virtual space and the selected object, the user can select while referring to the game information in the virtual space.

(Appendix 17) Further, the program according to one aspect of the present invention is a program for causing a computer to function as a terminal device according to any one of Supplementary notes 1 to 16.

According to the configuration of Appendix 17, the program changes the size (apparent size) of the determination area according to the positional relationship between the line-of-sight position corresponding to the line-of-sight direction from the virtual viewpoint and the determination area of the selected object. Therefore, it is possible to improve the operability when performing a selection operation for an option.

1 HMD system, 2 attachment, 3R right eye lens, 3L left eye lens, 5 strap, 10 terminal device, 12 display unit, 13 sensor, 14 timer, 15 storage unit, 16 communication unit, 17 CPU, 110 control unit, 111 Detection unit, 112 judgment unit, 113 object placement unit, 114 display control unit

Claims (4)

A terminal device that can be used as at least a part of a head-mounted display system capable of displaying a visual field image representing a visual field from a virtual viewpoint in a virtual space as a stereoscopic image using binocular parallax.
A detection unit that detects information about the direction of the terminal device, and
An object arranging unit for arranging a selected object having a determination area for determining whether or not a user has selected the object in the virtual space, and an object arranging unit.
A display control unit that displays the selected object arranged in the visual field direction from the virtual viewpoint in the virtual space on the display unit according to the detection result of the detection unit.
With
The determination time as a threshold value for determining whether or not the selected object is selected is set in advance.
The object arrangement part is
When the moving speed of the line-of-sight position when the line-of- sight position corresponding to the line-of-sight direction from the virtual viewpoint and the determination area of the selected object change from a positional relationship that does not overlap in the line-of-sight direction to an overlapping position is equal to or higher than a predetermined value. the changes the determination area before Symbol selected objects from getting to the positional relationship after the lapse of a predetermined time less than the judging time, when the moving speed of the line of sight position is smaller than the predetermined value, the The determination area of the selected object is changed even if the predetermined time has not elapsed since the positional relationship was established .
Terminal equipment. The object arrangement part is
When changing the determination area of the selected object, the determination area of the selected object is widened.
The terminal device according to claim 1. The object arrangement part is
The line-of-sight direction from the virtual viewpoint is determined based on the detection result of the detection unit.
An object for making the line-of-sight position visible is arranged in the virtual space.
The terminal device according to claim 1 or 2. A program for causing a computer to function as the terminal device according to any one of claims 1 to 3.

Images