JP6133344B2 - User interface display method and program for display on head mounted display - Google Patents

Description

  The present invention provides a head-mounted display system that is mounted on a user's head, displays an image in a virtual space within the user's visual field, and can operate a display object displayed on the image by an operation with a touch panel device. The present invention relates to a method and a program for displaying an object.

  There is known a non-transmissive head mounted display (HMD) that is mounted on a user's head and can present an image in a virtual space to the user with a display or the like placed in front of the user. In recent years, portable terminals in which a display and a touch panel are integrated are known. Various techniques for intuitively realizing the operation of a display object displayed on a display of a mobile terminal using such a touch panel of the mobile terminal have been proposed. (For example, Patent Document 1)

JP 2014-149653 A

When the non-transmissive head-mounted display according to the background art is mounted, the user cannot observe the hand because the user is completely immersed in the virtual space.
Moreover, the technique described in the said patent document 1 operates the display object displayed on this display using the portable terminal whose display is a touch panel type. In this case, the user operates the display object using a touch panel integrated with the display while looking at the display of the terminal being held. However, since the display is integrated with the touch panel, a part of the display may be hidden by a user's finger or the like with respect to the touch panel. Further, when the non-transparent HMD is mounted, the user cannot operate the display object on the touch panel integrated with the display while observing the display.

(1) The present invention provides a user interface for displaying a display object in a display area of a head mounted display in a head mounted display system including a head mounted display and a touch panel device connected to the head mounted display in a communicable manner. Display method,
Detecting a contact displacement amount from a contact start position by a user on the touch panel of the touch panel device;
Detecting a change in tilt of the touch panel device;
And a step of controlling display of a display object in the virtual space displayed in the display area based on the detected contact displacement amount from the operation start position and the change in the tilt of the touch panel device. The present invention relates to a user interface display method.
(2) Further, according to the present invention, the step of controlling the display includes
Displaying the orientation of the first object displayed in the display area based on the detected amount of contact displacement from the contact start position with respect to the touch panel;
Displaying the orientation of the second object displayed in the display area based on the detected tilt change of the touch panel device; and
It is characterized by providing.
(3) The present invention also provides:
Detecting the amount of contact displacement further includes detecting a direction of contact displacement from a contact start position by a user on the touch panel;
The display control step is
Based on the detected contact displacement amount and contact displacement direction from the contact start position with respect to the touch panel, the display control of the movement amount and the traveling direction of the first object displayed in the display area;
Based on the detected tilt change of the touch panel device, the display control of the tilt of the first object displayed in the display area of the head mounted display;
It is characterized by providing.
(4) Further, according to the present invention, in the above (3), the position of the first object displayed in the display area in the virtual space is a three-dimensional virtual position obtained by combining the contact displacement amount and the tilt change in the contact displacement direction. It is a position converted into coordinates in the acquisition space.
(5) Further, the present invention displays a display object in a display area of a head mounted display in a head mounted display system including a head mounted display and a touch panel device connected to the head mounted display in a communicable manner. A user interface display method,
Detecting a contact displacement amount from a contact start position by a user on the touch panel of the touch panel device;
Detecting a device displacement amount in a real space of the touch panel device;
And a step of controlling display of a display object in the virtual space displayed in the display area based on the detected contact displacement amount from the operation start position and the device displacement amount of the touch panel device. The present invention relates to a user interface display method.
(6) In the present invention, the display control step includes
A step of controlling display of the amount of movement of the first object displayed in the display area based on the detected amount of contact displacement from the operation start position on the touch panel;
Based on the detected device displacement amount in the real space of the touch panel device, the display control of the operation of the first object displayed in the display area;
It is characterized by providing.
(7) In the above (6), the position of the first object displayed in the display area in the virtual space is the position obtained by combining the device displacement amount with the contact displacement amount in the three-dimensional virtual space. It is a position converted into coordinates.
(8) Further, according to the present invention, the step of controlling the display includes
Based on the detected device displacement amount in the real space of the touch panel device and the swipe rotation speed with respect to the touch panel, a parameter related to the first object displayed in the display area of the head mounted display is calculated, and the calculated parameter is Controlling display of the display object based on,
It is characterized by providing.
(9) Further, the present invention is characterized in that the swipe rotational speed is calculated based on a contact displacement amount and a contact displacement direction from an operation start position on the touch panel.
(10) Moreover, this invention relates to the user interface display method as described in said (1) to (9) whose touchscreen apparatus is a smart phone.
(11) Further, according to any of the above (1) to (10), the present invention causes a processor mounted on a head mounted display system to execute a user interface display method for displaying a display object in a display area. The interface program.

  These and other features and advantages of the present invention will become apparent from the following more specific description of the embodiments of the invention, the accompanying drawings, and the claims.

  According to the present invention, it is possible to easily and intuitively specify a display object in a three-dimensional virtual space displayed in a display area of an HMD that is communicably connected to a touch panel device using the touch panel device. it can.

1 shows a schematic diagram of an HMD system, according to one embodiment of the present invention. 1 shows a functional configuration for realizing display processing of an HMD system according to an embodiment of the present invention. 3 shows an orthogonal coordinate system of a three-dimensional real space of a touch panel device, an orthogonal coordinate system of a touch panel plane, and various parameters used in these coordinate systems according to an embodiment of the present invention. (A) an operation start state of a spacecraft and a cannon when operating a spacecraft and a cannon in a three-dimensional virtual space according to an embodiment of the present invention; (b) an operation start state for a touch panel device; C) A schematic view of a cross section of the touch panel device shown in (b) cut along a broken line a is shown. (B) a state during operation of the spacecraft and cannon, and (b) a state during operation of the touch panel device when operating the spacecraft and cannon in the three-dimensional virtual space according to an embodiment of the present invention. (C) The schematic of the cross section which cut | disconnected the touchscreen type apparatus shown by (b) with the broken line a is shown. In a three-dimensional virtual space displayed in a display area that can be observed by a user wearing an HMD when operating a spacecraft and a cannon in the three-dimensional virtual space according to an embodiment of the present invention The image of is shown. In the case of operating an airplane in a three-dimensional virtual space according to an embodiment of the present invention, (a) the operation start state of the airplane, (b) the operation start state for the touch panel device, and (c) (b) The schematic of the cross section which cut | disconnected the touchscreen type apparatus shown by the broken line a is shown. In the case of operating an airplane in a three-dimensional virtual space according to an embodiment of the present invention, (a) a state during operation of the airplane, (b) a state during operation on the touch panel device, and (c) (b) The schematic of the cross section which cut | disconnected the touchscreen type | formula apparatus shown by the broken line a is shown. In the case of operating a dragon in a three-dimensional virtual space according to an embodiment of the present invention, (i) the operation start state of the dragon, (b) the operation start state for the touch panel device, (c) The schematic of the cross section which cut | disconnected the touchscreen type apparatus shown by the broken line a is shown. According to an embodiment of the present invention, when operating a dragon in a three-dimensional virtual space, (a) a state during operation of the dragon, (b) a state during operation on the touch panel device, (c) (b) The schematic of the cross section which cut | disconnected the touchscreen type | formula apparatus shown by the broken line a is shown. In the case of operating a fishing rod in a three-dimensional virtual space according to an embodiment of the present invention, (i) an operation start state of the fishing rod, (b) an operation start state for a touch panel device, and (c) (b) The schematic of the cross section which cut | disconnected the touchscreen type apparatus shown by the broken line a is shown. In the case of operating a fishing rod in a three-dimensional virtual space according to an embodiment of the present invention, (a) a state during operation of the fishing rod, (b) a state during operation on the touch panel device, (c) (b) The schematic of the cross section which cut | disconnected the touchscreen type | formula apparatus shown by the broken line a is shown. 4 shows a process flow for controlling display in an HMD system according to an embodiment of the present invention.

Hereinafter, this embodiment will be described. FIG. 1 is a configuration diagram of a system according to an embodiment. In this system, the touch panel type device 130 and the main body device 120 are connected so as to be communicable wirelessly or by wire, and the main body device 120 and the HMD 110 are communicably connected by wireless or wire. The HMD 110 includes a display area 112, and a user wearing the HMD can observe only an image displayed in the display area 112.
<Hardware configuration of main unit>
The main device 120 can include a processor, a memory, and a communication unit (not shown). Furthermore, the main body device 120 can be detachably inserted with an information storage medium (for example, an optical disk) that is used interchangeably with the main body device 120. An application program executed by the main device can be stored on the optical disc.

  The processor of the main device 120 executes application processing (game processing) by executing an application program stored in a memory or an optical disc.

  The memory of the main device 120 can store various application programs executed by the processor and various data (game result data or intermediate data, etc.) generated by the execution of the various application programs. For example, a game application program for a spaceship equipped with a cannon to attack enemy aircraft can be stored.

  The communication unit of the main body apparatus can receive various data output from the touch panel device 130 and can transmit the generated display image to the display area 112 of the HMD 110.

The main unit 120 can be realized by a known personal computer, a stationary game machine, or the like. The main device 120 is a device for realizing the functions shown in FIG. In FIG. 1, the main body device 120 is configured separately from the touch panel device 130 and the HMD 110, but in order to realize some or all of the functions shown in FIG. 2 realized by the main body device 120. It can be mounted on the touch panel device 130 or the HMD 110 provided with the required hardware. In this case, the main body device 120 can be integrated with the touch panel device 130 and / or the HMD 110.
<Hardware configuration of touch panel device>
The touch panel device 130 can include a touch sensor 132 disposed on the touch panel, a tilt detection sensor 134, a triaxial acceleration sensor 136, a transmission unit, a memory), and a processor (not shown). The touch panel device 130 is held by a user and can be used as an HMD controller. The touch panel device 130 can be realized by a known smartphone, tablet terminal, or PDA.

  The touch sensor 132 of the touch panel device detects a position (contact position) where a touch input by the user to the touch panel is performed. When the user performs a touch operation on the touch panel of the touch panel device using a finger or a stylus pen, the touch sensor 132 can detect various data related to the operation (contact position on the touch panel). The contact displacement amount calculation unit 204 uses the detected contact position information to perform a continuous contact displacement amount (contact displacement amount) from the touch start position on the touch panel when performing a swipe operation. The contact displacement direction from the start position, the contact displacement speed, the number of contacts, the swipe rotation number, and the like can be calculated. When the non-transparent HMD is mounted, the user cannot observe the display surface of the touch panel device 130 that is held. Instead, the display area of the HMD can be observed. Accordingly, the touch panel only needs to have the touch sensor 132 function, and may not have the display function. In the present embodiment, the swipe operation means an operation of touching the touch panel with a user finger or a touch pen and moving the touch position on the touch panel so as to draw a line while touching.

  The tilt detection sensor 134 of the touch panel device is for detecting respective angular velocities around the XYZ axes in the three-dimensional real space of the touch panel device 130, and can be realized by a gyro sensor.

  The three-axis acceleration sensor 136 of the touch panel device is for detecting acceleration in the XYZ axis directions in the three-dimensional real space of the touch panel device. The acceleration detected by the acceleration sensor 136 changes according to the inclination (tilt angle) and movement of the casing of the touch panel device 130, and therefore, using the acquired acceleration data, the inclination of the casing of the touch panel device 120 and the like. It is possible to calculate the movement amount of the apparatus in the three-dimensional real space. For example, by monitoring the acceleration output from the three-axis acceleration sensor, the posture and posture change of the touch panel type device, for example, the posture in which one end of the casing of the touch panel type device 130 is lifted on the basis of the gravitational acceleration. When there is a change, its direction and degree can be detected.

  The memory of the touch panel type device stores operation input information related to a predetermined operation performed on the touch panel type device output from each sensor, that is, the touch position to the touch sensor, the three-dimensional real space of the touch panel type device 130. The angular velocity around the XYZ axis of the casing and the acceleration in the XYZ axis direction of the casing in the three-dimensional real space are stored as operation input information.

  The transmission unit of the touch panel device outputs operation input information related to user operations performed on the touch panel device stored in the memory of the touch panel device in response to a request from the application progression unit 202 of the main device 120. It can be sent as data. When the application progression unit 202 requests operation input information necessary for executing the application program, the touch panel device 130 outputs only necessary information without transmitting all detected operation input information to the main body device 120. It can be sent as data. Therefore, the communication load between the main body device 120 and the touch panel device 130 can be reduced.

  FIG. 2 is a functional configuration diagram for realizing display processing of the HMD system according to the embodiment of the present invention. The main device 120 includes an application progression unit 202, a contact displacement amount calculation unit 204, an inclination change calculation unit 206, and a device displacement amount calculation unit 208, which are controlled by a program incorporated in a calculation unit such as a processor. Yes.

  First, the application progression unit 202 determines what user operation is requested to be input by the application program based on the contents of the application program being executed by the main device 120. In addition, the application progression unit 202 determines a period (predetermined period) for acquiring output data based on the contents of the application program being executed. The required user operation is determined based on the contents of the application program being executed, the situation that the user is currently operating in the application program being executed, and the user operation on the controller that allows the input of the application program being executed. That's it. For example, in a state where a user operates a spaceship in a virtual space and an operation state is in battle with an enemy, a required user operation corresponding to the operation state is, for example, “operating the traveling direction of the spacecraft”, “Control the direction of the cannon”. Thus, by determining the user operation required based on the contents of the application program, it is possible to eliminate an operation that is not intended by the user during execution of the application. The user may unintentionally perform a posture operation or a movement operation of the touch panel type device. In particular, a user wearing an HMD cannot observe a touch panel device held by the user, and may perform an unintended operation.

  Next, the application progression unit 202 requests operation input information corresponding to information about the requested user operation as output data to the touch panel type device 130 and sets a period (predetermined period) for acquiring the output data to the touch panel type device 130. Send to. For example, when the requested user operation is “manipulate the traveling direction of the spacecraft” or “manipulate the direction of the cannon”, the operation input information corresponding to the requested user operation is the user's touch from the touch sensor 132. Information on operation (contact position), information on change in tilt of touch panel device 130 from tilt detection sensor 134 (roll angular velocity). The application progression unit 202 may not request operation input information that does not correspond to a required user operation, for example, pitch angular velocity, yaw angular velocity of the touch panel device, and acceleration in the three-dimensional real space of the touch panel device 130. By requesting only necessary information, the communication load between the main device 120 and the touch panel device 130 can be reduced.

  Next, the transmission unit of the touch panel device 130 was obtained from the touch sensor, the tilt detection sensor, and the triaxial acceleration sensor stored in the memory for a predetermined period in response to a request from the application progress unit 202. Of the information related to various operation input information, information corresponding to the requested user operation is sequentially output to the main unit 120 as output data.

  When the output data is information related to the touch operation on the touch panel of the user, the contact displacement amount calculation unit 204 is based on the information related to the touch operation, and the contact displacement amount from the contact start position of the continuous touch on the touch panel. The contact displacement direction from the contact start position, the contact displacement speed, the number of contacts, the number of swipe rotations, the contact time, etc., or a part of these are calculated as operation data. The contact displacement amount calculation unit 204 repeatedly executes calculation of these operation data while continuous input is performed on the touch panel over a predetermined period.

  When the output data is information (angular velocity) regarding the tilt operation in the three-dimensional real space for the user's touch panel device 130, the tilt change calculation unit 206, based on the information regarding the tilt operation, outputs the housing of the touch panel device 130. A change in angle about three axes in a three-dimensional real space, an angular acceleration, or a part thereof is calculated as operation data. The inclination change calculation unit 206 repeatedly calculates these operation data over a predetermined period.

  When the output data is information relating to a displacement operation in the three-dimensional real space for the touch panel device 130, the device displacement amount calculation unit 208 is based on the information (acceleration) relating to the displacement operation, and the three-dimensional real space of the touch panel device 130. The apparatus speed in the XYZ axis direction, the apparatus displacement amount, etc., or a part of them are calculated as operation data. The apparatus displacement amount calculation unit 208 repeatedly executes calculation of these operation data over a predetermined period.

  The display control unit 210 associates the operation to be performed on the display object with the operation data based on the user operation that is requested to be input by the application program determined by the application progression unit 202. Then, the display control unit 210 performs display calculation of the display object based on operation data associated with the operation of the display object, thereby generating an image of the display object for each frame. For example, when the requested user operation determined by the application progression unit 202 is “manipulate the traveling direction of the spacecraft” or “manipulate the direction of the cannon”, the display objects “spacecraft”, “cannon” ”For each change in the direction of travel of the“ spacecraft ”based on a part or all of“ roll angle change and roll angular velocity in the three-dimensional real space of the touch panel device 130 ”which is the operation data. Then, a change in the direction of the “cannon” is calculated based on a part or all of the “contact displacement amount, contact displacement direction, contact displacement speed with respect to the touch panel” which is the operation data. The display control unit 210 generates an image of the display object based on the display object display calculation, and outputs the generated image to the display area 112. When generating a so-called three-dimensional image, a display parameter including data of each vertex of the display object (position coordinates of the vertex in the world coordinate system, color data) is calculated, and based on the calculated display parameter, The image of the display object is sequentially generated for each frame.

  According to the present invention, a user can easily and intuitively instruct an operation to be performed on a display object in a three-dimensional virtual space using a touch panel device. In particular, a user wearing a non-transparent HMD cannot observe his / her hand, and thus cannot instruct the display object while observing the touch panel of the touch panel device. Furthermore, when the instruction is made using only the touch sensor of the touch panel, only the two-dimensional position on the screen can be designated, and the direction and speed in the three-dimensional virtual space cannot be designated. According to the present invention, using a touch panel type device including a gyro sensor, an acceleration sensor, and a touch sensor, the direction of the display object in the three-dimensional virtual space displayed in the display area of the HMD without the user observing the hand. Can be easily and intuitively indicated.

  In addition, when an instruction is given to a display object displayed on the touch panel using the touch sensor of the touch panel, the display object may be hidden by a finger or the like that is operated and may not be visible. According to the present invention, the touch panel for performing the touch operation and the display area are configured separately. Therefore, the user can operate the display object without worrying about the display object being hidden by the operation.

  FIG. 3 shows an orthogonal coordinate system (X, Y, Z) in the three-dimensional real space of the casing of the touch panel device 130 and the two-dimensional orthogonal of the touch panel plane of the touch device according to an embodiment of the present invention. A coordinate system (x, y) and various parameters used in these coordinate systems are shown. θr, θp, and θy are roll angles (θr) indicating rotation about the front and rear of the touch panel device in the orthogonal coordinate system (X, Y, Z) in the three-dimensional real space of the housing of the touch panel device. , A pitch angle (θp) indicating rotation about the left and right axes, and a yaw angle (θy) indicating rotation about the upper and lower axes. P0 (xp0, yp0) and P1 (xp1, yp1) respectively indicate coordinate values on the two-dimensional coordinate system (x, y) of the touch panel plane, and P0 (xp0, yp0) is a contact start position by a user's touch operation. , P1 (xp1, yp1) indicates a contact end position by a touch operation.

Next, processing for operating a display object displayed in the three-dimensional virtual space displayed in the display area of the HMD by operating the touch panel device 130 will be described with reference to the drawings.
Regarding the operation of the display object displayed in the display area, the following four embodiments according to the contents set by the application,
First Embodiment Two display objects are separately operated using a touch panel and a gyro sensor.
Second Embodiment A tilt and a traveling direction of one display object are separately operated using a touch panel and a gyro sensor.
<Third Embodiment> The operation and the traveling direction of one display object are separately operated using a touch panel and an acceleration sensor.
<Fourth Embodiment> One operation related to one display object is operated using a touch panel and an acceleration sensor.
It is assumed that one of them is selected for operation.
<First Embodiment>
First, an overview of operations in the first embodiment will be described with reference to FIG. The first embodiment relates to a game in which the direction of two display objects (spacecraft, cannon) is manipulated in a three-dimensional virtual space by a user operation on a touch panel device. The user can operate the two display objects separately in the three-dimensional virtual space by performing a swipe operation on the touch panel 132 of the touch panel device 130 and operating the posture of the housing of the touch panel device 130. 4A and 4B show the state transition (a) in the three-dimensional virtual space when operating the spacecraft 402 and the cannon 404 according to the first embodiment, and the state transition (b, c) of the touch panel device. FIG. FIG. 4C illustrates a three-dimensional virtual image displayed in a display area that can be observed by a user wearing the HMD when operating the spacecraft 302 and the cannon 304 in the three-dimensional virtual space according to the first embodiment. It is a figure which illustrates the image in space.

  First, in the application progression unit 202, based on the contents of the application program being executed in the main device 120, user operations that are requested by the application program are “manipulate the traveling direction (or direction) of the spacecraft” and “ To control the direction of the cannon. The user can instruct the traveling direction (yaw angle) of the spacecraft 402 displayed in the display area 112 of the HMD by changing the tilt (roll angle) of the housing of the touch panel device 130. By swiping on the touch panel 132 of the apparatus, the direction (yaw angle) of the cannon mounted on the spacecraft 402 displayed in the display area 112 of the HMD can be instructed.

  Next, in response to a request from the application progression unit 202, the touch panel device 130 outputs, as output data, information corresponding to a user operation that requires input from among the operation input information stored in the memory of the touch panel device 130. The data is sequentially output to the main device 120 over a predetermined period. In the present embodiment, the output data includes information on the touch operation of the user from the touch sensor 132 (contact position), information on the tilt operation of the touch panel device 130 from the tilt detection sensor 134 (angular velocity about the front-rear direction (roll Angular velocity)).

  Next, the transmission unit of the touch panel device 130 sequentially outputs information corresponding to a user operation obtained over a predetermined period to the main body device 120 as output data. And the contact displacement amount calculation part 204 calculates the contact displacement amount from a contact start position, etc. as operation data based on the information regarding a user's touch operation. Further, the tilt change calculation unit 206 calculates, as operation data, a roll angle change around the Y axis in the three-dimensional real space of the touch panel device 130 based on information (angular velocity) related to the tilt operation of the touch panel device 130.

  Next, the display control unit 210 associates the operation to be performed on the display object with the operation data based on the user operation that is requested by the application program determined by the application progression unit 202. Then, the display control unit 210 performs display calculation of the display object based on operation data (contact displacement amount, roll angle change, etc.) associated with the operation of the display object. To generate.

  FIG. 4A is a diagram illustrating an operation start state of the spacecraft 402 and the cannon 404 operated in the three-dimensional virtual space. The image shown in FIG. 4A (A) can also be displayed in the display area 112 of the HMD. FIG. 4A (a) shows a top view of the operation start state of the spacecraft 402 and the cannon 404 in the three-dimensional virtual space. The spacecraft 402 is ψy0 in the three-dimensional virtual space, and the cannon 404 is ψ It has an operation start angle (yaw angle) corresponding to 'y0. The operation start state of the touch panel device shown in FIG. 4A (B) corresponds to the operation start state of the spacecraft 402 and cannon 404 in the three-dimensional virtual space shown in FIG. 4A (A). In FIG. 4A (B), P0 (xp0, yp0) indicates the contact start position (black circle) on the touch panel plane by the user. θr0 represents an operation start angle (roll angle) about the front-rear direction of the touch panel device held by the user. FIG. 4A (C) shows a schematic diagram of a cross section of the touch panel device obtained by cutting the touch panel device shown in FIG. 4A (B) along a broken line a. In FIG. 4A (B), the operation start position is represented by a black circle, but this black circle may not actually be displayed.

  FIG. 4B is a diagram illustrating a state during the operation of the spacecraft 402 and the cannon 404 operated in the three-dimensional virtual space. The traveling direction of the spacecraft 402 is determined based on the rotation angle (roll angle) about the front-rear direction of the touch panel device, and the direction of the cannon 404 is determined based on the contact displacement amount from the contact start position with respect to the touch panel. The That is, the direction of the cannon 404 is calculated based on a line (input locus 406) input to the touch panel of the touch panel device.

  FIG. 4B (a) shows a top view of the spacecraft 402 and the cannon 404 in operation in the three-dimensional virtual space, and the spacecraft 402 operates in the three-dimensional virtual space corresponding to Δψy. The traveling direction changes counterclockwise by an angle, and the cannon 404 changes its inclination clockwise by an operation angle corresponding to Δψ′y. The state during operation of the spacecraft 402 and the cannon 404 in the three-dimensional virtual space shown in FIG. 4B (a) corresponds to the state during operation of the touch panel device shown in FIG. 4B (b).

  FIG. 4B (b) shows an input locus 406 when a swipe operation is performed from the touch start position P0 (xp0, yp0) to the touch end position P1 (xp1, yp1) on the touch panel of the touch panel device, and an operation start angle in the real space. A touch panel device tilted counterclockwise by θr0 to Δθr is shown. As shown in FIG. 4B (B), the user operates the direction of the cannon by performing an operation of drawing a line on the touch panel. In FIG. 4B (b), the input trajectory 406 when performing the swipe operation is drawn as a straight line, but the input trajectory may be a curve or a round-trip line. In FIG. 4B (b), the tilting operation on the touch panel type device 130 is tilted counterclockwise by Δθr. However, the tilt can be continuously tilted clockwise within a predetermined period. Furthermore, within the predetermined period, even if the touch operation is once ended at the contact end position P1, it is possible to newly set the contact start position P0 and perform the swipe operation. Further, the display control unit 210 can change the changing speed of the direction of the cannon 404 based on the displacement speed of contact with the touch panel. Further, the display control unit 210 can change the speed of change in the traveling direction of the spacecraft based on the angle change and the angular speed of the touch panel device. In FIG. 4B (b), the operation start position is indicated by a dotted line circle, the operation end position is indicated by a black circle, and a line drawn by the user is indicated by a dotted arrow, but these may not be displayed in practice. In addition, the dotted arrows shown in FIG. 4B (B) and the black arrows shown in (B) and (C) correspond to the dotted arrows and black arrows shown in FIG. 4B (A), respectively. Yes.

FIG. 4B (C) shows a schematic view of a cross section of the touch panel device obtained by cutting the touch panel device shown in FIG. 4B (B) along a broken line a.
A user wearing the non-transmissive HMD cannot operate the touch panel while observing the touch panel. In the present embodiment, the user can operate the display object displayed in the display area of the HMD with an intuitive operation such as tilting the touch panel device or performing a touch operation on the touch panel. The operability is very good for the user who is.

  In addition, it may be difficult for such a user to perform an operation of drawing a straight line on the touch panel in a certain direction (for example, just beside). In the present embodiment, for example, the direction of the cannon can be manipulated by focusing only on the contact displacement in the x-axis direction without considering the contact displacement in the y-axis direction from the contact start position P0 to the contact end position P1.

  4B (A) shows the operation end state of the spacecraft 402 and the cannon 404 when the contact end position is (xp1, yp1) and the operation end angle is θr1 in FIG. 4B (B). The contact end position can be (xp1, yp1) and the operation end angle can be left as θr0. This is because the touch operation on the touch panel and the posture operation of the touch panel device can be performed independently. In this case, the operating angle of the cannon 404 changes from ψ′y0 by Δψ′y, but the traveling direction of the spacecraft 402 remains ψy0.

FIG. 4C shows an example in the three-dimensional virtual space displayed in a display area that can be observed by the user wearing the HMD when operating the spacecraft 402 and the cannon 404 in the three-dimensional virtual space. It is an image. The user can observe an image as if he were in the command room of the spacecraft 402.
<Second Embodiment>
Next, an overview of operations in the second embodiment will be described with reference to FIG. The second embodiment relates to a game for manipulating the traveling direction and inclination of one display object (airplane) in a three-dimensional virtual space by a user operation on a touch panel device. The user swipes the touch panel 132 of the touch panel type device 130 and operates the posture of the casing of the touch panel type device 130 to separately operate the tilt and the traveling direction of one display object in the three-dimensional virtual space. be able to. FIG. 5A and FIG. 5B are diagrams exemplifying state transitions (a) in the three-dimensional virtual space and state transitions (b, c) of the touch panel device when operating the airplane 502 according to the second embodiment. It is.

  First, the application advancing unit 202 determines that the user operation requested by the application program based on the contents of the application program being executed on the main body device 120 is “manipulate the advancing direction and movement amount of the airplane” and “ To “manipulate the tilt”. At this time, the user can instruct the inclination (pitch angle) of the airplane displayed in the display area 112 of the HMD by changing the inclination (pitch angle) of the housing of the touch panel device 130, and the touch panel 132. On the other hand, as shown in FIG. 5B (B), by swiping to draw a line, it is possible to instruct the traveling direction and movement amount of the airplane 502 displayed in the display area 112 of the HMD. By instructing the pitch angle, traveling direction, and amount of movement of the airplane, the user can take off and land the airplane in the virtual space and perform flight operations.

  Next, in response to a request from the application progression unit 202, the touch panel device 130 outputs, as output data, information corresponding to a user operation that requires input from among the operation input information stored in the memory of the touch panel device 130. The data is sequentially output to the main device 120 over a predetermined period. In the present embodiment, the output data includes information on the touch operation of the user from the touch sensor 132 (contact position), information on the tilt operation of the touch panel device 130 from the tilt detection sensor 134 (angular velocity (pitch Angular velocity)).

  FIG. 5A is a diagram illustrating an operation start state of the airplane 502 operated in the three-dimensional virtual space. The image shown in FIG. 5A (a) can also be displayed in the display area 112 of the HMD. FIG. 5A (a) shows a side view of the operation start state of the airplane 502 in the three-dimensional virtual space, and the airplane 502 corresponds to (xv0, yv0, zv0) in the three-dimensional virtual space. It has an operation start position and an operation start angle (pitch angle) corresponding to ψp0. The operation start state of the touch panel device shown in FIG. 5A (B) corresponds to the operation start state of the airplane 502 in the three-dimensional virtual space shown in FIG. 5A (A). In FIG. 5A (B), P0 (xp0, yp0) indicates a contact start position (black circle) on the touch panel plane by the user. θp0 indicates an operation start angle (pitch angle) about the horizontal direction of the touch panel device held by the user.

  FIG. 5A (c) shows a schematic view of a cross section of the touch panel device obtained by cutting the touch panel device shown in FIG. 5A (b) along a broken line a. In FIG. 5A (B), the operation start position is indicated by a black circle, but this black circle may not actually be displayed.

  FIG. 5B is a diagram illustrating a state during operation of the airplane 502 operated in the three-dimensional virtual space. The inclination of the airplane 502 with respect to the left-right direction is determined based on the rotation angle (pitch angle) about the left-right direction of the touch panel device, and the traveling direction and movement amount of the airplane 502 are determined from the touch start position on the touch panel. It is determined based on the contact displacement direction and the contact displacement amount. Here, the traveling direction of the airplane 502 is calculated based on a line (input locus 506) input to the touch panel of the touch panel device.

  FIG. 5B (a) shows a side view of the state after the operation of the airplane 502 in the three-dimensional virtual space, and the airplane 502 is shown as (xv1, yv1, zv1) in the three-dimensional virtual space. It has a corresponding position during operation, and the inclination changes by an angle corresponding to Δψp. Here, the operation end position (xv1, yv1, zv1) in the three-dimensional virtual space is the contact displacement amount and displacement direction (P0 →) from the operation start position with respect to the touch panel with respect to the operation start position (xv0, yv0, zv0). In addition to P1), a position obtained by adding a change in inclination (θp0 → θp1) about the horizontal direction of the touch panel device is converted into coordinates in the three-dimensional virtual space. Therefore, the user can control the display object in more detail using the touch panel device. In addition, without using the contact displacement direction from the operation start position on the touch panel, only the contact displacement amount and the inclination change can be used to convert the coordinates into the three-dimensional virtual space. The state during operation of the airplane 502 in the three-dimensional virtual space shown in FIG. 5B (a) corresponds to the state during operation of the touch panel device shown in FIG. 4B (b).

  FIG. 5B (b) shows an input trajectory when a swipe operation is performed from the contact start position P0 (xp0, yp0) to the contact end position P1 (xp1, yp1) on the touch panel of the touch panel type device, and the operation start angle θp0 in the real space. Shows a touch panel type device tilted by Δθp. In the present embodiment, the touch operation on the touch panel and the posture operation of the touch panel device can be performed independently. For example, it is possible to change only the traveling direction of the airplane 502 and keep the pitch angle of the airplane 502 unchanged. In FIG. 5B (b), the operation start position is indicated by a dotted circle, the operation end position is indicated by a black circle, and a line drawn by the user is indicated by a dotted arrow, but these may not be actually displayed.

FIG. 5B (C) shows a schematic view of a cross section of the touch panel device obtained by cutting the touch panel device shown in FIG. 5B (B) along a broken line a.
<Third Embodiment>
Next, an outline of operations in the third embodiment will be described with reference to FIG. The third embodiment relates to a game in which a movement amount and movement of a display object (dragon) are operated in a three-dimensional virtual space by a user operation on a touch panel device. The user performs a swipe operation on the touch panel 132 of the touch panel device 130 and operates the position of the housing of the touch panel device 130 in the real space (X, Y, Z), thereby displaying one display in the three-dimensional virtual space. The movement and movement amount of the object can be operated separately. FIGS. 6A and 6B are diagrams illustrating state transitions (A) in the three-dimensional virtual space when operating the dragon 602 based on the third embodiment, and state transitions (B, C) of the touch panel device. It is.

First, the application progression unit 202 determines that the user operations requested by the application program based on the contents of the application program being executed in the main device 120 are “dragon movement amount operation” and “dragon jump operation”. Determine that there is. At this time, the user can instruct the dragon displayed in the display area 112 of the HMD to jump by changing the position (height Z) in the real space of the casing of the touch panel device 130 in a short time. As shown in FIG. 6B (B), the amount of movement of the dragon 602 displayed in the display area 112 of the HMD can be instructed by performing a swipe operation for drawing a line on the touch panel 132.
In the present embodiment, the operation of changing the position (height Z) in the real space of the housing in a short time is, for example, an operation of lifting the housing in a short time or immediately after lifting the housing in a short time. This means a lowering operation. By manipulating the height of the housing in the real space, the user can intuitively jump the dragon 602 displayed in the display area of the HMD.

  Next, in response to a request from the application progression unit 202, the touch panel device 130 outputs, as output data, information corresponding to a user operation that requires input from among the operation input information stored in the memory of the touch panel device 130. The data is sequentially output to the main device 120 over a predetermined period. In this embodiment, the output data is information related to the user's touch operation from the touch sensor 132 (contact position), information related to the displacement operation of the touch panel device 130 from the triaxial acceleration sensor 136 (acceleration in the Z-axis direction in real space). ).

  Next, the transmission unit of the touch panel device 130 sequentially outputs information corresponding to a user operation obtained over a predetermined period to the main body device 120 as output data. And the contact displacement amount calculation part 204 calculates the contact displacement amount from a contact start position, etc. as operation data based on the information regarding a user's touch operation. Further, the device displacement amount calculation unit 208 calculates, as operation data, the device displacement amount in the XYZ-axis directions in the three-dimensional real space of the touch panel device 130 based on information regarding the displacement operation of the touch panel device 130.

  Next, the display control unit 210 associates the operation to be performed on the display object with the operation data based on the user operation that is requested by the application program determined by the application progression unit 202. Then, the display control unit 210 performs display calculation of the display object based on the operation data (contact displacement amount, device displacement amount, etc.) associated with the operation of the display object, and thereby displays the display object image for each frame. To generate.

  FIG. 6A is a diagram illustrating an operation start state of the dragon 602 operated in the three-dimensional virtual space. The image shown in FIG. 6A can be displayed in the display area 112 of the HMD. FIG. 6A (a) shows a side view of the operation start state of the dragon 602 in the three-dimensional virtual space, and the dragon 602 corresponds to (xv0, yv0, zv0) in the three-dimensional virtual space. Has an operation start position. The operation start state of the touch panel type device shown in FIG. 6A corresponds to the state of the dragon 602 in the three-dimensional virtual space shown in FIG. In FIG. 6A (B), P0 (xp0, yp0) indicates a contact start position (black circle) on the touch panel plane by the user.

  FIG. 6A (c) shows a schematic view of a cross section of the touch panel device obtained by cutting the touch panel device shown in FIG. 6A (b) along a broken line a. In FIG. 4A (B), the operation start position is represented by a black circle, but this black circle may not actually be displayed.

  FIG. 6B is a diagram illustrating a state during operation of the dragon 602 operated in the three-dimensional virtual space. The operation of causing the dragon 602 to jump is determined based on a short-term displacement (ΔZ) in the Z-axis direction in the real space of the touch panel device, and the traveling direction of the dragon 602 is based on the amount of contact displacement from the contact start position with respect to the touch panel. Determined. Here, the traveling direction of the dragon 602 is calculated based on a line (input locus 606) input to the touch panel of the touch panel device.

  FIG. 6B (a) shows a view of the state after the operation of the dragon 602 in the three-dimensional virtual space as viewed from the side. The dragon 602 is (xv1, yv1, zv1) in the three-dimensional virtual space. Once there is a corresponding position during operation, immediately after that, landing (xv1, yv1, 0). The state during operation of the dragon 602 in the three-dimensional virtual space shown in FIG. 6B (a) corresponds to the state during operation of the touch panel device shown in FIG. 6B (b).

  FIG. 6B (b) shows an input locus when swiping from the touch start position P0 (xp0, yp0) to the touch end position P1 (xp1, yp1) on the touch panel of the touch panel device, and in the Z axis direction in the real space. A touch panel type device moved by ΔZ is shown. In FIG. 6B (b), the operation start position is indicated by a dotted circle, the operation end position is indicated by a black circle, and a line drawn by the user is indicated by a dotted arrow, but these may not be actually displayed.

FIG. 6B (C) shows a schematic view of a cross section of the touch panel device obtained by cutting the touch panel device shown in FIG. 6B (B) along a broken line a.
<Fourth Embodiment>
Next, an outline of operations in the fourth embodiment will be described with reference to FIG. The fourth embodiment relates to a game in which a tension (parameter) applied to a fishing line of a display object (fishing rod) is calculated in a three-dimensional virtual space by a user operation on the touch panel device, and the fishing rod is operated based on the tension. The user performs a swipe operation on the touch panel 132 of the touch panel device 130 and operates the position of the housing of the touch panel device 130 in the real space (X, Y, Z), thereby displaying one display in the three-dimensional virtual space. One parameter of the object (tension applied to the fishing line) can be calculated, and the fishing rod 702 can be operated based on the calculated parameter. 7A and 7B show the state transition (a) in the three-dimensional virtual space when operating the fishing rod 702 in the three-dimensional virtual space based on the fourth embodiment, and the state transition (b, It is a figure which illustrates c).

  First, the application progression unit 202 determines that the user operation requested by the application program based on the contents of the application program being executed on the main body device 120 is “reel operation of fishing rod (operation to wind fishing line)” and “ It is determined that it is a “lifting operation”. At this time, the user changes the position (height Z) in the real space of the housing of the touch panel device 130, thereby instructing the fishing rod raising operation (the amount by which the fishing rod is lifted) displayed in the display area 112 of the HMD. As shown in FIG. 7B (B) on the touch panel 132, a swipe rotation operation for drawing a circle makes a reel operation amount of a fishing rod displayed on the display area 112 of the HMD (winding of fishing line by the reel). Can be instructed. Here, the swipe rotation operation means an operation of touching the touch panel with a finger or a touch pen and moving the touch position on the touch panel so as to draw a periodic line (circle, ellipse, or reciprocal line) while touching. In the present embodiment, the touch panel device is disposed so that the Z axis of the three-dimensional real space corresponds to the left and right axes of the touch panel device, that is, the touch surface is directed to the right hand side of the user. Thus, the user can intuitively perform the winding operation with the reel as if the touch panel is regarded as a fishing rod reel (tool for winding fishing line).

  Next, in response to a request from the application progression unit 202, the touch panel device 130 outputs, as output data, information corresponding to a user operation that requires input from among the operation input information stored in the memory of the touch panel device 130. The data is sequentially output to the main device 120 over a predetermined period. In the present embodiment, the output data includes the application progression unit 202, information related to the user's touch operation from the touch sensor 132 (contact position)), information related to the displacement operation of the touch panel device 130 from the triaxial acceleration sensor 136 (actual Acceleration in the Z-axis direction in space).

  Next, the transmission unit of the touch panel device 130 sequentially outputs information corresponding to a user operation obtained over a predetermined period to the main body device 120 as output data. Then, the contact displacement amount calculation unit 204 calculates, as operation data, the contact displacement amount from the contact start position, the contact displacement direction, the contact displacement speed, the swipe rotation number, and the like based on information related to the user's touch operation. In addition, the device displacement amount calculation unit 208 calculates, as operation data, the device speed, the device displacement amount, and the like in the XYZ-axis directions in the three-dimensional real space of the touch panel device 130 based on information related to the displacement operation of the touch panel device 130. .

  Next, the display control unit 210 associates the operation to be performed on the display object with the operation data based on the user operation that is requested by the application program determined by the application progression unit 202. Then, the display control unit 210 performs display calculation of the display object based on operation data (contact displacement amount, contact displacement direction, contact displacement speed, swipe rotation number, device displacement amount, etc.) associated with the operation of the display object. Thus, an image of the display object is generated for each frame.

  FIG. 7A is a diagram illustrating an operation start state of a fishing rod 702 operated in a three-dimensional virtual space. The image shown in FIG. 7A (a) can also be displayed in the display area 112 of the HMD. FIG. 7A (a) shows a side view of the operation start state of the fishing rod 702 in the three-dimensional virtual space. The fishing rod 702 is the number of windings t0 and the tip position of the fishing rod in the three-dimensional virtual space. It has an operation start state corresponding to the height zv0. The operation start state of the touch panel device shown in FIG. 7A (B) corresponds to the operation start state of the fishing rod 702 in the three-dimensional virtual space shown in FIG. 7A (A). In FIG. 7A (B), P0 (xp0, yp0) indicates the contact start position (black circle) on the touch panel plane by the user.

  FIG. 7A (C) shows a schematic view of a cross section of the touch panel device obtained by cutting the touch panel device shown in FIG. 7A (B) along a broken line a. In FIG. 7A (B), the operation start position is indicated by a black circle, but this black circle may not actually be displayed.

  FIG. 7B is a diagram illustrating a state in which the fishing rod 702 operated in the three-dimensional virtual space is being operated. The fishing line winding amount by the reel operation of the fishing rod 702 is determined based on the number of swipes from the touch start position on the touch panel, and the lifting operation of the fishing rod 702 is the displacement (ΔZ) in the Z-axis direction in the real space of the touch panel device. To be determined. The tension (parameter) applied to the fishing line is determined based on the swipe speed and the displacement in the Z-axis direction in the real space of the touch panel device. For example, when the tension (parameter) applied to the fishing line is large, the user can observe in the HMD display area how the fishing rod is enlarged. The swipe rotation number is calculated by the contact displacement amount calculation unit 204 based on a line (input locus 706) input to the touch panel.

  FIG. 7B (a) shows a side view of the state after the operation of the fishing rod 702 in the three-dimensional virtual space, and the fishing rod 02 is the winding number t1, the tip of the fishing rod in the three-dimensional virtual space. It has an operating state corresponding to the position height zv1. The state during operation of the touch panel type device shown in FIG. 7B (B) corresponds to the state during operation of the fishing rod 702 in the three-dimensional virtual space shown in FIG. 7A (A).

  FIG. 7B (b) shows an input locus 706 when swiping and rotating from the touch start position P0 (xp0, yp0) to the touch end position P1 (xp1, yp1) on the touch panel of the touch panel type device, and the Z axis in the real space. A touch panel type device moved by ΔZ in the direction is shown. In FIG. 7B (b), the operation start position is indicated by a dotted line circle, the operation end position is indicated by a black circle, and a line drawn by the user is indicated by a dotted arrow, but these may not be actually displayed.

FIG. 7B (C) shows a schematic view of a cross section of the touch panel device obtained by cutting the touch panel device shown in FIG. 6B (B) along a broken line a. In FIG. 7B (B), the operation start position is indicated by a dotted line circle, the operation end position is indicated by a black circle, and a line drawn by the user is indicated by a dotted arrow, but these may not be actually displayed.
<Flow of HMD system display control processing>
A flow of processing for controlling display of the HMD system will be described with reference to FIG. Note that the processing of each step in the flowchart shown in FIG. 8 is merely an example, and the order of the processing of each step may be changed as long as a similar result is obtained.

  First, in step S1, initial processing is executed. The application processing unit 202 activates an application program in the main body device 120, sets initial values of various parameters, constructs a three-dimensional virtual space, and arranges display objects appearing in the virtual space at initial positions. The application processing unit 202 outputs the constructed display objects arranged in the three-dimensional virtual space and the initial position to the display area of the HMD. In the initial processing, for example, in the first embodiment, a three-dimensional virtual space composed of land and sea that can be observed from the control room of the spacecraft is observed from the cockpit of the airplane in the second embodiment. In the third embodiment, a three-dimensional virtual space composed of land, sea, or the like that can observe a dragon that exists on a plane obliquely from above. In the fourth embodiment, it is possible to construct a three-dimensional virtual space composed of the sea and the sky that can be observed when the user holds the touch panel device 130 as if it were a fishing rod.

  In step S <b> 2, the touch panel device 130 acquires each information output from the touch sensor 132, the tilt detection sensor 134, and the triaxial acceleration sensor as operation input information related to the user operation and stores it in the memory. That is, information (touch position) related to the touch operation on the touch panel by the user from the touch sensor 132, and information related to the tilt operation on the touch panel type device 130 by the user (three axes of XYZ in a three-dimensional real space) from the tilt detection sensor 134. From the three-axis acceleration sensor 136, information on the displacement operation on the touch panel device by the user (acceleration in the three-axis direction in the three-dimensional real space) is acquired. Then, the touch panel device sequentially stores the acquired information as operation input information in the memory.

  Next, in step S <b> 3, the application progression unit 202 determines a user operation that is requested to be input by the application program based on the content of the application program being executed by the main body device 120. In addition, the application progression unit 202 determines a period (predetermined period) for acquiring output data based on the operation input information based on the contents of the application program being executed.

  Next, in step S4, the application progression unit 202 requests the touch panel type device 130 for operation input information corresponding to the information related to the requested user operation, and sets a period (predetermined period) for acquiring output data to the touch panel type device. To 130.

  Next, in step S <b> 5, the transmission unit of the touch panel device 130 receives the requested user operation from the various operation input information stored in the memory for a predetermined period in response to a request from the application progression unit 202. Are sequentially output to the main unit 120 as output data.

In step S <b> 6, main device 120 determines the type of output data transmitted from touch panel device 130.
Next, in step S <b> 7, when the output data is information related to the touch operation on the touch panel of the user, the contact displacement amount calculation unit 204 performs the touch operation on the touch panel based on the information related to the touch operation (contact position). The contact displacement amount from the start position, the contact displacement direction, the contact displacement speed, the swipe rotation speed, and the like are calculated as operation data.

  Next, in step S8, when the output data is information regarding the tilt operation in the three-dimensional real space for the user's touch panel device 130, the tilt change calculation unit 206, based on the information regarding the tilt operation (angular velocity), An angle change, angular acceleration, and the like around three axes in the three-dimensional real space of the touch panel device 130 are calculated as operation data.

  Next, in step S9, when the output data is information related to the displacement operation in the three-dimensional real space with respect to the touch panel device 130, the device displacement amount calculation unit 208 uses the touch panel based on the information (acceleration) regarding the displacement operation. The apparatus speed and the apparatus displacement amount of the XYZ axes in the three-dimensional real space of the apparatus 130 are calculated as operation data.

  Next, in step S <b> 10, the display control unit 210 associates the calculated operation data with the operation to be performed by the display object based on the user operation that is requested by the application program determined by the application progression unit 202. . Then, the display control unit 210 performs display calculation of the display object based on operation data associated with the operation of the display object, thereby generating an image of the display object for each frame.

  Next, in step S11, the display control unit 210 sequentially outputs the generated image of the display object to the display area. The HMD displays the image of the display object output by the display control unit 210 in the three-dimensional virtual space displayed in the display area 112.

  As described above, the embodiment of the present invention has been described, but the present invention is not limited to the above embodiment. Those skilled in the art will recognize that various modifications can be made without departing from the spirit and scope of the invention as set forth in the appended claims.

Claims (11)

Method for operating display object in virtual space displayed on head mounted display in head mounted display system comprising non-transparent head mounted display and touch panel type device connected to said head mounted display for communication Because
Detecting a contact displacement amount from a contact start position by a user to the touch panel of the touch panel device;
A detecting device displacement in the change and the real space of the inclination of the touch-panel device, the device displacement amount is the amount of movement of the position of the touch-panel device in a real space, comprising the steps,
Manipulating the display object based on the contact displacement amount, the change in tilt, and the device displacement amount . The step of operating comprises
Manipulating the orientation of the first display object based on the contact displacement amount;
Operating the orientation of the second display object based on the detected tilt change of the touch panel device;
The method of claim 1, comprising: The step of detecting the contact displacement amount further includes a step of detecting a contact displacement direction from a start position of contact by the user with respect to the touch panel,
The step of operating comprises
Manipulating the movement amount and the traveling direction of the first display object based on the contact displacement amount and the contact displacement direction;
Manipulating the tilt of the first display object based on the tilt change;
The method of claim 1, comprising: The method according to claim 3, wherein the position of the first display object in the virtual space is a three-dimensional virtual space obtained by combining the contact displacement amount, the inclination change in the contact displacement direction, and the device displacement amount. A method characterized in that the position is converted into a coordinate within. Method for operating display object in virtual space displayed on head mounted display in head mounted display system comprising non-transparent head mounted display and touch panel type device connected to said head mounted display for communication Because
Detecting a contact displacement amount from a contact start position by a user with respect to the touch panel of the touch panel device;
Detecting a device displacement amount in real space of the touch panel device, wherein the device displacement amount is a moving amount of a position of the touch panel device in real space; and
Operating the display object based on the contact displacement amount and the device displacement amount. The step of operating comprises
Manipulating the amount of movement of the display object based on the amount of contact displacement;
Manipulating the operation of the display object based on the device displacement amount;
The method of claim 5, comprising:   The method according to claim 6, wherein the position of the display object in the virtual space is a position obtained by converting a position obtained by combining the device displacement amount with the contact displacement amount into coordinates in a three-dimensional virtual space. Feature method. The step of operating comprises
Calculating a parameter relating to the display object based on the device displacement amount and a swipe rotation speed with respect to the touch panel, and operating the display object based on the calculated parameter;
The method of claim 5, comprising:   The method according to claim 8, wherein the swipe rotation speed is calculated based on the contact displacement amount and a contact displacement direction from a contact start position by a user with respect to the touch panel.   The method according to claim 1, wherein the touch panel device is a smartphone.   The program for making the processor mounted in the head mounted display system perform the said method of any one of Claim 1 to 10.

Images