JP6580373B2 - Program, system, and method for controlling head mounted display - Google Patents

Description

  The present invention relates to a program, a system, and a method for controlling a head mounted display.

  Patent Document 1 discloses a railway simulation game device. This railway simulation game includes a seat, a control panel, and a display. By operating an operation tool while looking at the screen of the display, the railway vehicle is driven according to a predetermined operation pattern and competes for its accuracy.

Japanese Patent Laid-Open No. 10-244074

  Since the invention disclosed in Patent Document 1 requires a large-scale device, there is room for improvement in the sense of realism that the user is driving a vehicle easily.

  An object of the present invention is to provide a high sense of realism when simulating driving of a vehicle.

According to the present invention, a computer connected to a display and a head-mounted display that includes a sensor that detects inclination is provided with a virtual space and functions to operate a vehicle object or a background object in the virtual space. A vehicle object based on a display control unit that displays a vehicle object including a control unit and a background object that is the outside of the vehicle object on a display, and a tilt of a head mounted display detected by a sensor. Alternatively, a program including an object control unit that operates the background object and a determination unit that determines whether the vehicle object or the background object is operated so as to satisfy a predetermined condition is obtained.

In addition, according to the present invention, a display and a computer connected to the first head mounted display and the second head mounted display having a sensor for detecting inclination are provided with a virtual space on the display, and the vehicle object in the virtual space is provided. Alternatively, a program that functions to operate a background object, a display control unit that displays a vehicle object including a control unit and a background object that is an external world of the vehicle object on a display, and a first detected by a sensor. Based on the tilt of one head-mounted display and the command from the second head-mounted display, the object control unit for operating the vehicle object or the background object, and the vehicle object or the background object so as to satisfy a predetermined condition Includes a determination section for determining whether or not the work, and a condition by the operation from the first head-mounted display, and conditions by the operation of the second head-mounted display are different, the program can be obtained.

According to another aspect of the present invention, there is provided a system including a head mounted display including a display, a sensor for detecting tilt, and a control circuit unit, the control circuit unit including a vehicle object including a control unit on the display. A virtual space including a background object that is an external environment of the vehicle object, and operating the vehicle object or the background object based on the inclination of the head mounted display detected by the sensor, so that the vehicle object or the background object is predetermined. A system for determining whether or not an operation is performed so as to satisfy the above condition is obtained.

Moreover, according to this invention, it is a control method of a head mounted display provided with the display and the sensor which detects an inclination, Comprising: The method provided with the following is obtained.
1) Providing the display with a virtual space including a vehicle object including a control unit and a background object that is an external world of the vehicle object;
2) Manipulating the vehicle object or the background object based on the tilt of the head mounted display detected by the sensor;
3) It is determined whether or not the vehicle object or the background object has been operated to satisfy a predetermined condition.

  According to the present invention, it is possible to provide a high sense of realism when simulating driving of a vehicle.

1 is a diagram illustrating an HMD system according to an embodiment of the present invention. An orthogonal coordinate system in a three-dimensional space defined around the head of a user wearing an HMD is shown. It is an XYZ space figure which shows an example of the correspondence of the arrangement position of virtual space and real space. It is a XZ top view which shows an example of the correspondence of the arrangement position of virtual space and real space. It is a block diagram which shows the function of the control circuit part for implement | achieving the function of a HMD system. It is a three-dimensional schematic diagram which shows a visual field area | region. It is the YZ plane view which looked at the visual field area from the X direction. It is the XZ plane figure which looked at the visual field area from the Y direction. It is a block diagram explaining the function of the object control part shown in FIG. It is a figure which shows an example of an object operation table. It is a flowchart which shows the process for implement | achieving the function of a HMD system. It is a figure which shows an example of the visual field image displayed on HMD. It is a figure which shows an example of the visual field image displayed on HMD. It is a figure which shows an example of the visual field image displayed on HMD. It is a figure which shows an example of the visual field image displayed on HMD. It is a figure which shows an example of the visual field image displayed on HMD. It is a figure which shows an example of the visual field image displayed on HMD. It is a table which shows an example of the determination content in a determination part. It is a figure which shows an example of the determination method in a determination part. It is a figure which shows the HMD system by other embodiment of this invention.

[Description of Embodiment of the Present Invention]
First, the contents of the embodiment of the present invention will be listed and described. A program, system, and method for controlling a head mounted display according to an embodiment of the present invention have the following configurations.
(Item 1)
A program for causing a computer connected to a display and a head-mounted display having a sensor for detecting tilt to provide a virtual space for the display and to function to operate a vehicle object or a background object in the virtual space. ,
A display control unit for displaying the vehicle object including a control unit on the display, and the background object that is the outside of the vehicle object;
An object control unit for operating the vehicle object or the background object based on the inclination of the head-mounted display detected by the sensor;
A determination unit that determines whether or not the vehicle object or the background object is operated so as to satisfy a predetermined condition.
According to the program of this item, based on the inclination of the head mounted display, it is determined whether or not the vehicle object or the background object is operated so as to satisfy a predetermined condition. Therefore, for example, an element such as whether or not the line of sight has been appropriately sent to a signal outside the vehicle window can be newly added to the vehicle driving simulation. Therefore, a high sense of reality can be provided.
(Item 2)
The program further includes a field-of-view determination unit that defines a reference line of sight in the virtual space based on tilt information detected by the sensor,
The display control unit further displays a confirmation object on the background object,
The program according to item 2, wherein the determination unit determines whether or not the reference line of sight intersects the confirmation object.
According to the program of this item, it is possible to accurately determine whether or not the line of sight has been properly sent to a specific object. Therefore, a high sense of realism can be provided when simulating driving of a vehicle.
(Item 3)
The confirmation object changes relative distance information with the vehicle object according to the movement of the vehicle object,
The program according to item 2, wherein the determination unit determines whether the reference line of sight intersects with the confirmation object when the relative distance information is within a predetermined range.
According to the program of this item, when the relative distance between the confirmation object and the vehicle object is within a predetermined range, it can be determined whether or not the line of sight has been sent to the confirmation object. Therefore, it can be determined whether or not the line of sight has been appropriately sent to a specific object at an appropriate timing, and a high sense of presence can be provided.
(Item 4)
The computer is connected to an external controller;
The program according to any one of items 1 to 3, wherein the object control unit further operates the vehicle object or the background object based on an input to the external controller.
According to the program of this item, a specific object can be further operated based on an input to the external controller. Therefore, a high sense of realism can be provided when simulating driving of a vehicle.
(Item 5)
The program further includes a field-of-view determination unit that defines a reference line of sight in the virtual space based on tilt information detected by the sensor,
The display control unit further displays a confirmation object on the background object,
The determination unit determines whether an input to the external controller is made within a predetermined time after the reference line of sight intersects the confirmation object when the reference line of sight intersects the confirmation object. Item 4 program.
According to the program of this item, it can be further determined whether or not a predetermined action has been appropriately performed before and after the action of sending a line of sight to the confirmation object. Therefore, for example, it is possible to add a new element to the vehicle driving simulation such as appropriately sending a line of sight to a signal outside the vehicle window and confirming the pointing. Therefore, a high sense of reality can be provided.
(Item 6)
The head mounted display further includes a microphone for receiving voice input,
The program according to any one of Items 1 to 5, wherein the determination unit determines whether or not the voice input is accepted when the vehicle object or the background object satisfies a predetermined condition.
According to the program of this item, it can be determined whether or not the vehicle has been appropriately driven based on the voice input. Therefore, it is possible to newly add an element such as making an announcement to the passenger of the vehicle at an appropriate timing to the vehicle driving simulation. Therefore, a high sense of reality can be provided.
(Item 7)
A display and a computer connected to a first head mounted display and a second head mounted display having a sensor for detecting tilt provide a virtual space to the display and operate a vehicle object or a background object in the virtual space. A program that functions like
A display control unit for displaying the vehicle object including a control unit on the display, and the background object that is the outside of the vehicle object;
An object control unit for operating the vehicle object or the background object based on an inclination of the first head mounted display detected by the sensor and a command from the second head mounted display;
A determination unit that determines whether or not the vehicle object or the background object is operated to satisfy a predetermined condition,
The program in which the condition by the operation from the first head mounted display is different from the condition by the operation from the second head mounted display.
According to the program of this item, the first player wearing the first head mounted display and the second player wearing the second head mounted display having a different role from the first player can appear in the virtual space. . Therefore, a high sense of realism can be provided when simulating driving of a vehicle.
(Item 8)
A system including a head-mounted display comprising a display, a sensor for detecting tilt, and a control circuit unit,
The control circuit unit is
Providing the display with a virtual space including a vehicle object including a control unit and a background object that is an external world of the vehicle object;
Based on the inclination of the head mounted display detected by the sensor, the vehicle object or the background object is operated,
A system for determining whether the vehicle object or the background object has been operated to satisfy a predetermined condition.
According to the system of this item, based on the inclination of the head mounted display, it is determined whether or not the vehicle object or the background object is operated to satisfy a predetermined condition. Therefore, for example, an element such as whether or not the line of sight has been appropriately sent to a signal outside the vehicle window can be newly added to the vehicle driving simulation. Therefore, a high sense of reality can be provided.
(Item 9)
A method for controlling a head mounted display comprising a display and a sensor for detecting tilt, comprising:
1) Providing the display with a virtual space including a vehicle object including a control unit and a background object that is an external world of the vehicle object;
2) Manipulating the vehicle object or the background object based on the tilt of the head mounted display detected by the sensor;
3) It is determined whether or not the vehicle object or the background object has been operated to satisfy a predetermined condition.
According to the control method of this item, based on the tilt of the head mounted display, it is determined whether or not the vehicle object or the background object is operated so as to satisfy a predetermined condition. Therefore, for example, an element such as whether or not the line of sight has been appropriately sent to a signal outside the vehicle window can be newly added to the vehicle driving simulation. Therefore, a high sense of reality can be provided.

[Details of the embodiment of the present invention]
Specific examples of a program, a system, and a method for controlling a head mounted display according to an embodiment of the present invention will be described below with reference to the drawings. In addition, this invention is not limited to these illustrations, is shown by the claim, and intends that all the changes within the meaning and range equivalent to the claim are included. In the following description, the same reference numerals are given to the same elements in the description of the drawings, and redundant descriptions are omitted.

  FIG. 1 shows an HMD system 100 including a head mounted display (hereinafter referred to as “HMD”) 110 according to the present embodiment. The HMD system 100 includes an HMD 110 that is worn on the user's head, a control circuit unit 120, a sensor 130, and an external controller 140. The HMD 110 includes a display 112 that is a non-transmissive display device and a sensor unit 114. The control circuit unit 120 displays a right-eye image and a left-eye image on the display 112, thereby providing a three-dimensional image using parallax between both eyes as a virtual space. By arranging the display 112 in front of the user's eyes, the user can be immersed in the virtual space. The virtual space includes various objects that can be operated by the background and the user, as will be described later.

  The display 112 may include a right-eye sub-display that provides a right-eye image and a left-eye sub-display that provides a left-eye image. Moreover, as long as the image for right eyes and the image for left eyes can be provided, you may be comprised by one display apparatus. For example, the right-eye image and the left-eye image can be provided independently by switching the shutter so that the display image can be recognized by only one eye at high speed.

  The control circuit unit 120 is a computer connected to the HMD 110, provides a virtual space to the display 112, and functions to operate various objects displayed in the virtual space. The control circuit unit 120 stores a program for controlling such an operation. The control circuit unit 120 may not be mounted on the HMD 110 and may be configured as another hardware (for example, a known personal computer or a server computer via a network). In addition, the control circuit unit 120 may implement only a part of functions in the HMD 110 and implement the remaining functions in different hardware.

  The sensor 130 detects information related to the position and movement of the HMD 110. The sensor 130 includes a sensor unit 114 and a detection unit 132. The sensor unit 114 may include a plurality of light sources. The light source is, for example, an LED that emits infrared rays. The detection unit 132 is, for example, an infrared sensor, and detects information about an angle in the real space of the HMD 110 according to the user's movement by detecting infrared rays from the light source as detection points of the HMD 110 over time. Then, it is possible to determine the time change of the angle of the HMD 110 based on the change with time of the information detected by the detection unit 132, and to detect information regarding the position and movement of the HMD 110.

  Information regarding the angle acquired by the sensor 130 will be described with reference to FIG. XYZ coordinates are defined around the head of the user wearing the HMD 110. The vertical direction in which the user stands upright is the Y axis, the front-rear direction orthogonal to the Y axis and connecting the center of the display 112 and the user is the Z axis, and the horizontal direction orthogonal to the Y axis and the Z axis is the X axis. Then, the inclination angle θx (so-called pitch angle), θy (so-called yaw angle), and θz (so-called roll angle) of the HMD 110 around each axis are detected, and information on the position and movement of the HMD 110 is detected by their change over time. can do.

  The sensor 130 may be configured by only one of the sensor unit 114 fixed near the display 112 and the detection unit 132. The sensor unit 114 may be a geomagnetic sensor, an acceleration sensor, or an angular velocity (gyro) sensor, and the inclination of the HMD 110 (particularly, the display 112) mounted on the user's head is detected using at least one of them. To do. Thereby, the information regarding the position and movement of the HMD 110 can be detected. For example, the angular velocity sensor can detect the angular velocities around the three axes of the HMD 110 over time according to the movement of the HMD 110, and determine the temporal change in the angle (tilt) around each axis. In this case, the detection unit 132 is not necessary. The detection unit 132 may include an optical camera. In this case, information regarding the position and movement of the HMD 110 can be detected based on the image information, and the sensor unit 114 is unnecessary.

The function of detecting information related to the position and movement of the HMD using the sensor 130 is referred to as position tracking. The relationship between the position tracking by the sensor 130 and the virtual camera 1 arranged in the virtual space 2 will be described with reference to FIGS. 3A and 3B. In order to describe the positional relationship between the virtual camera 1 and the sensor 130, the position of the sensor 130 is the position of the detection unit 132 when the detection unit 132 is provided, and the sensor unit 114 when the detection unit 132 is not provided. The position of 3A is a three-dimensional schematic diagram showing the relationship between the sensor 130 in the real space and the virtual space 2, and FIG. 3B is a plan view of the relationship between the sensor 130 in the real space and the virtual space 2 as seen from the Y direction. . The virtual camera 1 is disposed inside the virtual space 2, and the sensor 130 is virtually disposed outside the virtual space 2 (real space).

The virtual space 2 is formed into a celestial sphere having a plurality of meshes 3 that are substantially square or substantially rectangular. Each mesh is associated with spatial information of the virtual space 2, and a visual field image to be described later is formed based on this spatial information. In the present embodiment, as shown in FIG. 3B, it is preferable to adjust so that the center point 21 of the celestial sphere is always arranged on the line connecting the virtual camera 1 and the sensor 130 in the XZ plane. For example, when the user wearing the HMD moves and the position of the virtual camera 1 moves in the X direction, the area of the virtual space 2 so that the center 21 is located on the line segment between the virtual camera 1 and the sensor 130. Is changed.

  The external controller 140 is a device capable of communication so that various commands can be sent to the control circuit unit 120, and is preferably configured by a portable terminal capable of wireless communication. The external controller 140 can be any portable device that includes a CPU, a main memory, an auxiliary memory, a transmission / reception unit, a display unit, and an input unit that are bus-connected to each other. For example, a smartphone, a PDA, a tablet computer, a game console, and a notebook PC can be applied, and a portable terminal including a touch panel is preferable. The user can influence the object displayed in the virtual space by performing various touch operations including tap, swipe, and hold on the touch panel of the external controller 140.

  The HMD system 100 may include headphones including a microphone as any element. Thereby, the user can give a voice instruction to a predetermined object (game object or the like described later) in the virtual space. Further, in order to receive a broadcast of a television program on a virtual television in a virtual space, the HMD system 100 may include a television receiver in any element.

  FIG. 4 is a diagram illustrating functions of the control circuit unit 120 for realizing display processing of the virtual space 2 in the HMD system 100 and operation of objects displayed in the virtual space 2. The control circuit unit 120 controls an output image to the display 112 mainly based on inputs from the sensor 130 and the external controller 140.

The control circuit unit 120 includes a display control unit 200, an object control unit 300, and a determination unit 400. The display control unit 200 includes a motion detection unit 210, a visual field determination unit 220, a visual field image generation unit 230, and a spatial information storage unit 240. The object control unit 300 includes an object information storage unit 310 and a virtual camera information storage unit 320. The sensor 130, the display control unit 200, the object control unit 300, and the display 112 are connected to be communicable with each other, and can be connected via a wired or wireless communication interface. The external controller 140 is communicably connected to the object control unit 300. The spatial information storage unit 240, the object information storage unit 310, and the virtual camera information storage unit 320 include various tables for providing output information corresponding to inputs from the sensor 130 and the external controller 140 to the display 112. The determination unit 400 determines whether or not an input from the sensor 130 or the external controller 140 satisfies a predetermined condition and various objects are operated so as to satisfy the predetermined condition.

  Based on the angle information detected over time by the sensor unit 114 and the angle information of the HMD 110 detected over time by the detection unit 132, the sensor 130 displays information regarding the position and movement of the HMD 110, and the display control unit 200 and Output to the object control unit 300. The motion detection unit 210 detects the motion of the HMD 110 based on input information from the sensor 130. Then, the visual field direction serving as a reference line of sight that defines the direction of the HMD 110 in the virtual space 2 is detected. Then, the detected information is output to the visibility determining unit 220.

  The visual field determination unit 220 relates to the visual field region of the virtual camera 1 in the virtual space 2 based on the virtual space information stored in the spatial information storage unit 240 and input information from the sensor 130 (information from the motion detection unit 210). Determine information. The view image generation unit 230 generates a part of the 360 degree panorama constituting the virtual space as a view image based on the view information. The view field image is output to the display 112. The field-of-view image includes two two-dimensional images for the left eye and right eye, and these are superimposed on the display 112, thereby providing the user with a virtual space 2 as a three-dimensional image.

  The object control unit 300 identifies an operation target object based on object information in the virtual space stored in the object information storage unit 310 and user commands from the sensor 130 and the external controller 140. Then, the virtual camera information stored in the virtual camera information storage unit 320 is adjusted based on a predetermined user operation command to the operation target object. The adjusted virtual camera information is output to the display control unit 200, and the view field image is adjusted. In addition, an operation corresponding to a predetermined user operation command is performed on the operation target object, and object control information is output to the display 112 and the display control unit 200. Specific processing of the object operation will be described later.

  Note that hardware elements for realizing each function of the control circuit unit 120 can be constituted by a CPU, a memory, and other integrated circuits. Each function is realized by various programs as software elements loaded in the memory. Accordingly, those skilled in the art will understand that these functional blocks can be realized by hardware, software, or a combination thereof.

  With reference to FIGS. 5A to 5C, the visual field region 6 along the celestial sphere of the virtual space 2 that is determined by the visual field determination unit 220 will be described. 5A is a three-dimensional schematic diagram showing the visual field region 6, FIG. 5B is a YZ plane view of the visual field region 6 viewed from the X direction, and FIG. 5C is an XZ plane view of the visual field region 6 viewed from the Y direction. It is. As shown in FIG. 5A, the visual field region 6 constitutes a part of the virtual space 2. The visual field area 6 is determined based on the visual field direction 5, and the visual field direction 5 is determined based on the position and direction of the virtual camera 1. The visual field region 6 is a first region 61 that is a predetermined range around the X axis at the intersection between the virtual space 2 and the visual field direction 5, and a predetermined range around the Y axis at the intersection between the virtual space 2 and the visual field direction 5. 2 viewing areas 62. The first region 61 is set as a range including the polar angle α with the visual field direction 5 as the center. The second region 62 is set as a range including the azimuth angle β with the visual field direction 5 as the center. That is, the view field region 6 can be a region on a spherical surface having the polar angle θ in the XY plan view and the azimuth angle β in the XZ plan view with respect to the view direction 5 from the position of the virtual camera 1.

  With reference to FIGS. 6 and 7, the processing operation of the control circuit unit 120 related to the object operation in the virtual space 2 will be described. FIG. 6 is a block diagram illustrating functions of the object control unit 300 shown in FIG. The object control unit 300 includes an object specifying unit 330 for specifying an operation target object, and an object operation unit 340 for operating the specified operation target object.

  The object specifying unit 330 includes a reference line-of-sight calculation unit 331, an object determination unit 332, and an operation target object selection unit 333. The object specifying unit 330 specifies an object arranged on the visual field direction 5 that is the reference line of sight shown in FIGS. 5A to 5C in the virtual space 2 by the tilting operation of the HMD 110. For this purpose, the reference line-of-sight calculation unit 331 sets the field-of-view direction 5, and the object determination unit 340 determines whether an object is placed in the field-of-view direction 5. Further, when an object is arranged in the visual field direction 5, the operation target object selection unit 360 selects the object as the operation target object.

  The object operation unit 340 includes an external controller operation determination unit 341, a tilt operation determination unit 342, a virtual camera adjustment unit 343, an object operation specification unit 344, an object operation execution unit 345, and an object operation table 346. The object operation unit 340 performs an operation according to the tilt operation of the HMD 110 on the operation target object specified by the object specifying unit 300.

The external controller operation determination unit 341 determines whether a user's touch operation is input to the external controller 140. The tilt motion determination unit 342 determines whether a tilt motion (particularly the tilt direction) is input to the HMD 110. The virtual camera adjustment unit 343 adjusts the position and direction of the virtual camera 1 during the object operation. The object operation specifying unit 344 specifies an operation for the operation target object according to the object operation table 346. The object operation execution unit 345 performs an object operation and generates a resulting image as a view image.

  FIG. 7 shows an example of the object operation table 346. The object operation table 346 associates the user action with the action of the operation target object. That is, one-to-one correspondence is made so that the operation target object takes a predetermined action with respect to an operation such as a tilt change operation on the HMD 110 by the user or an input to the external controller 140. The action of the operation target object may be associated by an input operation to the HMD 110 or the external controller 140 alone, or may be associated with a coordinated operation of the HMD 110 and the external controller 140. Details of each action for the input will be described later.

Note that when the object operation table 346 includes information on the user action and the operation target object, the object operation unit 340 may have a function as the object specifying unit 330. That is, the operation target object information included in the object operation table 346 may be specified by the object specifying unit 330 or may be set in advance.

  Next, a processing flow for object operation in the virtual space 2 according to the present embodiment will be described with reference to FIGS. The object operation process can be realized by the interaction of the external controller 140, the sensor 130, the HMD 100, and the control circuit unit 120.

First, the position of the HMD 110 is detected by the sensor 130 (step S130-1).
The detection information of the sensor 130 is transmitted to the control circuit unit 120, and the position information and the tilt information of the HMD 110 are determined by the motion detection unit 210 (step S120-1). The field-of-view determination unit 220 determines field-of-view information in the virtual space 2 based on position information and / or tilt information of the HMD 110. And the visual field image generation part 230 produces | generates the visual field image for displaying on the display 112 of HMD110 based on the said visual field information (step S120-2).

At this time, the visual field determination unit 220 determines the visual field direction 5 that is the reference visual line (step S120-3). The visual field direction 5 is determined in association with a predetermined position of the visual field image as described above. The predetermined position is preferably the center point of the view image, but is not limited thereto, and may be any settable position of the view image. Further, the visual field direction may be displayed in a superimposed manner on the visual field image as some mark (for example, a circular or palm icon).

  Next, a view field image is displayed on HMD110 (step S100-2). As will be described later, a plurality of objects that can be operated are displayed in the view field image. When the user wearing the HMD 110 moves the head while the view image is displayed on the HMD 110, at least one of the plurality of objects arranged in the virtual space 2 is the view image. Alignment is performed so as to correspond to a predetermined position.

An example of a view field image displayed on the HMD 110 is shown in FIG. The view image G1 includes a vehicle object V1 (first object) and a background object B1 (second object). The vehicle object V1 corresponds to, for example, a cockpit of a train, and includes a control unit M and a display unit D that displays a control result of the control unit M. The control unit M includes a control bar M1 for accelerating the train and a brake bar M2 for decelerating the train. The display unit D includes a speedometer D1 that reflects the speed of the train by the control bar M1 and the brake bar M2, and a pressure gauge D2 that reflects the degree of deceleration of the train by the operation of the brake bar M2. The background object B1 corresponds to the background of the vehicle object V1, and in this embodiment is a landscape of the outside world viewed from the driver's seat of the train. A mark O corresponding to the visual field direction 5 is displayed near the center of the background object B1.

The field-of-view image G1 is configured as a virtual space 2 that is a three-dimensional object, and the speed at which the background object B1 changes is changed according to the speed of the vehicle object V1. Thereby, the user can experience the situation as a train driver with a high sense of reality. Note that the vehicle object V1 is not limited to a train, and various vehicles such as passenger cars, commercial vehicles such as buses and taxis, and airplanes can be used, and driving of various vehicles can be simulated.

  Next, when the position of the HMD 110 is further detected by the sensor 130, a user input to the HMD 110 is detected (steps S130-2 and 110-3). The detection information of the sensor 130 is transmitted to the object control unit 300, the position information and the tilt information of the HMD 110 are determined by the tilt motion determination unit 342, and the user input to the HMD 110 is accepted (step S120-4). The object operation specifying unit 344 refers to the object operation table 346 and selects an operation target object associated with the user input and the operation content (step S120-5).

In addition, when a user's touch operation on the external controller 140 is input, the input information is transmitted to the object control unit 300 and is received by the external controller operation determination unit 341. The object operation specifying unit 344 refers to the object operation table 346 and selects an operation target object associated with the user input and the operation content (step S120-5).

  With reference to FIGS. 10 to 15, an example of a mode of controlling the operation target object based on a user input to the HMD 110 or the external controller 140 is shown.

10 and 11 show a mode in which the vehicle object V1 as the operation target object is controlled by inputting a user's touch operation to a device having a touch panel as the external controller 140. FIG. As shown in FIG. 7, the input operation to the external controller 140 is associated with the operation target object and the operation content. Therefore, the vehicle object V1 can be controlled by inputting a predetermined operation to the external controller 140.

  For example, when a swipe operation is input to the touch panel of the external controller 140 with a linear trajectory from top to bottom, the vehicle object V1 is controlled to accelerate. Specifically, as shown in FIG. 10, when the swipe operation is input, an action of tilting forward is performed on the control bar M1 associated as the operation target object. In accordance with this, the vehicle object V1 accelerates, and the meter of the speedometer D1 that is similarly associated as the operation target object varies according to the speed of the vehicle object V1. Then, the changing speed of the background object B1 is changed according to the speed of the vehicle object V1. Thereby, the user can operate the vehicle object V1 including the control unit M by operating the external controller 140.

  Further, when a swipe operation is input to the touch panel of the external controller 140 in a circular arc from right to left, the vehicle object V1 is controlled to decelerate. Specifically, as shown in FIG. 11, when the swipe operation is input, an action of twisting clockwise is performed on the brake bar M2 associated as the operation target object. In accordance with this, the vehicle object V1 decelerates, and the meter of the pressure gauge D2 similarly associated as the operation target object changes according to the swipe amount. The pressure gauge D2 reflects the strength of the brake, and the vehicle object V1 decelerates in response to the brake action, and the meter of the speedometer D1 changes. Then, the changing speed of the background object B1 is changed according to the speed of the vehicle object V1. Thereby, the user can operate the vehicle object V1 including the control unit M by operating the external controller 140.

FIG. 12 shows a mode in which the background object B1 as the operation target object is controlled based on a user input to the HMD 110. As shown in FIG. 7, the input operation to the HMD 110, the operation target object, and the operation content are associated with each other. Therefore, the background object B1 can be controlled by inputting a predetermined operation to the HMD 110.

  For example, the user's signal confirmation operation is executed when the user turns his / her line of sight toward the signal C1 (an example of the confirmation object) included in the background object B1. Thereby, based on the control of the background object B1, game points are awarded as described later. Specifically, as shown in FIG. 12, when the user turns his / her head in the right direction and directs his / her line of sight to the signal C <b> 1, the inclination of the HMD 110 is detected by the sensor 130, thereby the mark corresponding to the viewing direction 5. The position of O changes to the right. A predetermined recognition area C2 is set near the outside of the signal C1, and at least part of the recognition area C2 and the mark O overlaps, so that the signal C1 is determined and selected as an operation target object. As a result, it can be determined that the user has looked at the signal C1. Thus, by operating the HMD 110, the background object B1 including the signal C1 can be operated. The recognition area C2 is preferably a virtual area that the user cannot visually recognize.

As illustrated in FIGS. 13 and 14, the background object B <b> 1 as the operation target object may be controlled based on both the user input to the HMD 110 and the user input to the external controller 140. As shown in FIG. 7, the input operation to the HMD 110 and the external controller 140 is associated with the operation target object and the operation content. Therefore, the background object B1 can be controlled by inputting a predetermined operation to the HMD 110 and the external controller 140.

  For example, as shown in FIG. 13, the user touches the touch panel of the external controller 140 within a predetermined time following the operation in which the user turns to the right and directs his / her line of sight toward the signal C1, and the user points to the signal C2. You may make it confirm a difference. Further, the image O1 corresponding to the operation may be further displayed on the view image G1. Thereby, it can be fed back to the user that the finger pointing confirmation is completed. And you may make it provide further the game point according to finger pointing confirmation.

  Further, as shown in FIG. 14, following the operation in which the user turns to the left and directs his / her line of sight toward the horn mark C3, the user touches the touch panel of the external controller 140 within a predetermined time, and the user rings the horn. You may do it. Further, the image O2 corresponding to the operation may be further displayed on the view image G1. Thereby, it can be fed back to the user that the sounding of the horn has been completed. And when a horn can be sounded at an appropriate timing, you may make it give a game point further.

  Then, the object operation specifying unit 344 refers to the object operation table 346, determines whether the above operation satisfies a predetermined condition, and outputs the result to the determination unit 400. The determination unit 400 determines whether or not the above-described operation satisfies a predetermined condition based on the table associating the input operation with the determination content illustrated in FIG. 15 (step S120-6). That is, whether or not a predetermined touch operation has been performed on the touch panel of the external controller 140 so as to satisfy a predetermined condition, and whether or not a predetermined tilt operation has been input to the HMD 110 so as to satisfy a predetermined condition. Is determined.

  Here, the object operation specifying unit 344 determines whether or not the user wearing the HMD 110 has aligned his / her line of sight with the object to be operated based on whether or not the visual field direction 5 serving as the reference line of sight intersects the operation target object. It is preferable to judge. Further, as described above, it is preferable to provide a virtual recognition area in the vicinity of the operation target object. Accordingly, it is possible to prevent a situation in which the user is not accepted as an operation input although the user visually recognizes the operation target object.

The determination unit 400 preferably determines whether or not the viewing direction 5 intersects the operation target object at a predetermined timing. For example, in a train driving simulation, a driver's safety behavior can be given as a game point by visually recognizing a signal at an appropriate timing and determining whether or not a finger is confirmed at an appropriate timing.

In this case, the signal C1 is preferably set so that the relative distance information with respect to the train V1 changes according to the movement of the train V1. As shown in FIG. 16, the virtual camera 1 is set to the position of the train V1, the relative distance d1 between the train V1 and the signal C1 is set, and the relative distance d1 changes according to the movement of the train V1. When the relative distance d1 is in the range of the long distance d2, it is not determined that the signal confirmation has been performed because the visual recognition timing is too early. When the relative distance d1 is in the range of the short distance d4, it is not determined that the signal confirmation has been performed because the visual recognition timing is too late. When the relative distance d1 is in the range of the appropriate distance d3, it is determined that the signal confirmation is performed at an appropriate timing.

  Furthermore, as described above, the time until the tap operation is input to the touch panel of the external controller 140 is measured continuously after the signal confirmation operation, and when it is determined that the input is confirmed within a predetermined time, It is preferable to determine that the pointing confirmation has been input.

  Thus, when it is determined that the input operation satisfies the predetermined condition, a game point is awarded (step S120-7). By determining the driving level of the driver based on the size of the game point, an element such as whether or not the line of sight has been appropriately sent to a signal outside the vehicle window can be newly added to the vehicle driving simulation. Therefore, a high sense of reality can be provided.

  Subsequently, an operation is performed on the operation target object by the object operation execution unit 345, and a series of result images including the operation result is generated (step S120-8). Further, the position and direction of the virtual camera 1 are adjusted, and the result is transmitted from the virtual camera information storage unit 320 to the field of view determination unit 220. The view image generation unit 230 acquires the updated view image from the spatial information storage unit 240 based on the adjusted virtual camera information. Then, an operation result image is generated by synthesizing a series of result images including the operation result of the operation target object with the view field image, and this is output to the HMD 110 (step S120-9). The HMD 110 displays the operation result image on the display 112 (step S110-4).

As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment. Without departing from the spirit and scope of the invention as set forth in the appended claims
Those skilled in the art will appreciate that various modifications of the embodiments can be made.

  For example, as shown in FIG. 17, the HMD system 100 includes a first player P1 wearing the first HMD 110-1 and a second player P2 wearing the second HMD 110-2. You may make space 2 an experience. That is, images corresponding to the same virtual space 2 are displayed on the first HMD 110-1 and the second HMD 110-2. The first player P1 may be given a role as a driver as described above, while the second player P22 may be given a role different from the first HMD 110-1 (for example, a role as a conductor). In this case, it is preferable that the predetermined condition for controlling the vehicle object V1 or the background object B1 is different between the first HMD 110-1 and the second HMD 110-2. Thus, the first player P1 wearing the first HMD and the second player P2 wearing the second HMD having a role different from that of the first player can appear in the virtual space. Therefore, a high sense of realism can be provided when simulating driving of a vehicle. Such a process controls the control circuit unit 120 to display the same view image on the first HMD 110-1 and the second HMD 110-2, and to the operation target object from each of the first HMD 110-1 and the second HMD 110-2. The operation input image is received, an operation result image is generated based on the input, and control is performed so that the same view image is displayed on the first HMD 110-1 and the second HMD 110-2.

At this time, the second HMD 110-2 may further include a microphone 142 that accepts voice input as the external controller 140. In this case, in step S120-6, it is preferable to determine whether or not a voice input is accepted when the vehicle object V1 or the background object B1 satisfies a predetermined condition. For example, if the first player makes a voice input for announcing the next station name within a predetermined time after the deceleration operation is input to the control unit M of the vehicle object V1, the second player can train at an appropriate timing. A simulation can be performed to announce the passengers. Also, when the background object B1 satisfies a predetermined condition (for example, when passing near a dangerous railroad crossing, no announcement is made), an audio input for announcement may be made. Thus, it can be determined whether or not the vehicle has been appropriately driven based on the voice input. Therefore, it is possible to newly add an element such as making an announcement to the passenger of the vehicle at an appropriate timing to the vehicle driving simulation. Therefore, a high sense of reality can be provided.

Further, the second player P2 as a conductor may be able to visually recognize an in-vehicle object as another background object. In this case, the conductor pays attention to the passengers in the vehicle and is given a role of giving an appropriate instruction to the first player P1 who is the driver, so that the sense of reality can be further improved.

  DESCRIPTION OF SYMBOLS 100 ... Head mounted display (HMD) system, 110 ... HMD, 112 ... Display, 114 ... Sensor part, 120 ... Control circuit part, 130 ... Sensor, 132 ... Detection part, 140 ... External controller, 200 ... Display control part, 300 DESCRIPTION OF SYMBOLS ... Object control part, 400 ... Determination part, 1 ... Virtual camera, 2 ... Virtual space, 5 ... View direction, 6 ... View area, G1 ... View image, V1 ... Vehicle object, B1 ... Background object, C1 ... Signal (confirmation) Object), M ... Control part, D ... Display part, O ... Mark

Claims (6)

On your computer,
Defining a virtual space including a virtual viewpoint associated with the user;
Detecting the movement of the user's head;
Controlling the movement of the line of sight from the virtual viewpoint based on the movement of the user's head;
Displaying a view image corresponding to the view from the virtual viewpoint on a head-mounted display associated with the user;
Placing one or more target objects operated by at least one of the movement of the line of sight and an input of a controller in the virtual space;
Determining whether the movement of the line of sight with respect to the target object and the input of the controller satisfy a predetermined condition;
A program for running The program according to claim 1 , further causing the computer to execute a step of assigning a point associated with the user according to the result of the determination . The step of applying the points, the points assigned by the determination result to the movement of the sight line, including varying the, the points given by the determination result to the input of the controller, the program according to claim 2 . Wherein the predetermined condition and movement of the line of sight to the input of the controller that different respective program according to any one of claims 1 to 3. The program according to any one of claims 1 to 4 , wherein the plurality of target objects include target objects that are operated only by an input of the controller . The step of giving the point includes giving the point each time it is determined whether the movement of the line of sight and the input of the controller satisfy the predetermined condition,
The program according to claim 2 or 3 for causing a computer to further execute a step of summing up each point given to the user .

Images