JP2018037034A - Information processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To eliminate the sense of strangeness felt by a user because of aiming error by sensually matching an aiming direction supposed by the user on the basis of an attitude of a controller and an aiming direction actually pointed on the basis of a motion sensor of the controller regardless of variation and change in detection characteristics of the triaxial geomagnetic sensor.SOLUTION: A spatial pointing process includes a calibration process for calibrating a relationship between a reference point pointed by a controller, which is estimated by means of a first motion sensor, and a reference point pointed by a head-mounted gear, which is estimated by means of a second motion sensor, to match the two reference points.SELECTED DRAWING: Figure 13

Description

  The present invention relates to an information processing system that includes a controller in which a motion sensor is incorporated and a head-mounted device in which a motion sensor and a display are incorporated, and enables head tracking display and space pointing in a virtual space.

  The applicant previously has a controller in which a first motion sensor including at least a three-axis geomagnetic sensor is incorporated, and a head-mounted device in which a second motion sensor including at least a three-axis geomagnetic sensor and a display are incorporated. The posture information of the controller and the head-mounted device obtained from each of the first and second motion sensors is converted from local coordinates to world coordinates, and the head in the virtual space is converted based on the posture information on the world coordinates. A novel information processing system including an information processing unit that executes tracking display processing and spatial pointing processing has been proposed (see Patent Document 1).

Patent No. 5944600

  According to the information processing system described above, the posture of the controller or the head-mounted device detected via the motion sensor is converted from the local coordinate value to the world coordinate value, and then used to determine the pointing direction. Therefore, as long as the change in posture at that time is correctly reflected in the current posture, between the aiming direction intended by the controller's posture and the aiming direction in the virtual space detected via the motion sensor, In principle, no major sensory errors should occur.

  However, the triaxial geomagnetic sensor that constitutes the motion sensor incorporated in the controller and the head-mounted device has many variations in detection characteristics for each product, and the detection characteristics are also small depending on environmental conditions such as ambient temperature. Therefore, in this type of information processing system, it is indicated via the aiming direction and the motion sensor that the controller intends to indicate due to the variation and fluctuation in the detection characteristics of the above-mentioned three-axis geomagnetic sensor. It was found that there was a considerable sensory error between the indicated aiming directions.

  The present invention has been made paying attention to the above-mentioned problems, and the object of the present invention is to perform head tracking using the posture of the controller and the head-mounted device converted from local coordinates to world coordinates. In an information processing system that enables display and spatial pointing, variation in detection characteristics of the three-axis geomagnetic sensor indicates the aiming direction that is intended to be indicated through the attitude of the controller and the aiming direction that is indicated through the motion sensor of the controller. It is to match the senses regardless of fluctuations and to eliminate the operator's uncomfortable feeling due to the sight misalignment.

  Other objects and operational effects of the present invention should be easily understood by those skilled in the art by referring to the following description of the specification.

  It is considered that the above technical problem can be solved by an information processing system and a computer program having the following configurations.

  That is, the information processing system according to the present invention includes a controller in which a first motion sensor including at least a triaxial geomagnetic sensor is incorporated, a head mounted in which a second motion sensor including at least a triaxial geomagnetic sensor and a display are incorporated. And the posture information of the controller and the head-mounted device obtained from each of the first and second motion sensors, from local coordinates to world coordinates, and based on the posture information on the world coordinates, An information processing unit that performs a head tracking display process and a spatial pointing process in a virtual space, and the spatial pointing process includes a reference point in a direction indicated by a controller estimated via the first motion sensor and the first point Head wearing estimated via 2 motion sensors It contains calibration process of calibrating the relationship between the two to be consistent with the reference point in a direction indicated by the can.

  According to such a configuration, since the reference point of each estimated direction of the controller and the head-mounted device is matched by the action of the calibration process included in the space pointing process, it is indicated through the attitude of the controller. The intended aiming direction and the aiming direction indicated via the motion sensor of the controller are matched sensuously regardless of variations or fluctuations in the detection characteristics of the 3-axis geomagnetic sensor. Can be resolved.

  In a preferred embodiment, the calibration processing is estimated via the second motion sensor and the direction indicated by the controller estimated via the first motion sensor when a predetermined operation is performed. Correction value acquisition processing for acquiring a difference from the direction indicated by the head-mounted device as a correction value, and thereafter adding or subtracting the correction value with respect to the direction indicated by the controller and / or the head-mounted device. The correction process which correct | amends those directions to these may be included.

  According to such a configuration, when it is recognized that an error has occurred between the aiming direction in the virtual space and the aiming direction in the virtual space specified via the motion sensor, the controller performs a predetermined operation. By executing the correction value acquisition process and the correction process each time, the direction reference of the two motion sensors can be matched, and the operator's uncomfortable feeling due to misalignment of the aim can be eliminated.

  At this time, prior to the correction value acquisition process, operation guidance may be given to the user to indicate a target direction to be aimed through the attitude of the controller.

  According to such a configuration, user convenience can be improved by guiding the operation method of the controller necessary for starting the correction value acquisition process.

  At this time, the operation guidance may be directed to point the target mark through the attitude of the controller by drawing a target mark indicating the target direction on the screen of the display.

  According to such a configuration, in order to activate the correction value acquisition process, it is possible to reliably tell the user that the target mark has to be pointed through the attitude of the controller.

  At this time, if the target direction is a direction indicating the screen center of the display, the target direction can be easily taught to the user without drawing a target mark indicating the target direction on the display screen. it can.

  In a preferred embodiment, the calibration process may be executable at the start of executing any application program in the information processing system.

  According to such a configuration, it is possible to cope with the aiming deviation that originally exists due to variations in detection characteristics of each product of the three-axis geomagnetic sensor.

  In a preferred embodiment, the calibration process may be executed in response to a predetermined trigger operation at an arbitrary time after the start of execution of an arbitrary application program in the information processing system. Good.

  According to such a configuration, it is possible to cope with a misalignment of the aim due to the change over time in the detection characteristics of the triaxial geomagnetic sensor.

  In a preferred embodiment, the information processing unit is a smartphone incorporated in the head-mounted device and capable of communicating with the controller, and the second motion sensor and the display are incorporated in the smartphone. Motion sensors and displays.

  According to such a configuration, since it is only necessary to purchase a predetermined application program that operates on the smartphone in addition to the head-mounted device and the controller, this type of user can possess this type of smartphone. An information processing system can be realized at low cost.

  The present invention viewed from another aspect can also be grasped as a computer program for causing a smartphone to function as a smartphone incorporated in the information processing system described above.

  According to such a computer program, the information processing system of the present invention can be realized only by preparing a controller, a head-mounted device, and a smartphone.

  According to the present invention, the reference point of each estimated direction of the controller and the head-mounted device is matched by the action of the calibration process included in the space pointing process, and therefore it is intended to be indicated through the attitude of the controller. The aiming direction and the aiming direction indicated via the motion sensor of the controller are matched sensuously regardless of variations or fluctuations in the detection characteristics of the 3-axis geomagnetic sensor, thus eliminating the operator's uncomfortable feeling due to misalignment of the aiming. be able to.

FIG. 1 is an explanatory diagram showing a use state of the system. FIG. 2 is a perspective view showing an appearance of the head-mounted device. FIG. 3 is an explanatory diagram of a procedure for incorporating the smartphone into the head-mounted device. FIG. 4 is an explanatory diagram illustrating a usage state of the head-mounted device. FIG. 5 is a block diagram showing an electrical hardware configuration of the smartphone. FIG. 6 is a perspective view showing the appearance of the controller. FIG. 7 is a block diagram showing an electrical hardware configuration of the controller. FIG. 8 is a conceptual diagram of an omnidirectional panoramic image. FIG. 9 is a conceptual diagram showing the relationship between the local coordinate space and the world coordinate space. FIG. 10 is an explanatory diagram of an initial screen when an application program is started. FIG. 11 is an explanatory diagram showing an operation method during calibration. FIG. 12 is a flowchart showing preparation work for executing the application program. FIG. 13 is a general flowchart schematically showing the entire process. FIG. 14 is a flowchart (part 1) showing details of each process. FIG. 15 is a flowchart (part 2) showing details of each process. FIG. 16 is a detailed flowchart of the start time calibration process. FIG. 17 is a detailed flowchart of the post-start calibration process. FIG. 18 is a conceptual diagram of a horizontal 180 degree panoramic image. FIG. 19 is a conceptual diagram of a hemisphere panoramic image. FIG. 20 is an explanatory diagram illustrating a usage state of a system having an information processing unit independently.

  Hereinafter, a preferred embodiment of an information processing system according to the present invention will be described in detail with reference to the accompanying drawings.

<General configuration of information processing system>
As described above, the information processing system according to the present application includes a controller in which a first motion sensor including at least a triaxial geomagnetic sensor is incorporated, and a second motion sensor and display including at least a triaxial geomagnetic sensor. The posture information of the built-in head-mounted tool, the controller and the head-mounted tool obtained from each of the first and second motion sensors is converted from local coordinates to world coordinates, An information processing unit that performs a head tracking display process and a spatial pointing process in a virtual space based on the posture information, and the spatial pointing process includes a direction indicated by the controller estimated via the first motion sensor Estimated via the reference point and the second motion sensor A reference point in the direction indicated by the head mounting device which is included but contains calibration process of calibrating the relationship between the two to be consistent.

  The controller is configured as a hand-held type (hand-held type) or a body-mounted type (wearable type), and its external shape is a convenient shape for pointing in any direction, aiming, or aiming. It is preferable that

  The first motion sensor incorporated in the controller is preferably a combination of a triaxial geomagnetic sensor and a triaxial acceleration sensor, or a combination of a triaxial geomagnetic sensor, a triaxial acceleration sensor, and a triaxial angular velocity sensor. Those sensors are attached to the controller body in a fixed posture.

  In addition to the above-described first motion sensor, the controller processes one or more operators, and signals generated by the operations of those operators, and communicates with the outside. An information processing unit may be provided.

  Headgear type, glasses type, hat type, hairband type, etc., in other words, it is held by the head even when released, and supports the display so as to face the user's eyes If it is possible, there is no problem.

  As the second motion sensor incorporated in the head-mounted device, a combination of a three-axis geomagnetic sensor and a three-axis acceleration sensor, or a combination of a three-axis geomagnetic sensor, a three-axis acceleration sensor, and a three-axis angular velocity sensor Is preferred. Those sensors are attached to the head-mounted device in a fixed posture. As will be described later, those sensors incorporated in a portable high-performance information terminal called a so-called smartphone or the like can be substituted.

  The display incorporated in the head-mounted device (including the meaning of being attached) may be a display device alone, or may be substituted with a display screen of a portable high-performance information terminal called a so-called smartphone or the like. There may be. The display device may be a color liquid crystal device or a color organic EL device.

  The information processing unit is a general term for means having such a function. Specifically, one information processing unit (personal computer, game computer provided separately from the controller and the head-mounted device) is provided. Etc.), etc. (see FIG. 20), as will be described later, when a smartphone is incorporated in the head-mounted device, the information processing unit (control unit 501) of the smartphone substitutes for it. (See FIG. 5). Further, the function may be shared between an information processing unit (CPU) of a smartphone incorporated in the head and an information processing unit (CPU) incorporated in the controller.

  As is well known to those skilled in the art, the three-axis geomagnetic sensor is a type of magnetic sensor that detects the direction of geomagnetism and has three orthogonal axes (X axis, Y axis, Z axis). A sensor that is calculated by the value of (axis). That is, this sensor has a third geomagnetic sensor that detects the vertical geomagnetism in addition to the first and second magnetic sensors in the front-rear direction and the left-right direction. For this reason, for example, even when the electronic compass is tilted at an angle with the operating element, if it is determined how many times the electronic compass is tilted, the horizontal magnetism can be calculated by subtracting the tilt and the correct orientation can be displayed. There are mainly three types of magnetic sensor elements that are used in geomagnetic sensor ICs incorporated in triaxial geomagnetic sensors. They are MR (magneto-resistive) elements, MI (magneto-impedance) elements, and Hall elements. The MR element uses, for example, the MR effect of a soft metal magnetic material such as permalloy (NiFe) whose resistance value changes with the intensity change of the external magnetic field, and the MI element uses the MI effect of an amorphous wire whose impedance changes when the external magnetic field changes. . A pulse current is allowed to flow through the wire. The Hall element measures the change in the external magnetic field by detecting the potential difference caused by the Hall effect of the semiconductor. In addition to such magnetic sensor elements, the geomagnetic sensor IC generally incorporates a drive circuit, an amplifier circuit, and the like.

  As is well known to those skilled in the art, a three-axis acceleration sensor is a sensor that can measure the acceleration in the three directions of the X-axis, Y-axis, and Z-axis with a single device, and detects three-dimensional acceleration. It can also be used to measure gravity (static acceleration). Many 3-axis accelerometers apply microfabrication technology such as semiconductor manufacturing technology and laser processing technology to integrate micromechanical elements on a silicon substrate. “Micro Electro Mechanical Systems, MEMS” , Mems, Micromachine) ”. The MEMS triaxial acceleration sensor can be measured in a range of ± several g, and is called a “low g” type capable of following acceleration fluctuations from 0 Hz to several hundred Hz. In this case, 0 Hz is a state in which only gravitational acceleration is applied to the sensor, and the direction to the ground can be measured from the sum of the X-axis, Y-axis, and Z-axis acceleration vectors at this time. . There are three types of MEMS triaxial acceleration sensors: semiconductor piezoresistive triaxial acceleration sensors, capacitive triaxial acceleration sensors, and thermal detection (gas temperature distribution type) triaxial acceleration sensors. The measurement method is different. The semiconductor piezoresistive triaxial acceleration sensor measures the acceleration by detecting the distortion of the diaphragm generated when the acceleration acts on the weight. Capacitance type 3-axis acceleration sensor measures the change in capacitance for measuring acceleration, and heat detection type (gas temperature distribution type) 3-axis acceleration sensor measures the acceleration using the movement of gas heated by the heater To do.

  As is well known to those skilled in the art, the triaxial angular velocity sensor is a kind of inertial sensor that realizes measurement of rotational angular velocity with three orthogonal axes (X axis, Y axis, Z axis). Also called a sensor. Angular velocity sensors measure rotational movements that are not responsive to acceleration sensors. The gyro sensor can be classified by a method for detecting rotation. At present, vibration gyro sensors using IC type micro electro mechanical system (MEMS) technology are most commonly installed in consumer devices. As the name suggests, an inertial sensor using MEMS technology constitutes a sensor with a technology that combines an element that makes mechanical movement and an electronic circuit that processes the signal, and detects the movement. Among vibration-type gyro sensors, there are a capacitive type using silicon and a piezo type using crystal and other piezoelectric materials. Types other than the vibration type gyro sensor include a geomagnetic type, an optical type, and a mechanical type. The three axes are generally defined as three axes of up / down, left / right, and front / rear, and the up / down axis is often called a “yaw axis”, the left / right axis is called a “pitch (pitching) axis”, and the front / rear axis is often called a “roll axis”. All vibratory gyro sensors detect rotation using Coriolis force (turning force).

<About a specific example of an information processing system>
[System overall configuration]
FIG. 1 shows an explanatory diagram showing a use state of a more specific example of the information processing system according to the present invention. As shown in the figure, the information processing system 1 includes a hand-held controller 10 that is held by a user's 40 hand, a headgear-type head mounting tool 20 that is mounted on a head 41 of the user 40, It is comprised from the smart phone 50 (refer FIG. 3) integrated in the inside of the head mounting tool 20 so that the display screen 51 may oppose the user's 40 eyes. In the figure, reference numeral 21 denotes a head circumference belt for mounting the head mounting tool 20 on the user's head.

[Configuration of head-mounted device]
FIG. 2 is a perspective view showing the appearance of the head-mounted device, FIG. 3 is an explanatory diagram of a procedure for incorporating the smartphone into the head-mounted device, and FIG. 4 is an explanatory diagram showing the usage state of the head-mounted device. Has been. As shown in these figures, the head-mounted device 20 has an opening 24 on the front surface and a main body 23 that is applied to the eyes 43 of the user 40 on the rear surface. The main body 23 includes a head belt 21 applied to the periphery of the head of the user 40 and a head belt 22 applied to the top of the head. A lid plate 25 that opens outward is hinged to the front opening 24 so as to be openable and closable. Appropriate holding tools 26 for holding the smartphone 50 with the display screen 51 facing inward are provided on the four edges on the inner surface side of the cover plate 25. Further, a female fastener 27a is provided on one edge of the cover plate 25, and a male fastener 27b is provided on a corresponding one edge of the opening 24, and the cover plate 25 is provided via the fasteners 27a and 27b. Can hold the opening 24 in a closed state.

  In assembling the smartphone 50 into the head-mounted device 20, first, the smartphone 50 in which a necessary application program is installed is prepared in a state where the front cover plate 25 of the head-mounted device 20 is opened (FIG. 3A )reference). Next, the smartphone 50 is held on the inner surface side of the lid plate 25 via the four holders 26 (see FIG. 3B). After that, if the cover plate 25 is closed, the smartphone 50 is assembled into the head-mounted device 20 (see FIG. 1).

  When the user 40 wears the head-mounted device 20 in which the smartphone 50 is incorporated in the head 41, as shown in FIG. 4, in front of the user 40 through the optical system 29 held by the partition plate 28. Then, the display screen 51 of the smartphone 50 faces. Therefore, the display screen 51 of the display of the smartphone 50 functions as a so-called head mounted display (HMD).

[Configuration of smartphone]
As is well known, a smartphone has a display at the center of one surface, an operation button, a speaker, and an in-camera above and below, and an out-camera on the other surface. As will be described in detail later with reference to FIG. 5, the smartphone incorporates a triaxial acceleration sensor, a triaxial angular velocity sensor, and a triaxial geomagnetic sensor that constitute a motion sensor. Therefore, it is possible to freely detect changes including the position, posture, and / or movement of the smartphone 10 by these sensors.

  A block diagram showing an example of the electrical hardware configuration of the smartphone is shown in FIG. As shown in the figure, the electric circuit of the smartphone 50 includes a control unit 501, a storage unit 502, a display 503, an input operation unit (touch panel) 504, an input operation unit (button) 505, a triaxial acceleration sensor 506, and a triaxial unit. Angular velocity sensor 507, triaxial geomagnetic sensor 508, GPS unit 509, illuminance sensor 510, timer 511, battery 512, vibration unit 513, communication unit 514, audio output unit 515, audio input unit 516, speaker unit 517, camera unit 518, And a connector 519.

  The control unit 501 includes a CPU (including a SoC (System-on-chip), MCU (Micro_Control_Unit), or FPGA (Field-Programable_Gate_Array)) that executes a program for controlling various operations of the smartphone.

  The storage unit 502 includes ROM, RAM, and / or storage, and performs various types of application programs for realizing the general functions of the smartphone 50, as well as head tracking display processing and spatial pointing processing according to the present invention. Various application programs and data (for example, horizontal 180 degree image data (see FIG. 18), horizontal 360 degree image data, hemisphere image data (see FIG. 19), and omnidirectional image data (see FIG. 8) , Etc., various panoramic image data) are stored.

  The display 503 includes a display device such as a liquid crystal display, an organic EL display, or an inorganic EL display, and displays characters, figures, symbols, and the like. This display 503 also functions as a head mounted display (HMD) in connection with the present invention. That is, on the display screen of the display 503, panoramic images (for example, a horizontal 180 degree panoramic image (see FIG. 18), a horizontal 360 degree panoramic image, a hemispherical panoramic image (see FIG. 19), and a global The upper panoramic image (see FIG. 8), etc.) is displayed using a so-called head tracking display technique. This panoramic image is preferably subjected to 3D processing by a known method. As an example of a technique for performing 3D display, for example, a technique in which a horizontally long display screen of the display 503 is divided into left and right parts and two images each having a parallax are displayed is employed.

  The communication unit 514 performs wireless communication with an external device via a LAN or the Internet. Examples of wireless communication standards include 2G, 3G, 4G, IEEE 802.11, Bluetooth (registered trademark), and the like. In particular, in connection with the present invention, the smartphone 50 communicates with the controller 10 via Bluetooth (registered trademark).

  The speaker unit 517 outputs music or the like from the smartphone, and the speaker unit 517 is also used to generate various application-specific sounds such as game sounds in connection with the present invention.

  The triaxial acceleration sensor 506 detects the direction and magnitude of acceleration acting on the smartphone casing, and the triaxial angular velocity sensor 507 detects the angle and angular velocity of the smartphone casing, and the triaxial geomagnetic sensor 508. Detects the direction of geomagnetism. These three sensors 506, 507, and 508 constitute a second motion sensor in the present invention. Based on the outputs of the three sensors 506, 507, and 508, the posture and orientation of the head 41 of the user 40 are detected. The details of the sensors 506, 507, and 508 are as described above.

  The camera unit 518 is connected to, for example, an in camera or an out camera provided on the front surface or the back surface of the housing of the smartphone 50, and is used when capturing a still image or a moving image. As will be described later, the camera unit 518 captures an image of the real world using an out camera provided on the back side of the display 503 or is provided on the same surface as the display 503 in connection with the present invention. It can also be used to acquire a line-of-sight image by using the in camera.

  The voice input unit (such as a microphone) 516 is for inputting voice when making a call or the like using the smartphone 50. As will be described later in connection with the present invention, the voice input via the voice input unit 516 may be analyzed to perform various operations on the smartphone or application program.

  In addition, the vibration part 513 is a vibration part used by various application programs etc., for example, can be comprised with the vibration mechanism using an electromagnetic coil, various other vibration motors, etc. The connector 519 is a terminal used for connection with other devices. The terminal may be a general-purpose terminal such as USB or HDMI (registered trademark). The audio output unit 515 outputs audio when making a call or the like using the smartphone. The input operation unit 504 is, for example, a touch-type input operation unit, and performs various inputs by detecting contact with a finger, pen, stylus pen, or the like. A GPS (Gloval Positioning System) unit 509 detects the position of the smartphone. The illuminance sensor 510 detects illuminance. The illuminance indicates light intensity, brightness, or luminance. The timer 511 measures time. Note that the timer may be provided integrally with the CPU or the like. The battery 512 functions as a power source for the smartphone by charging.

[Controller configuration]
A perspective view showing the external appearance of the controller 10 is shown in FIG. As shown in the figure, the controller 10 has a substantially L-shape in a side view including a main body rear portion 11 having a substantially horizontal upper surface for button arrangement and a main body front portion 12 having a substantially vertical front surface for button arrangement. It has an external appearance form. The main body rear portion 11 also functions as a grip portion for gripping the controller.

  Three push buttons (center button 13, left button 14, and right button 15) each functioning as an operator of a momentary type push button switch are disposed on the upper surface of the main body rear portion 11. Two push buttons (an upper button 16 and a lower button 17) each functioning as an operator of a momentary type push button switch are arranged on the front surface of the main body front portion 12. Reference numeral 18 denotes a slide operator that functions as a power switch provided on the left side surface of the main body rear portion 11.

  As will be described later with reference to FIG. 7, the controller 10 includes a three-axis acceleration sensor 1003, a three-axis angular velocity sensor 1004, and a three-axis geomagnetic sensor 1005 that constitute the first motion sensor according to the present invention. It has been incorporated. Therefore, it is possible to freely detect changes including the position, posture, orientation, and / or movement of the controller 10 by these sensors. These sensors 1003 to 1005 are incorporated so as to have a certain physical relationship with the controller main body 10.

  A block diagram showing the electrical hardware configuration of the controller 10 is shown in FIG. As shown in the figure, an electric circuit inside the controller is configured by a microprocessor, an ASIC, and the like, and realizes a short-range communication such as Bluetooth (registered trademark) and the like with a control unit 1001 that performs overall control of the entire controller. Communication unit 1002, three sensors (three-axis acceleration sensor 1003, three-axis angular velocity sensor 1004, and three-axis magnetic sensor 1005) constituting the first motion sensor, and a storage unit 1007. . In the figure, reference numeral 1006 denotes a rechargeable battery serving as a power source for the controller.

  The above-described three buttons (center button 13, left button 14, right button 15) and two buttons (upper button 16, lower button 17) have contacts 13a to 17a that are turned on and off in conjunction with these operations. Have. On / off signals generated by turning on and off the contacts are read as various commands into the control unit 1001 and used for various calculations, and if necessary, incorporated into the head-mounted device 20 via the communication unit 1002. Sent to the smartphone 50 via short-range communication, and used for various calculations on the smartphone side. In addition, the outputs of the three sensors (three-axis acceleration sensor 1003, three-axis angular velocity sensor 1004, and three-axis magnetic sensor 1005) constituting the first motion sensor are also directly processed or appropriately processed. After that, transmission to the smartphone 50 side via the communication unit 1002 is possible.

[About head tracking display processing and spatial pointing processing]
As described above, in the system of the present invention, the controller 10 incorporating the first motion sensor including at least the triaxial geomagnetic sensor 1005, the second motion sensor including at least the triaxial geomagnetic sensor 508, and The head mounting device 20 incorporating the display 503, the controller 10 obtained from each of the first and second motion sensors, and the posture information of the head mounting device 20 are converted from local coordinates to world coordinates, and the world And an information processing unit that executes a head tracking display process and a space pointing process in the virtual space based on each posture information on the coordinates. This point will be schematically described below with reference to the coordinate system conceptual diagram of FIG. 9 and the flowcharts of FIGS.

  Assume that a smartphone (HMD) 50 and a controller (CTRL) 10 exist on one world coordinate space (X, Y, Z), and that each device has a built-in motion sensor. To do. Here, each of the motion sensors includes a triaxial acceleration sensor, a triaxial angular velocity sensor, and a triaxial geomagnetic sensor. In any device, the basic posture can be obtained by calculation using a triaxial acceleration sensor and a triaxial geomagnetic sensor (see steps 201 and 204 in FIG. 13).

First, the attitude of the smartphone (HMD) 50 and the attitude of the controller (CTRL) 10 are calculated using the triaxial acceleration sensor and the triaxial angular velocity sensor incorporated in each device, and these are calculated as the basic attitude of each device. (Refer to Steps 201 and 204 in FIG. 13 and their detailed contents, Steps 301 to 303 and 304 to 306 in FIG. 14).
At this time, the basic posture obtained from the value of the triaxial acceleration sensor is described as follows according to the program language. The value of the three-axis acceleration sensor is divided into a stationary component and a motion component through a low-pass filter and / or a high-pass filter, and what is acquired here is the stationary component.

sensor_pose = Quaternion (-_ mAccel.Pose.z, _mAccel.Pose.y, -_mAccel.Pose.x,
_mAccel.Pose.w);

Next, by correcting the basic attitude described above using the detection result of the triaxial angular velocity sensor, the attitude when each device is moved is obtained (steps 202 and 205 in FIG. 13 and their detailed contents). 14 steps 401-402, 403-404). Even if a triaxial angular velocity sensor is not used, posture control is possible if a triaxial acceleration sensor and a triaxial geomagnetic sensor are provided.

At this time, the change value correction by the triaxial angular velocity sensor with respect to the previous posture is as follows.

Quaternion rotation = this.transform.rotation;
Quaternion qx = Quaternion.AngleAxis (-gyro.RotZ, Vector3.right);
Quaternion qy = Quaternion.AngleAxis (-gyro.RotY, Vector3.up);
Quaternion qz = Quaternion.AngleAxis (gyro.RotX, -Vector3.forward);
q = (qz * qy * qx);
rotation = rotation * q;
Quaternion sensor_pose = clSingleton <clBLESensorManager>.Instance.SenserPose;
this.transform.rotation = Quaternion.Slerp (rotation, sensor_pose, 0.04f);

Next, a series of processing for converting the basic posture into world coordinates is performed (see steps 203 and 206 in FIG. 13 and steps 501 to 506 and 507 to 512 in FIG. 15 which are detailed contents thereof). In short, the series of processes described above mainly uses a triaxial acceleration sensor and a triaxial geomagnetic sensor to calculate the magnetic north direction of each device from the triaxial geomagnetic sensor. At this time, since the magnetic north calculation of the controller 10 includes the tilt information in the component of the triaxial geomagnetic sensor, the tilt of the terminal obtained from the acceleration is corrected to the geomagnetic sensor information, and the horizontal geomagnetic component is extracted.

The gravity vector of the three-axis acceleration sensor is as follows (see steps 501 and 507 in FIG. 15).

Vector3 b = Vector3 (_mAccel.Gravity.x, _mAccel.Gravity.y, _mAccel.Gravity.z);
b = b.normalized;

The vertically downward vectors are as follows (see steps 502 and 508 in FIG. 15).

Vector3 a = Vector3 (0f, 1f, 0f);
a = a.normalized;
float value = ax * bx + ay * by + az * bz;
float gravity_angle = Mathf.Acos (value / a.magnitude * b.magnitude);
Vector3 axis = Vector3.Cross (a, b);
axis = axis.normalized;
Quaternion aq = Quaternion.AngleAxis (-gravity_angle * Mathf.Rad2Deg, axis);

In the following manner, the geomagnetic axes (mGeom.Z, _mGeom.X, _mGeom.Y) are rotated so that (0, 1, 0) is on the top (see steps 503 and 509 in FIG. 15).

Vector3 geom_v = new Vector3 (-_ mGeom.Z, _mGeom.Y, -_mGeom.X);
geom_v = geom_v.normalized;

The acceleration gradient is aligned with the geomagnetic axis as follows (see steps 504 and 510 in FIG. 15).

Quaternion gq = Quaternion (-aq.z, -aq.x, aq.y, aq.w);

The horizontal magnetic north is calculated by further decomposing geomagnetic x, y, z as follows (see steps 505 and 511 in FIG. 15).

float m = Mathf.Sqrt (Mathf.Pow (geom.x, 2) + Mathf.Pow (geom.y, 2) +
Mathf.Pow (geom.z, 2));
geom_height = Mathf.Asin (geom.z / m);

The magnetic north calculated from the horizontal component of geomagnetism is as follows.

geom_height_head = Mathf.Acos (geom.x / (m * Mathf.Cos (geom_height)));

The magnetic north direction of each device and the basic posture of each device are synthesized by performing rotation calculation from the direction and the reference posture as follows (see steps 506 and 512 in FIG. 15). At this point, the orientation of each object in the world space and the direction it faces are known.

sensorRotation = q_head * pose;

  Subsequently, returning to FIG. 13, information of the smartphone (HMD) 50 and the controller (CTRL) 10 is arranged on the world coordinates as shown in the conceptual diagram of FIG. 9 (see step 207 of FIG. 13).

  Subsequently, the viewpoint of the camera is placed on the world coordinates at the same position / tilt posture as the smartphone (HMD) (see step 208 in FIG. 13). Here, the camera is a virtual camera that captures a virtual space at a predetermined position (for example, the center position) in the virtual space (for example, the spherical panoramic virtual space shown in FIG. 8), An image displayed in the field of view is cut out from the virtual space by a separate process and displayed on the display (HMD) of the smartphone, thereby enabling head tracking display (background display) in the so-called virtual space.

  Subsequently, world coordinates of a fixed distance in the forward direction of the current posture are calculated from the posture information of the controller 10 (see step 209 in FIG. 13).

When the end point calculated above is present on the field of view of the camera corresponding to the display screen 51 of the smartphone 10 (step 210 YES in FIG. 13), the world coordinates are converted into camera viewport coordinates (range: 0.0 to 1.0). The coordinates of the 2D coordinate system can be used as the 2D coordinates of the aim of the controller on the screen 51 of the display of the smartphone 50, and the aim mark 53 can be displayed on the screen 51 as necessary (step of FIG. 13). 211). If it is outside the range of the viewport coordinates (range: 0.0 to 1.0) (step 210 NO in FIG. 13), the aiming position is outside the screen of the smartphone display, and the aiming mark 53 is not displayed on the screen. As described above, space pointing in a so-called virtual space is possible.

Vector3 ray_s_pos = gameObject.transform.position;
gunRay.SetPosition (0, ray_s_pos);
Vector3 ray_e_pos = gameObject.transform.forward * kRayLength;

The process of converting the end point position into viewport coordinates (0.0 to 1.0) of the current camera is as follows.

Vector2 v2 = ViewPortPos (ray_e_pos);

Further, the processing for converting the viewport coordinates to the coordinate system of the center 0 and correcting is as follows.
.
Vector3 target_pos = _basePos + new Vector3 ((v2.x-0.5f), (v2.y-0.5f), 0.0f);

Furthermore, the process of moving the aim toward the next position calculated from the current position is as follows.

this.transform.position = Vector3.MoveTowards (this.transform.position, target_pos, 0.5f);

  Note that the processing shown in FIGS. 13 to 16 is focused on the head tracking display processing in the virtual space and the spatial pointing processing in the virtual space, and therefore the operation unique to the application program (matching matching between the aim and the target). Note that subsequent event details etc. are omitted.

[Aiming direction calibration process]
Next, the aiming direction calibration process which is the main part of the present invention will be described. According to the information processing system described above, the posture of the controller 10 detected via the first motion sensor is used for determination of the pointing direction after being converted from a local coordinate value to a world coordinate value. As long as the change in posture at that time is correctly reflected in the current posture by a predetermined update process (see step 205 in FIG. 13 and steps 403 to 404 in FIG. 14 which are their detailed contents), the posture of the controller In principle, no large sensory error should occur between the aiming direction intended to be pointed by and the aiming direction pointed (estimated) via the motion sensor.

  However, the triaxial geomagnetic sensor (see reference numeral 1005 in FIG. 7) constituting the first motion sensor incorporated in the controller 10 has a considerable variation in magnetic north detection characteristics for each product, and the ambient temperature, etc. The magnetic north detection characteristics fluctuate depending on the environmental conditions. Therefore, even in this type of information processing system, we intended to point to the controller due to the variation and fluctuations in the magnetic north detection characteristics of the three-axis geomagnetic sensor described above. It is recognized that there is a considerable sensory error between the aiming direction detected by the motion sensor and the aiming direction detected by the motion sensor, so that a so-called aiming error occurs and the operator feels uncomfortable.

  Therefore, the present inventors have made it possible to execute a predetermined calibration process at the start of execution of the application program and to detect the magnetic north detection characteristics depending on environmental conditions such as the ambient temperature. These problems are solved by making it possible to execute a predetermined calibration process at an arbitrary time during the execution of the application program.

[Calibration process at the start of application program execution]
FIG. 12 is a flowchart showing preparation work for executing the application program. As shown in the figure, an arbitrary application program (including a head tracking display process and a spatial pointing process) is executed in an information processing system composed of a controller 10, a smartphone 50, and a head-mounted device 20. In order to do this, the user performs the following preparatory work.

  That is, the user first performs communication settings (for example, Bluetooth (registered trademark) communication settings) between the smartphone 50 and the controller 10 (step 101). Subsequently, the user performs an operation of selecting an application program installed in advance on the smartphone 50 to start the program (step 202). Subsequently, the user incorporates the smartphone 50 into the head-mounted device 20 according to the incorporation procedure shown in FIG. 3 (step 103). Finally, the user wears the head wearing tool 20 in which the smartphone 50 is incorporated on the head.

  When the above preparatory work is completed, a series of processes shown in FIGS. 13 to 15 are executed, and the reference point in the direction indicated by the controller 10 estimated via the first motion sensor and the second motion sensor are set. A calibration process for calibrating the relationship between the head wear tool 20 and the reference point in the direction indicated by the head wear tool 20 is executed. In this example, this calibration process is executed as a start time calibration process (step 212).

  A detailed flowchart of the start time calibration process (step 212) is shown in FIG. In the figure, when the process is started, first, it is determined whether or not the state of the calibrated flag is ON (step 601). This calibrated flag is set to OFF in advance by an initialization process (not shown). Therefore, the determination result is negative (NO in step 601), and subsequently, an operation guidance process (step 602) for guiding the user to the operation method for calibration is executed. This operation guidance process (step 602) is continuously executed until the user executes a predetermined trigger operation (NO in step 603).

  For operation guidance, a predetermined guidance display (characters, symbols, figures, sentences, etc.) is performed on the display screen 51 of the smartphone 50, a predetermined guidance voice is played from the speaker unit 517 of the smartphone 50, and further guidance is provided. It can be realized by various methods such as executing both display and guidance voice.

  The contents of the guidance display include, for example, a target mark indicating a target direction (for example, a direction toward the center of the screen) in the virtual space to be pointed through the posture of the controller 10, and posture information on the world coordinates of the controller 10. An aiming mark indicating the aiming direction in the virtual space pointed to based thereon, an operation guidance sentence such as “Please press the trigger button with the controller facing the center of the screen”, and the like can be cited. The same applies to the guidance voice, and examples include an operation guidance voice such as “please press the trigger button with the controller facing the center of the screen”.

  FIG. 10 shows an explanatory diagram of an initial screen at the start of the application program (that is, an example of guidance display). In this example, on the display screen 51, a target mark 52 indicating a target direction to be pointed by the controller 10, and a motion sensor (three-axis acceleration sensor 1003, three-axis angular velocity sensor 1004) incorporated in the controller 10. An aiming mark 53 indicating the aiming direction of the controller 10 determined based on the attitude of the controller 10 detected via the triaxial geomagnetic sensor 1005) is displayed.

  In this example, the direction toward the center of the screen is selected as the target direction, and therefore the target mark 52 is displayed at the center position of the display screen 51 in the vertical and horizontal directions. The aiming mark 53 moves according to the attitude or orientation of the controller 10, and the aiming mark 53 in the illustrated example indicates a state when the controller 10 is aimed at the screen center position. As is clear from the figure, when the controller 10 is aiming at the target mark 52 at the center position of the screen, the aim mark 53 is displayed at a position slightly shifted to the upper right from the center position of the screen. This means that there is an error between the aiming direction that the controller intends to point to and the aiming direction pointed to through the motion sensors (including triaxial geomagnetic sensors) in the controller. This error is considered to be caused by variations in magnetic north detection characteristics among products of the triaxial geomagnetic sensor 1005 in the controller 10 and the triaxial geomagnetic sensor 508 in the smartphone 50.

  In such a state, as shown in FIG. 11, the user is predetermined in a state where the head-mounted device 20 is directed to the front and the controller 10 is directed to the center of the screen 51 (that is, the position of the target mark 52). Perform the trigger operation.

  Note that a user who wears the head-mounted device 20 on the head can visually recognize the target mark 52 on the display screen 51 but cannot visually recognize the posture and orientation of the controller 10 at his / her hand. However, although there are individual differences, in general, a person turns the controller 10 toward the target mark 52 on the display screen 51 by unconsciously changing the posture of the controller 10 at hand through the experience cultivated through experience. (Ie, aiming).

  The predetermined trigger operation may be arbitrarily set by appropriately combining one or two or more pressing (simultaneous pressing) or pressing time of the five buttons 13 to 17 provided in the controller 10. it can.

  When the trigger operation is performed, the direction (θ1, φ1) indicated by the controller 10 derived from the basic posture on the world coordinates detected via the first motion sensors (1003 to 1005) and the second motion sensor (506) Correction value acquisition processing (step 604) for acquiring, as a correction value, a difference from the direction (θ2, φ2) indicated by the head-mounted device (smartphone) 20 derived from the basic posture on the world coordinates detected through ˜508). ) Is executed. Specifically, as the correction value acquisition process (step 604), for example, the difference between the aim position SIGHT_pos (see step 208 in FIG. 13) and the center position of the camera field of view Cam_view (step 210 in FIG. 13) is used as the correction value. You may make it memorize | store (acquire). This is merely an example of a correction value. More generally speaking, as the correction value acquisition process (step 604), in short, paying attention to some two pieces of information used for calculation to represent the target direction and the aiming direction, the difference is calculated. Those skilled in the art will easily understand that the correction value may be obtained.

  The trigger operation is performed because the head-mounted device 20 is directed to the front and the controller 10 is also directed to the front (center of the screen). Since the estimated orientation (θ2, φ2) of the head-mounted device 20 should match, the difference between them corresponds to the so-called misalignment of the aim.

  When the correction value acquisition process (step 604) ends, the correction process (step 605) is subsequently executed. In this correction process (step 605), the estimated values indicated by the controller 10 and / or the motion sensor of the head-mounted device 20 are indicated by adding or subtracting correction values thereafter. Correct the direction. As a result, the reference points in the respective estimated directions of the controller 10 and the head-mounted device 20 are aligned, and therefore the pointing direction intended to be indicated through the attitude of the controller 10 is indicated via the motion sensor of the controller 10. It is possible to eliminate the uncomfortable feeling of the operator due to the misalignment of the sight, by matching the sighting direction to be sensed regardless of variations or fluctuations in the detection characteristics of the triaxial geomagnetic sensor.

  Subsequently, after the content of the calibrated flag is set to ON (step 606), the process ends. In the subsequent execution cycle, since the calibrated flag is determined to be ON in step 601 (step 601 YES), the operation guidance process (step 602), the correction value acquisition process (step 604), and the correction process (step 605) are performed. Because it is skipped, it is never executed. In this correction process (step 605), the basic postures of the controller 10 and the head-mounted device (smartphone) 20 are also corrected using the correction values described above.

  Thereafter, as a result of executing a series of processes (steps 205 to 211) using the corrected basic posture, the deviation of the aiming position SIGHT_pos is corrected, and thus, in the real space indicated through the posture of the controller 10. The target direction and the aiming direction in the virtual space indicated based on the posture information on the world coordinates of the controller 10 can be accurately matched regardless of the variation in the detection characteristics of the three-axis geomagnetic sensor for each product. It becomes.

[Aiming direction calibration process executed at an arbitrary point in the application program]
As described above, it has been recognized that the reason why the aim of the controller 10 is distorted may be a change with time of the triaxial geomagnetic sensor 1005 due to environmental conditions such as ambient temperature. In this case, even if there is no error at the time of use disclosure, the error occurs gradually during use. Therefore, the above-described calibration process at the start is not sufficient. This problem can be solved by the following start-up calibration process.

  A detailed flowchart of the calibration process after the start is shown in FIG. This process is started at an arbitrary time by a predetermined trigger operation. As the trigger operation, for example, an operation of simultaneously pressing and long-pressing two specific buttons among the five buttons 13 to 17 provided in the controller 10 is selected. Such a button operation is extremely rare during normal times.

  In the figure, when the process is started by an interruption by the trigger operation described above, first, an operation guidance process (step 701) is executed, and a screen having the same contents as the screen shown in FIG. 10 is displayed.

  In this state, the user keeps pressing the controller 10 toward the target mark 52 while continuing the long press state. Then, the timer activation process (step 702), the button press continuation determination process (step 703), and the time-up determination process (step 704) are executed, and as a result, two buttons previously assumed to be simultaneously pressed are performed. The correction value acquisition process (step 705) is executed only in the case where the correction value is obtained (YES in steps 703 and 704).

  The correction value acquisition process (step 705) is the same as the correction value acquisition process (step 604) described above with reference to FIG. 16, and the first motion sensors (1003 to 1005) are used. It is derived from the direction (θ1, φ1) indicated by the controller 10 derived from the basic posture on the world coordinates detected via the second coordinate system and the basic posture on the world coordinates detected via the second motion sensors (506 to 508). A correction value acquisition process (step 604) for acquiring a difference from the direction (θ2, φ2) indicated by the head-mounted device (smartphone) 20 as a correction value is executed. Specifically, as the correction value acquisition process (step 604), for example, the difference between the aim position SIGHT_pos (see step 208 in FIG. 13) and the center position of the camera field of view Cam_view (step 210 in FIG. 13) is used as the correction value. You may make it memorize | store (acquire). This is merely an example of a correction value. More generally speaking, as the correction value acquisition process (step 604), in short, paying attention to some two pieces of information used for calculation to represent the target direction and the aiming direction, the difference is calculated. Those skilled in the art will easily understand that the correction value may be obtained.

  As a result, the reference points in the respective estimated directions of the controller 10 and the head-mounted device 20 are aligned, and therefore the pointing direction intended to be indicated through the attitude of the controller 10 is indicated via the motion sensor of the controller 10. It is possible to eliminate the uncomfortable feeling of the operator due to the misalignment of the sight, by matching the sighting direction to be sensed regardless of variations or fluctuations in the detection characteristics of the triaxial geomagnetic sensor.

  When the correction value acquisition process (step 705) ends, the correction process (step 706) is executed in the same manner as before. That is, in this correction process (step 605), by adding or subtracting a correction value to the estimated direction indicated through the motion sensor of the controller 10 and / or the head-mounted device 20 thereafter, Correct the direction indicated by. As a result, the reference points in the respective estimated directions of the controller 10 and the head-mounted device 20 are aligned, and therefore the pointing direction intended to be indicated through the attitude of the controller 10 is indicated via the motion sensor of the controller 10. It is possible to eliminate the uncomfortable feeling of the operator due to the misalignment of the sight, by matching the sighting direction to be sensed regardless of variations or fluctuations in the detection characteristics of the triaxial geomagnetic sensor.

  According to the above-described post-start calibration process, when it is recognized that the aim of the controller 10 has become unstable due to environmental conditions such as the ambient temperature after the start of use of the information processing system, a predetermined trigger operation (for example, 2 The aiming calibration can be performed so that the aiming direction in the real space coincides with the aiming direction in the virtual space.

  In addition, although the example which displays both the target mark 52 and the aiming mark 53 was shown in the above operation guidance process (steps 602 and 701), these are not essential. For example, although the target mark 52 is displayed, the display of the aiming mark 53 can be omitted. Further, if the target position is determined to be the center of the screen, anyone can easily recognize the center of the screen, and therefore the display of the target mark 52 can be omitted.

  According to the present invention, the reference point of each estimated direction of the controller and the head-mounted device is matched by the action of the calibration process included in the space pointing process, and therefore it is intended to be indicated through the attitude of the controller. The aiming direction and the aiming direction indicated via the motion sensor of the controller are matched sensuously regardless of variations or fluctuations in the detection characteristics of the 3-axis geomagnetic sensor, thus eliminating the operator's uncomfortable feeling due to misalignment of the aiming. be able to.

DESCRIPTION OF SYMBOLS 1 Information processing system 10 Controller 11 Main body rear part 12 Main body front part 13 Center button 14 Left button 15 Right button 16 Upper button 17 Lower button 18 Slide operator 20 Head wearing tool 21 Head circumference belt 22 Head belt 23 Main body 24 Opening 25 Cover plate 26 Holder 27a Female fastener 27b Male fastener 28 Partition plate 29 Lens 40 Human 41 Head 43 Eye 44 Hand 50 Smartphone 51 Screen 52 Target mark 53 Aim mark 501 Control unit 502 Storage unit 503 Display 504 Input operation Part (touch panel)
505 Input operation unit (button)
506 Triaxial acceleration sensor 507 Triaxial angular velocity sensor 508 Triaxial geomagnetic sensor 509 GPS
510 Illuminance sensor 511 Timer 512 Battery 513 Vibration unit 514 Communication unit 515 Audio output unit 516 Audio input unit 517 Speaker unit 518 Camera unit 519 Connector

Claims (9)

A controller incorporating a first motion sensor including at least a three-axis geomagnetic sensor;
A head mounted device incorporating a second motion sensor and display including at least a triaxial geomagnetic sensor;
The posture information of the controller and the head-mounted tool obtained from each of the first and second motion sensors is converted from local coordinates to world coordinates, and in the virtual space based on the posture information on the world coordinates. An information processing unit that performs head tracking display processing and spatial pointing processing;
The spatial pointing process includes a reference point in the direction indicated by the controller estimated via the first motion sensor, and a reference point in the direction indicated by the head-mounted device estimated via the second motion sensor. An information processing system that includes a calibration process that calibrates the relationship between the two so that they match. The calibration process is
When a predetermined operation is performed, the difference between the direction indicated by the controller estimated via the first motion sensor and the direction indicated by the head-mounted device estimated via the second motion sensor is corrected. Correction value acquisition processing to be acquired as a value;
2. The information according to claim 1, further comprising: a correction process for correcting the direction indicated by the controller and / or the head-mounted tool by subsequently adding or subtracting the correction value. Processing system.   The information processing system according to claim 2, wherein prior to the correction value acquisition process, operation guidance is provided to a user to indicate a target direction to be aimed through the attitude of the controller.   The information processing according to claim 3, wherein the operation guide is directed to point the target mark through a posture of the controller by drawing a target mark indicating the target direction on the screen of the display. system.   The information processing system according to claim 2, wherein the target direction is a direction indicating a screen center of the display.   The information processing system according to claim 1, wherein the calibration process can be executed at a start time when an arbitrary application program is executed in the information processing system.   2. The information processing according to claim 1, wherein the calibration process can be executed in response to a predetermined operation at an arbitrary time after the start of executing an arbitrary application program in the information processing system. system. The information processing unit is a smartphone incorporated in the head-mounted device and capable of communicating with the controller, and the second motion sensor and the display are a motion sensor and a display incorporated in the smartphone. The information processing system according to any one of claims 1 to 9.   A computer program for causing a smartphone to function as a smartphone incorporated in the information processing system according to claim 9.

Images