JP5757791B2 - Input system, head-mounted display device, information terminal device, and program - Google Patents

Description

  The present invention relates to an input system, a head-mounted display device, an information terminal device, a program, and the like.

  In recent years, HMD (Head Mounted Display) has been used as a display device. However, it is not preferable to increase the size of the device because of the characteristic that the HMD is worn on the head. Therefore, an operation unit (for example, an operation button) for operating the HMD has a simple configuration, and it is difficult to perform a complicated operation.

  In addition, an interface device that performs an operation based on a user's operation without using an operation unit such as a button is widely used. For example, an interface for recognizing a user's operation using sensor information of a sensor worn by the user, information on a captured image obtained by capturing the user, and the like, and executing an operation instruction associated with the recognized operation can be considered. However, when recognizing a motion using a sensor, if the sensor state changes (for example, when the sensor mounting position or mounting direction changes), the user's motion is recognized unless initialization processing is performed. There is a problem that cannot be performed properly.

JP 2000-353046 A JP 2010-231290 A

  In Patent Document 1, a first sensor attached to a first part (for example, a head) of a user's body, and a first part from a second sensor attached to a second part (for example, a hand). There has been proposed an interface device that generates action information by obtaining a relative position of a second part and determines a user instruction corresponding to the generated action information. However, Patent Document 1 is a user interface device mainly used for games, and does not describe selection of menu items on a menu screen displayed on a head-mounted display device. Therefore, there is no mention of initialization in selecting a menu item or the like.

  Patent Document 2 proposes an input device that detects a joystick-like movement of the head and operates a device based on the detected movement (for example, selects a menu item displayed on a display unit or the like). However, Patent Document 2 describes an initialization process in response to detection of a change in posture (standing position, sitting position, etc.), but does not describe initialization in menu item selection or the like. That is, there is a possibility that it is not possible to cope with the case where the head is not facing the front direction at the start of the operation.

  According to some aspects of the present invention, the movement of the head is detected based on sensor information from sensors provided in the head-mounted display device and the information terminal device, respectively, and based on the detected head movement. Therefore, it is possible to provide an input system for obtaining user input information, a head-mounted display device, an information terminal device, a program, and the like.

  In addition, according to some aspects of the present invention, the movement of the head is appropriately detected by updating the reference value of the sensor information at the initialization timing, and the user input information based on the detected movement of the head An input system, a head-mounted display device, an information terminal device, a program, and the like can be provided.

  One aspect of the present invention is the first sensor information from the first sensor provided in the head-mounted display device, and the information terminal device provided in the information terminal device that performs communication connection with the head-mounted display device. A sensor information acquisition unit that acquires second sensor information from two sensors, a storage unit that stores reference value information that is information about a reference value of the first sensor information and the second sensor information, and Based on reference value information, the first sensor information, and the second sensor information, first direction information that is direction information of the head-mounted display device and second information that is direction information of the information terminal device. And a processing unit that obtains user input information based on the acquired first direction information and second direction information.

  In one aspect of the present invention, based on sensor information of a sensor provided in a head-mounted display device, sensor information of a sensor provided in an information terminal device, and reference value information, information on a head direction and the information terminal device Get direction information. By obtaining user input information based on the acquired direction information, an interface based on the relative movement of the head with respect to the terminal can be realized.

  Further, the storage unit stores the first sensor information and the second sensor information acquired by the sensor information acquisition unit when the processing unit determines that it is an initialization timing, the reference value information May be stored as

  As a result, the reference value can be initialized, and the head direction and the like can be obtained appropriately even if the sensor state (mounting position, mounting direction, etc.) changes.

  In addition, the processing unit includes first state information in a first state that is a state of the information terminal device at a first timing when the reference value information is stored in the storage unit, and a current determination timing. A comparison process with the second state information in the second state which is the state of the information terminal apparatus at the second timing is performed, and it is determined that the state of the information terminal apparatus has changed due to the comparison process. In this case, the second timing may be determined as the initialization timing.

  As a result, when the state of the information terminal device changes, it is possible to perform initialization processing and the like.

  Further, the processing unit is information relating to an inclination of the information terminal device with respect to a direction of gravity at the first timing as the first state information by an acceleration sensor provided in the information terminal device. In addition to acquiring tilt information, as the second state information, second tilt information that is information related to the tilt of the information terminal device with respect to the direction of gravity at the second timing is acquired, and the first tilt Information and the second inclination information are compared, and when it is determined that the state of the information terminal device has changed by the comparison process, the second timing is determined to be the initialization timing. Also good.

  This makes it possible to determine a change in the state of the information terminal device using the acceleration sensor.

  The processing unit may be configured such that an i-th (i is an integer of 1 ≦ i ≦ N) surface in a reference direction among first to Nth (N is an integer of 2 or more) surfaces constituting the information terminal device. If it is facing, it is determined based on the first inclination information, and the jth surface (j is an integer of 1 ≦ j ≦ N, i ≠ j) among the first to Nth surfaces is the reference. If it is directed to the direction, it is determined that the state of the information terminal device has changed when it is determined based on the second inclination information, and the second timing is determined to be the initialization timing. Also good.

  Thereby, it becomes possible to identify the surface of the information terminal device facing the reference direction based on the sensor information of the acceleration sensor, and to determine the change in the state of the information terminal device based on the identified surface. It is possible to determine the state change.

  Further, the processing unit may determine the timing at which an initialization operation is performed by the user as the initialization timing.

  Thereby, the initialization process based on a user's operation instruction becomes possible.

  The head-mounted display device includes a display unit that displays a selection menu screen in which a plurality of icons are displayed side by side along a first direction, and the processing unit is arranged in the first direction. When it is determined that the amount of change in the first direction information with respect to the second direction information in the direction along the direction is larger than a given threshold, the icon selected on the selection menu screen is the plurality of icons. You may perform the process changed to the other icon among icons.

  Thereby, processing based on the movement of the head in the direction in which icons are displayed side by side becomes possible.

  In addition, when the processing unit determines that the amount of change in the first direction information with respect to the second direction information in a second direction different from the first direction is greater than a given threshold value. In addition, the selected icon may be confirmed.

  Thereby, processing based on the movement of the head in a direction different from the display direction in which icons are arranged can be performed.

  The first sensor is a first azimuth sensor provided in the head-mounted display device, and the second sensor is a second azimuth sensor provided in the information terminal device, The sensor information acquisition unit acquires first azimuth information that is azimuth information of the head-mounted display device as first sensor information, and azimuth information of the information terminal device as second sensor information. A certain second azimuth information may be acquired, and the processing unit may obtain the input information based on azimuth difference information that is information representing a difference between the first azimuth information and the second azimuth information. Good.

  Thereby, the process based on the movement of the head in the horizontal direction acquired by the azimuth sensor becomes possible.

  The head-mounted display device includes a display unit that displays a selection menu screen in which a plurality of icons are displayed side by side along the horizontal direction, and the processing unit includes a direction along the horizontal direction. When the orientation difference represented by the orientation difference information is determined to be larger than a given threshold, the icon selected on the selection menu screen is changed to another icon of the plurality of icons You may perform the process to do.

  This makes it possible to display a selection menu screen in which icons are arranged in the horizontal direction and perform icon selection processing based on the movement of the head in the horizontal direction.

  In addition, when the information indicating the difference between the elevation angle information of the head-mounted display device and the elevation angle information of the information terminal device is the elevation angle difference information, the processing unit uses the elevation angle difference information in the vertical direction. When it is determined that the represented elevation angle difference is larger than a given threshold value, the selected icon may be determined.

  This makes it possible to perform icon determination processing based on the movement of the head in the vertical direction on the selection menu screen in which icons are arranged in the horizontal direction.

  The first sensor is a first elevation angle sensor provided in the head-mounted display device, and the second sensor is a second elevation angle sensor provided in the information terminal device, The sensor information acquisition unit acquires first elevation angle information, which is elevation angle information of the head-mounted display device, as first sensor information, and uses elevation angle information of the information terminal device as second sensor information. Even if the second elevation angle information is obtained, the processing unit obtains the input information based on elevation angle difference information that is information representing a difference between the first elevation angle information and the second elevation angle information. Good.

  Thereby, processing based on the movement of the head in the vertical direction acquired by the elevation sensor can be performed.

  The head-mounted display device includes a display unit that displays a selection menu screen in which a plurality of icons are displayed side by side along the vertical direction, and the processing unit includes a direction along the vertical direction. When the elevation angle difference represented by the elevation angle difference information is determined to be larger than a given threshold, the icon selected on the selection menu screen is changed to another icon of the plurality of icons You may perform the process to do.

  As a result, a selection menu screen in which icons are arranged in the vertical direction is displayed, and an icon selection process based on the movement of the head in the vertical direction can be performed.

  Further, the processing unit is represented by the azimuth difference information in a horizontal direction when information indicating a difference between the azimuth information of the head-mounted display device and the azimuth information of the information terminal device is used as the azimuth difference information. When it is determined that the orientation difference to be performed is larger than a given threshold value, the selected icon may be confirmed.

  Thereby, on the selection menu screen in which icons are arranged in the vertical direction, it is possible to perform icon determination processing based on the movement of the head in the horizontal direction.

  Another aspect of the present invention relates to a head-mounted display device including the input system according to any one of claims 1 to 14.

  Another aspect of the present invention relates to an information terminal device including the input system according to any one of claims 1 to 14.

  Another aspect of the present invention is provided in a first sensor information from a first sensor provided in a head-mounted display device and an information terminal device that performs communication connection with the head-mounted display device. A sensor information acquisition unit that acquires second sensor information from a second sensor; a storage unit that stores reference value information that is information related to a reference value of the first sensor information and the second sensor information; Based on the reference value information, the first sensor information, and the second sensor information, first direction information that is direction information of the head-mounted display device and direction information of the information terminal device. The present invention relates to a program that causes a computer to function as a processing unit that acquires direction information of 2 and obtains user input information based on the acquired first direction information and second direction information.

The system configuration example of this embodiment. The figure explaining the screen displayed on the display part of a head mounted display apparatus, and the selection method of an item. The flowchart for demonstrating the process of 1st Embodiment. 4A and 4B are diagrams for explaining the surface of the information terminal device and the axis of the acceleration sensor, and FIG. 4C is a diagram for explaining the relationship between the value of the acceleration sensor and the surface facing the reference direction. . The figure displayed on the display part of a head mounted display, and the other figure explaining the selection method of an item. The flowchart for demonstrating the process of 2nd Embodiment. The other system configuration example of this embodiment.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. First, the method of this embodiment will be described. In recent years, head-mounted display devices (HMD, Head Mounted Display) used as display devices are limited in size and the like due to the characteristics of being mounted on the head, and as a result, there is only an operation unit with a simple configuration. Often I can't have it. For this reason, there is a problem that the interface for operation is not sufficient and complicated operation is difficult.

  Intuitive interfaces using user actions are widely used. For example, Patent Document 1 proposes an interface that uses the relative positions of two sensors attached to two locations (for example, a head and a hand) of a user's body. Japanese Patent Application Laid-Open No. 2004-228561 proposes a method of using the movement of the user's head for a joystick operation.

  However, the above-described method is not sufficient when a menu operation using the movement of the head is realized. The reason will be described using a case of selecting a menu item as shown in FIG. In this case, for example, the movement of the head in the horizontal direction (head orientation) is detected, and “Yes” is selected when facing the left, “No” when facing the front, “No” when facing the front, The implementation of selecting “Cancel” is natural and intuitive. However, a problem arises because the user is not always facing the front at the start of the operation. For example, assume that the user has already turned to the left at the start of the operation. In this case, if the selected item is “No”, the user cannot select “Yes” because he cannot turn to the left any more.

  In order to avoid the above problems, instead of looking at the amount of change in head movement from the start of the operation, the direction of the head relative to the body (for example, the direction in which the head is displaced several times relative to the front of the body) It is necessary to ask. This is done by obtaining the reference direction of the head and the reference direction of the body, obtaining the amount of deviation from the reference direction of the head and body, and calculating the difference between the obtained deviation amounts (using formula (1)). Will be described later). However, if the sensor is not fixed to the user's body and the mounting position or mounting direction changes, the head and body reference directions will change, and the head direction is obtained appropriately. I can't.

  By providing the head-side sensor in the head-mounted display device, it is considered that the change in the reference direction is unlikely to occur. This is because the mounting position of the head-mounted display device does not seem to change greatly in consideration of the configuration of the display unit and the support member. However, regarding the sensor on the body side, it is necessary to consider the change in the reference direction described above. The method of fixing the sensor to the user's body is not preferable because it forces the user to wear the sensor each time the interface device is used. In this embodiment, the part that the user carries and corresponds to the body This is because a sensor is mounted on an information terminal device (for example, a mobile phone) mounted on the device. In addition, as long as it is attached to the part corresponding to the body, it may be installed in a location that satisfies the requirement of moving following the movement of the body, and may be in a situation where it can be put in a bag held by the hand. .

  In other words, the information terminal device can be placed in a breast pocket, put in a pants pocket, put in a bag held by hand, etc. The state of the sensor on the body side also changes, and the reference value needs to be changed accordingly.

  Therefore, the applicant of the present invention provides the first sensor information from the first sensor provided in the head-mounted display device, the second sensor information from the second sensor provided in the information terminal device, This is an input system that obtains the direction of the head relative to the body (strictly, the direction of the head relative to the information terminal device) based on information (reference value information) regarding the reference value of the first sensor information and the second sensor information. Thus, an input system that can initialize the reference value information is proposed.

  Specific embodiments are described below. In the first embodiment, menu item selection processing corresponding to head movement in the horizontal direction using an azimuth sensor will be described. In the second embodiment, head movement in the vertical direction using an elevation angle sensor will be described. A corresponding menu item selection process will be described.

2. First Embodiment A menu item selection process corresponding to the movement of the head in the horizontal direction will be described. Specifically, after describing the system configuration example, an example of a screen displayed on the display unit of the head-mounted display device will be described, and finally, details of the processing will be described using a flowchart.

2.1 System Configuration Example FIG. 1 shows a configuration example of the head-mounted display device 100 and the information terminal device 200 including the input system according to the present embodiment. As shown in FIG. 1, the head-mounted display device 100 includes a display unit 110, an operation unit 120, a head direction detection unit 130, and a communication unit 140. Further, the information terminal device 200 includes a terminal direction detection unit 210, a storage unit 220, a processing unit 230, and a communication unit 240.

  The display unit 110 of the head-mounted display device 100 displays various display screens. Specifically, a selection menu screen or the like as described later with reference to FIG. 2 is displayed. The operation unit 120 is an interface for operating the head-mounted display device 100 and includes buttons and the like. Specifically, an initialization button or the like that explicitly indicates that the current determination timing is the initialization timing is included. The head direction detection unit 130 detects head direction information based on first sensor information from a first sensor provided in the head-mounted display device 100. In the present embodiment, the first sensor may be an orientation sensor, and the head direction information may be head direction information. The communication unit 140 performs data communication with the information terminal device 200. Communication may be performed by wire or wirelessly.

  The terminal direction detection unit 210 of the information terminal device 200 detects the direction information of the information terminal device 200 based on the second sensor information from the second sensor provided in the information terminal device 200. Similar to the first sensor, the second sensor may be an orientation sensor, and the direction information may be orientation information. The storage unit 220 serves as a work area for the processing unit 230 and the like, and its function can be realized by a memory such as a RAM, an HDD (hard disk drive), or the like. Further, the storage unit 220 stores reference value information that is information related to the reference values of the first sensor information and the second sensor information. The processing unit 230 performs various processes based on information stored in the storage unit 220 and information received by the communication unit 240. The function of the processing unit 230 can be realized by hardware such as various processors (CPU and the like), ASIC (gate array and the like), a program, and the like. Further, the processing unit 230 obtains the relative direction of the head with respect to the direction of the information terminal device 200 based on the direction information of the head, the direction information of the information terminal device 200, and the reference value information. Perform the selection process. The communication unit 240 performs data communication with the head-mounted display device 100. Communication may be performed by wire or wirelessly.

  Note that the system configuration of the present embodiment is not limited to that described above. For example, the operation unit 120 of the head-mounted display device 100 may be provided in the information terminal device 200. In addition, the first sensor needs to be provided in the head-mounted display device 100. The head direction detection unit 130 that obtains head direction information based on the sensor information from the first sensor is an information terminal. The apparatus 200 may be provided.

  Similarly, the storage unit 220 and the processing unit 230 of the information terminal device may be provided in the head-mounted display device 100. Moreover, although the 2nd sensor needs to be provided in the information terminal device 200, the terminal direction detection part 210 which calculates | requires the direction information of the information terminal device 200 based on the sensor information from a 2nd sensor is a head mounted type. The display device 100 may be provided.

2.2 Example of Selection Menu Screen FIG. 2 shows an example of the selection menu screen used in this embodiment. The selection menu screen is displayed on the display unit 110 of the head-mounted display device 100 in order to prompt the user to select an item. For example, as shown in FIG. 2, the currently selected item can be discriminated based on the thickness of a frame line surrounding the selected item.

  Then, as shown in FIG. 2, the user can change the item selected by the left and right head movements in the horizontal direction. Specifically, from the head direction information detected by the head direction detection unit 130 and the terminal direction information detected by the terminal direction detection unit 210 (corresponding to body direction information), the head direction relative to the terminal direction is determined. The amount of change in direction (for example, the size of the angle between the front direction of the terminal and the direction of the head) is obtained, and the selected item may be changed when the obtained amount of change exceeds a given threshold value. .

  Note that the selected item may be determined by another method such as changing the color or size of the currently selected item. Moreover, although the example of FIG. 2 shows the case where there are three selection items, any number may be used as long as there are two or more selection items.

2.3 Details of Processing Next, details of the processing will be described with reference to the flowchart of FIG. When this process is started, it is first determined whether or not the reference value information is stored in the storage unit 220 (S101). When the reference value information is not stored, the value of the current direction information is acquired from the head direction detection unit 130 and the terminal direction detection unit 210, and stored in the storage unit 220 as the reference value information (S102). Then, the processing unit 230 determines which item to select from the selection items on the selection menu screen from the reference value information, the current head direction information, and the current terminal direction information, and the menu screen on which the determined item is selected. Is displayed on the display unit 110 (S103).

  Here, processing for determining which menu item to select will be described. As an example, let us consider a case where the direction information is acquired as a sensor value of an azimuth sensor that takes a periodic boundary value of 0 to 359. In this case, the head direction value included in the reference value information is Dh, the terminal direction value included in the reference value information is Dm, and the current head direction value detected by the head direction detection unit 130 is Dh ′. If the value of the current terminal direction detected by the terminal direction detection unit 210 is Dm ′, the evaluation value Δ is given by the following equation (1).

    Δ = (Dh′−Dh) − (Dm′−Dm) (1)

  However, in consideration of the periodic boundary condition and the condition that the rotation of the head is within ± 180 degrees, Δ ′ obtained by the following equation (2) is used as the evaluation value.

    If Δ ≧ 180, Δ ′ = Δ−360

    If Δ <−180, Δ ′ = Δ + 360 (2)

  When Δ (or Δ ′) is 0, the menu item is selected at the center of the menu items. When Δ has a certain value positive or negative, a menu item located on either the left or right side may be selected. Which menu item is selected with which value depends on the number of menu items, the allocation of angle ranges, and the like.

  Thereafter, when a user operation is detected (S104), it is determined whether or not the head movement in the horizontal direction (left and right direction) exceeds a threshold (S105). The selection item is changed to the left or right according to the direction (S106). After the change, the process returns to S104. If NO in step S105, it is determined whether the initialization button has been pressed (S107). If the button is pressed, current head direction information and terminal direction information are acquired, and the acquired value is used as a reference. Stored in the storage unit 220 as value information. At the same time, since the current state is the initial state, a change is made to select the center item in the menu screen (S108).

  In the case of No in S107, it is determined whether the downward movement of the head exceeds the threshold (S109). Here, it is assumed that the selection item confirmation process is performed by an operation of moving the head downward (a motion of nodding). The operation of moving the head downward can be detected by using, for example, an elevation angle sensor as described in the second embodiment. If Yes in S109, the selected item is processed and the process ends (S110). If No in S109, the process returns to S104. However, in the present embodiment, the selection item determination process is not limited to that performed by the downward head movement. For example, a confirmation button may be provided in the operation unit 120 and the confirmation process may be performed by pressing the confirmation button.

  The above is the description of the basic processing and the initialization processing by the initialization button. However, the initialization timing is not limited to the timing when the initialization button is pressed. When the reference value information is stored in the storage unit 220 in S101, the reference value information is acquired from the storage unit 220 (S111). Then, current state information of the terminal is acquired from a sensor provided in the information terminal device 200 (this may be the same sensor as the second sensor described above or another sensor). S112). The terminal status information included in the reference value information stored in the storage unit 220 is compared with the current terminal status information, and the terminal status is determined between the timing at which the reference value information is stored and the current determination timing. Is changed (S113), and if it has changed, the process proceeds to S102, and the current head direction information and terminal direction information are stored in the storage unit 220 as reference value information. This is, for example, the situation where the information terminal device 200 that has been put in the breast pocket until now has been moved to the pocket of the trousers, or the manner of holding the bag containing the information terminal device 200 has been changed, and so on. This is a case where the reference value information is no longer appropriate. Therefore, since the reference value needs to be updated as long as the state of the terminal has changed greatly, the reference value information is updated to a value obtained from the current sensor information in S102, and initialization processing is performed.

  In the case of No in S113, it means that the state of the information terminal device 200 has not changed significantly, so the reference value information stored in the storage unit 220 is used as it is (S114), and the process proceeds to S103.

  A specific example of the state change determination method in S113 will be described with reference to FIG. Consider a case where the information terminal device 200 is a six-sided rectangular parallelepiped as shown in FIG. Then, the information terminal apparatus 200 is provided with a three-axis acceleration sensor that detects acceleration in the directions of the x, y, and z axes as shown in FIGS. 4 (A) and 4 (B). After that, by finding out which of the x, y, and z axes has the largest acceleration acting in which direction, the plane facing the vertically upward direction is specified. Here, as shown in FIG. 4C, by determining which axis and which direction has the greatest acceleration, it is possible to uniquely identify the surface facing the vertically upward direction. And the specified surface should just be made into the state information mentioned above. That is, the specified surface at the timing at which the reference value information is stored and the specified surface at the current determination timing become state information, and if the specified surface is the same, the state does not change, and the specified surface is If they are not identical, it is determined that the state has changed.

  Thus, it is possible to automatically perform the initialization process even if the user does not explicitly issue an initialization instruction, and it is possible to realize an input system that can cope with a change in the state of the terminal.

  In the above embodiment, as shown in FIG. 1, the input system includes a sensor information acquisition unit (corresponding to the head direction detection unit 130 and the terminal direction detection unit 210), and a storage unit 220 that stores reference value information. And a processing unit 230. The sensor information acquisition unit receives the first sensor information from the first sensor provided in the head-mounted display device 100 and the second sensor information from the second sensor provided in the information terminal device 200. get. The storage unit 220 stores information related to the reference values of the first sensor information and the second sensor information as reference value information. Based on the reference value information, the first sensor information, and the second sensor information, the processing unit 230 includes first direction information that is direction information of the head-mounted display device 100 and direction information of the information terminal device 200. The second direction information is obtained, and user input information is obtained based on the obtained direction information.

  As a result, the head-mounted display device 100 and the information terminal device 200 are each provided with a sensor, acquire sensor information of each sensor, and use the reference value of each sensor information together. It is possible to obtain first direction information that is 100 direction information and second direction information that is direction information of the information terminal device 200. Then, user input information is obtained based on the first direction information and the second direction information. Therefore, for example, by performing the calculation shown in the above-described equation (1), the deviation amount from the reference value in the head direction (Dh′−Dh) and the deviation amount from the reference value in the terminal direction (Dm′−Dm). ) To obtain the relative direction of the head relative to the direction of the terminal. For example, if the value when the body direction (corresponding to the terminal direction) and the head direction are both facing the front is set as the reference value, the amount of deviation from that can be obtained. The relative movement of the head can be detected appropriately. In other words, for example, even when the menu item selection is not started, the initial selection item can be appropriately selected (specifically, when the menu item is on the left from the beginning, the menu item arranged on the left is selected). Can be selected first).

  The storage unit 220 stores the first sensor information and the second sensor information acquired by the sensor information acquisition unit as reference value information when the processing unit 230 determines that it is the initialization timing. May be.

  As a result, initialization processing can be performed. The value stored in the storage unit 220 as the reference value information needs to be a reference value (for example, a value when the head and the body are in the same direction). However, in the present embodiment, it is assumed that a sensor on the body side is provided in an existing device (for example, a mobile phone) so that the user is not forced to attach the sensor individually as described above. Therefore, it is assumed that the relationship between the sensor provided in the information terminal device 200 and the direction of the body changes when the mounting position or mounting direction of the information terminal device 200 changes. Therefore, in such a case, the state of the information terminal device 200 is performed by performing initialization processing with an appropriate timing after the change (for example, a timing when the head and body face the same direction) as the initialization timing. It is possible to respond to changes in Specifically, as will be described later, the timing at which a change in state is detected from sensor information of a sensor provided in the information terminal device 200 or the initialization button is pressed may be considered as the initialization timing.

  In addition, the processing unit 230 includes first state information that is information representing the state of the information terminal device 200 at the first timing when the reference value information is stored in the storage unit 220, and the second determination timing. The second timing information is compared, and the second timing is initialized when it is determined that the status of the information terminal device 200 has changed. You may determine that it is timing.

  As a result, the initialization process can be automatically performed without an explicit instruction from the user. The first state information and the second state information are acquired by some means (for example, sensor information of the sensor as will be described later), and it is determined whether or not the state has changed by comparing the acquired two state information. Do. This determination may be made based on whether or not the first state information and the second state information are the same, or is represented by a value represented by the first state information and the second state information. It may be performed by obtaining a difference value from the value and comparing the obtained difference value with a given threshold value.

  In addition, the processing unit 230 uses the acceleration sensor provided in the information terminal device 200 to obtain first inclination information that is information about the inclination of the information terminal device 200 with respect to the gravity direction at the first timing as the first state information. May be obtained. At the same time, second inclination information that is information about the inclination of the information terminal device 200 with respect to the direction of gravity at the second timing is acquired as second state information. Then, based on the first tilt information and the second tilt information, it is determined whether the state has changed, and when it is determined that the state has changed, the determination that the second timing is the initialization timing. May be performed.

  Thereby, it becomes possible to determine whether or not the state has changed using the sensor information of the acceleration sensor. As shown in FIGS. 4A and 4B, when considering a case where the information terminal device 200 is provided with a triaxial acceleration sensor, the inclination information is the acceleration applied to each of the x, y, and z axes. Can be obtained as the value of. Therefore, it is possible to easily determine a change in state by comparing these values. Further, if an acceleration sensor is used to determine the state and an azimuth sensor is used to acquire direction information, these sensors may be different. In that case, the reference value information includes not only the reference value of the sensor information of the second sensor (for example, the azimuth sensor) but also the information of the reference value of the sensor for determining the state (for example, the acceleration sensor). become.

  In addition, the processing unit 230 includes the i-th (i is an integer of 1 ≦ i ≦ N) of the first to Nth (N is an integer of 2 or more) surfaces of the information terminal device 200 in the reference direction. The first tilt information is determined to be oriented, and the j-th (j is an integer of 1 ≦ j ≦ N, i ≠ j) of the first to N-th surfaces faces the reference direction. If it is determined by the second inclination information, it may be determined that the state of the information terminal device has changed, and the second timing may be determined as the initialization timing.

  Thereby, it becomes possible to determine the change of the state using the acceleration sensor more easily. For example, as shown in FIG. 4C, a surface corresponding to the obtained axis and direction is obtained by determining which direction of the x, y, and z axes has the greatest acceleration. Can be determined as a surface facing the reference direction. Then, it is determined whether the state has changed based on whether the surface specified by the first inclination information is the same as the surface specified by the second inclination information. Considering that accelerometer values are acquired by the number of axes (three if it is triaxial acceleration), the comparison of the surface facing the reference direction compared to the comparison of the sensor value itself Is easy.

  Further, the processing unit 230 may determine that the timing when the initialization operation is performed by the user is the initialization timing.

  Thus, initialization can be performed in response to an explicit operation instruction from the user. For example, the initialization operation may be performed by an initialization button or the like provided on the operation unit 120.

  The head-mounted display device 100 includes a display unit 110 that displays a selection menu screen on which a plurality of icons are displayed side by side along the first direction. If the change amount of the first direction information with respect to the second direction information in the direction along the first direction is determined to be larger than a given threshold, the processing unit 230 displays the selection menu screen. You may perform the process which changes the icon selected into the other icon of these icons.

  This makes it possible to display a menu screen and select items as shown in FIG. 2, for example. When the icons are arranged in the horizontal direction as shown in FIG. 2, the icon (item) is selected based on the movement of the head in the horizontal direction (left-right direction). However, the first direction is not limited to the horizontal direction, and may be a vertical direction or an oblique direction.

  In addition, when the processing unit 230 determines that the amount of change in the first direction information with respect to the second direction information in a second direction different from the first direction is greater than a given threshold, You may perform the confirmation process of the selected icon.

  Thus, by moving the head in a direction different from the direction in which the icons are arranged, it becomes possible to perform the confirmation process for the item selected at that time. For example, as described above, this corresponds to performing the confirmation process by the downward movement of the head (nodding motion). However, the direction of the movement of the head that performs the determination process does not have to be orthogonal to the first direction, and may be a direction that is determined not to be the first direction. For example, when the angle formed with the first direction is larger than a given threshold value, the movement in the direction may be determined as the movement in the second direction.

  Further, the first sensor may be a first azimuth sensor provided in the head-mounted display device 100, and the second sensor may be a second azimuth sensor provided in the information terminal device 200. The sensor information acquisition unit acquires first azimuth information that is azimuth information of the head-mounted display device 100 as first sensor information, and uses azimuth information of the information terminal device 200 as second sensor information. A certain second direction information is acquired. The processing unit 230 may obtain input information based on orientation difference information that is information representing a difference between the first orientation information and the second orientation information.

  Thereby, as described above, it is possible to use an orientation sensor as the sensor. In this case, since it is possible to detect the direction of the head and body (strictly speaking, the information terminal device 200) in the horizontal direction, the relative movement of the head in the horizontal direction can be detected. Therefore, as shown in FIG. 2, an operation by the head on the selection menu screen in which icons are displayed side by side in the horizontal direction (left-right direction) becomes possible.

  The head-mounted display device 100 may also include a display unit 110 that displays a selection menu screen on which a plurality of icons are displayed side by side along the horizontal direction. Then, when it is determined that the azimuth difference in the direction along the horizontal direction is larger than a given threshold, the processing unit 230 changes the icon selected on the selection menu screen to the other of the plurality of icons. Process to change to icon. Further, when the information representing the difference between the elevation angle information of the head-mounted display device 100 and the elevation angle information of the information terminal device 200 is the elevation angle difference information, the elevation angle difference in the vertical direction (value represented by the elevation angle difference information) is If it is determined that the value is larger than a given threshold value, the selected icon may be confirmed.

  This makes it possible to display the menu screen and select items as shown in FIG. In addition, as described above in the present embodiment, it is possible to perform a determination operation by moving the head in the vertical direction. In the above-described flowchart, only the downward movement is cited, but the present invention is not limited to this. Here, it is assumed that the determination operation can be performed even with the upward movement of the head.

  Further, the present embodiment described above relates to the head-mounted display device 100 including the input system described above.

  Thereby, it is possible to cause the head-mounted display device 100 to perform the processing of the input system according to the present embodiment. Specifically, in FIG. 1, the terminal direction detection unit 210 provided on the information terminal device 200 side is provided on the head-mounted display device 100 side, and the sensor information acquisition unit is combined with the head direction detection unit 130. It is equivalent to. The storage unit 220 and the processing unit 230 provided on the information terminal device 200 side are also provided on the head-mounted display device 100 side.

  Moreover, the above embodiment relates to the information terminal device 200 including the input system described above.

  Thereby, it is possible to cause the information terminal device 200 to perform the processing of the input system according to the present embodiment. Specifically, in FIG. 1, the head direction detection unit 130 provided on the head-mounted display device 100 side is provided on the information terminal device 200 side, and the sensor information acquisition unit is combined with the terminal direction detection unit 210. It is equivalent to. Further, the storage unit 220 and the processing unit 230 are provided on the information terminal device 200 side as in FIG.

  Further, the present embodiment described above causes a computer to function as a sensor information acquisition unit (corresponding to the head direction detection unit 130 and the terminal direction detection unit 210), a storage unit 220 that stores reference value information, and a processing unit 230. Related to the program. The sensor information acquisition unit receives the first sensor information from the first sensor provided in the head-mounted display device 100 and the second sensor information from the second sensor provided in the information terminal device 200. get. The storage unit 220 stores information related to the reference values of the first sensor information and the second sensor information as reference value information. Based on the reference value information, the first sensor information, and the second sensor information, the processing unit 230 includes first direction information that is direction information of the head-mounted display device 100 and direction information of the information terminal device 200. The second direction information is obtained, and user input information is obtained based on the obtained direction information.

  As a result, it is possible to realize a program that executes the processing in the input system described above in software. The program is recorded in the information storage medium 250. Here, as the information storage medium 250, various recording media that can be read by an information processing apparatus such as an optical disk such as a DVD or a CD, a magneto-optical disk, a hard disk (HDD), a memory such as a nonvolatile memory or a RAM are assumed. it can. For example, as illustrated in FIG. 7, a case where the program is stored in various recording media that can be read by an information processing apparatus such as the information terminal apparatus 200 and executed in the processing unit 230 can be considered.

3. Second Embodiment A menu item selection process corresponding to the movement of the head in the vertical direction will be described. Specifically, after describing the system configuration example, an example of a screen displayed on the display unit of the head-mounted display device will be described, and finally, details of the processing will be described using a flowchart.

3.1 System Configuration Example Configuration examples of the head-mounted display device 100 and the information terminal device 200 including the input system according to the present embodiment are as illustrated in FIG. 1 and are the same as those in the first embodiment. Detailed description is omitted.

3.2 Example of Selection Menu Screen FIG. 5 shows an example of the selection menu screen used in this embodiment. The first implementation described above with reference to FIG. 2 except that the selection items are arranged in the vertical direction (up and down direction) as shown in FIG. 5 and the items are selected by the movement of the head in the vertical direction of the user. It is the same as the form.

3.3 Details of Processing Details of the processing according to this embodiment are shown in the flowchart of FIG. S201 to S210 correspond to S101 to S110 of the first embodiment, the item is selected by the vertical movement of the head (S205, S206), and the selection item confirmation process is the right movement of the head (S209) is only different.

  However, since the direction sensor is not used for the process of determining which menu item is selected, it is different from the first embodiment. As an example, let us consider a case where the direction information is acquired as a sensor value of an elevation angle sensor having a value of 0 to 255. The value is 255 when the top surface of the head or the terminal is oriented vertically upward, and the value is 0 when it is oriented vertically downward. In this case, the head direction value included in the reference value information is Ah, the terminal direction value included in the reference value information is Am, and the current head direction value detected by the head direction detection unit 130 is Ah ′. If the current terminal direction value detected by the terminal direction detection unit 210 is Am ′, the evaluation value Δ is given by the following equation (3).

    Δ = (Ah′−Ah) − (Am′−Am) (3)

  Unlike the azimuth sensor described in the first embodiment, the elevation angle sensor does not take a periodic boundary value, and thus processing such as the above equation (2) is not necessary. Since the time when Δ is 0 is the front, the menu item located at the center is selected. When Δ has a certain value, either positive or negative, a menu item positioned either up or down may be selected. Which menu item is selected with which value depends on the number of menu items, the allocation of angle ranges, and the like.

  Also, S211 to S214 correspond to S111 to S114 of the first embodiment, and a change in the state of the terminal as described above with reference to FIGS. 4A to 4C is determined and an initialization process is appropriately performed. The point of performing is the same.

  In the above embodiment, the first sensor is the first elevation sensor provided in the head-mounted display device 100, and the second sensor is the second elevation sensor provided in the information terminal device 200. There may be. The sensor information acquisition unit acquires first elevation angle information that is elevation angle information of the head-mounted display device 100 as first sensor information, and the elevation angle information of the information terminal device 200 as second sensor information. A certain second elevation angle information is acquired. The processing unit 230 may obtain input information based on elevation angle difference information that is information representing a difference between the first elevation angle information and the second elevation angle information.

  Thereby, as described above, an elevation angle sensor can be used as the sensor. In this case, since it becomes possible to detect the direction of the head and body (strictly speaking, the information terminal device 200) in the vertical direction, it is possible to detect the relative movement of the head in the vertical direction. Therefore, as shown in FIG. 5, it is possible to perform an operation with the head on the selection menu screen in which icons are displayed side by side in the vertical direction (vertical direction).

  The head-mounted display device 100 may include a display unit 110 that displays a selection menu screen on which a plurality of icons are displayed side by side along the vertical direction. Then, when the processing unit 230 determines that the elevation angle difference in the direction along the vertical direction is larger than a given threshold, the processing unit 230 selects an icon selected on the selection menu screen as the other of the plurality of icons. Process to change to icon. Further, when information indicating the difference between the azimuth information of the head-mounted display device 100 and the azimuth information of the information terminal device 200 is used as the azimuth difference information, the azimuth difference in the horizontal direction (value represented by the azimuth difference information) is If it is determined that the value is larger than a given threshold value, the selected icon may be confirmed.

  This makes it possible to display the menu screen and select items as shown in FIG. In addition, as described above in the present embodiment, it is possible to perform a confirmation operation by moving the head in the horizontal direction. In the above-described flowchart, only the movement in the right direction is cited, but the present invention is not limited to this. Here, it is assumed that the determination operation can be performed even with the movement of the head in the left direction.

  The two embodiments 1 and 2 to which the present invention is applied and the modifications thereof have been described above, but the present invention is not limited to the embodiments 1 and 2 and the modifications as they are, The constituent elements can be modified and embodied without departing from the spirit of the invention. Various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above-described first and second embodiments and modifications. For example, you may delete a some component from all the components described in each Embodiment 1-2 or the modification. Furthermore, you may combine suitably the component demonstrated in different embodiment and modification. Thus, various modifications and applications are possible without departing from the spirit of the invention.

100 head-mounted display device, 110 display unit, 120 operation unit,
130 head direction detection unit, 140 communication unit, 200 information terminal device,
210 terminal direction detection unit, 220 storage unit, 230 processing unit, 240 communication unit,
250 Information storage media

Claims (17)

First sensor information from a first sensor provided in the head-mounted display device, and second from a second sensor provided in an information terminal device that performs communication connection with the head-mounted display device. A sensor information acquisition unit for acquiring sensor information of
A storage unit that stores reference value information that is information related to a reference value of the first sensor information and the second sensor information;
Based on the reference value information, the first sensor information, and the second sensor information, first direction information that is direction information of the head-mounted display device and direction information of the information terminal device. A processing unit that acquires direction information of 2 and obtains user input information based on the acquired first direction information and second direction information;
An input system comprising: In claim 1,
The storage unit
Storing the first sensor information and the second sensor information acquired by the sensor information acquisition unit as the reference value information when it is determined by the processing unit that it is an initialization timing. Input system. In claim 2,
The processor is
The first state information in the first state that is the state of the information terminal device at the first timing when the reference value information is stored in the storage unit, and the second timing that is the current determination timing The second state information in the second state, which is the state of the information terminal device, is compared, and when it is determined by the comparison processing that the state of the information terminal device has changed, It is determined that the timing is the initialization timing. In claim 3,
The processor is
With the acceleration sensor provided in the information terminal device, as the first state information, the first inclination information that is information related to the inclination of the information terminal device with respect to the direction of gravity at the first timing, As the second state information, second inclination information that is information relating to the inclination of the information terminal device with respect to the direction of gravity at the second timing is acquired, and the first inclination information and the second inclination are acquired. An input system that performs an information comparison process, and determines that the second timing is the initialization timing when it is determined that the state of the information terminal device has changed by the comparison process. In claim 4,
The processor is
If the i-th (i is an integer of 1 ≦ i ≦ N) of the first to Nth (N is an integer of 2 or more) surfaces constituting the information terminal device is oriented in the reference direction, the first 1 is determined based on the inclination information of 1 and the jth surface (j is an integer of 1 ≦ j ≦ N, i ≠ j) among the first to Nth surfaces is oriented in the reference direction, An input system characterized in that, when it is determined based on the second inclination information, it is determined that the state of the information terminal device has changed, and the second timing is determined to be the initialization timing. In claim 2,
The processor is
An input system, wherein a timing at which an initialization operation is performed by the user is determined as the initialization timing. In any one of Claims 1 thru | or 6.
The head-mounted display device is
A display unit that displays a selection menu screen in which a plurality of icons are displayed side by side along the first direction;
The processor is
When it is determined that the amount of change in the first direction information with respect to the second direction information in the direction along the first direction is larger than a given threshold, the selection is made on the selection menu screen. An input system that performs a process of changing an icon to another icon of the plurality of icons. In claim 7,
The processor is
The selected icon when it is determined that the amount of change in the first direction information with respect to the second direction information in a second direction different from the first direction is greater than a given threshold. An input system characterized by performing a confirmation process. In any one of Claims 1 thru | or 6.
The first sensor is a first azimuth sensor provided in the head-mounted display device, and the second sensor is a second azimuth sensor provided in the information terminal device,
The sensor information acquisition unit
While acquiring the 1st azimuth | direction information which is the azimuth | direction information of the said head mounted display apparatus as 1st sensor information, the 2nd azimuth | direction information which is the azimuth | direction information of the said information terminal device as said 2nd sensor information Acquired,
The processor is
An input system for obtaining the input information based on azimuth difference information which is information representing a difference between the first azimuth information and the second azimuth information. In claim 9,
The head-mounted display device is
Including a display unit that displays a selection menu screen in which a plurality of icons are displayed side by side along the horizontal direction ;
The processor is
When it is determined that an azimuth difference represented by the azimuth difference information in a direction along the horizontal direction is larger than a given threshold, an icon selected on the selection menu screen is selected from the plurality of icons. An input system characterized by processing to change to another icon. In claim 10,
The processor is
When the information representing the difference between the elevation angle information of the head-mounted display device and the elevation angle information of the information terminal device is the elevation angle difference information, there is an elevation angle difference represented by the elevation angle difference information in the vertical direction. An input system that performs a process of confirming a selected icon when it is determined that the value is larger than a given threshold value. In any one of Claims 1 thru | or 6.
The first sensor is a first elevation angle sensor provided in the head-mounted display device, and the second sensor is a second elevation angle sensor provided in the information terminal device,
The sensor information acquisition unit
As the first sensor information, first elevation angle information that is elevation angle information of the head-mounted display device is acquired, and as the second sensor information, second elevation angle information that is elevation angle information of the information terminal device is obtained. Acquired,
The processor is
An input system that obtains the input information based on elevation angle difference information that is information representing a difference between the first elevation angle information and the second elevation angle information. In claim 12,
The head-mounted display device is
Including a display unit for displaying a selection menu screen in which a plurality of icons are displayed side by side along the vertical direction ;
The processor is
When it is determined that an elevation angle difference represented by the elevation angle difference information in a direction along the vertical direction is larger than a given threshold, an icon selected on the selection menu screen is selected from the plurality of icons. An input system characterized by processing to change to another icon. In claim 13,
The processor is
When the information representing the difference between the heading display device orientation information and the information terminal device orientation information is the orientation difference information, the orientation difference represented by the orientation difference information in the horizontal direction is given. An input system that performs a process of confirming a selected icon when it is determined that the value is larger than a given threshold value.   A head-mounted display device comprising the input system according to claim 1.   An information terminal device comprising the input system according to claim 1. First sensor information from a first sensor provided in the head-mounted display device, and second from a second sensor provided in an information terminal device that performs communication connection with the head-mounted display device. A sensor information acquisition unit for acquiring sensor information of
A storage unit that stores reference value information that is information related to a reference value of the first sensor information and the second sensor information;
Based on the reference value information, the first sensor information, and the second sensor information, first direction information that is direction information of the head-mounted display device and direction information of the information terminal device. As a processing unit that acquires direction information of 2 and obtains user input information based on the acquired first direction information and second direction information,
A program characterized by causing a computer to function.

Images