JP2017157120A - Display device, and control method for the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To alleviate restriction in operability due to a size of an operation screen, and to allow an operator to easily control an end position of operation in a display device in which the operation can be performed on an operation screen.SOLUTION: An HMD 100 comprises an image display section 20 transmitting an outside scenery while causing a user to visually recognize an image. The HMD 100 comprises: an operation detection section detecting operation on a track pad 14; and a control section displaying a display object of an operational target on the image display section 20. The control section controls a display position of the display object according to the operation with respect to the track pad 14, obtains a direction of movement of the display object according to a change of an operation position in the track pad 14, moves the display position of the display object, and stops movement of the display object when the operation with respect to the track pad 14 corresponds to preliminarily set termination conditions.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a display device and a display device control method.

  Conventionally, as an input device for operating a computer or the like, an input device having an operation surface for detecting a contact operation is known (for example, see Patent Document 1). Patent Document 1 discloses a configuration in which a touch panel terminal is used to operate a computer on the premise that the computer is operated using a mouse. In an input device that detects an operation on the operation surface, operability is limited by the size of the operation surface. In order to eliminate this restriction, in the configuration described in Patent Document 1, the movement distance is calculated from the speed at which the touch panel detects the contact, and the mouse pointer is moved on the computer according to the movement distance.

JP 2013-143076 A

However, as described in Patent Document 1, when an operation index such as a mouse pointer is moved from an operation speed, it is difficult for an operator (user) to control the position where the index stops, that is, the operation end position. Met.
The present invention has been made in view of the above circumstances, and in a display device capable of performing an operation on the operation surface, the operability restriction due to the size of the operation surface is relaxed, and the operation end position is operated. It is intended to enable a person to easily control.

To achieve the above object, the present invention provides a display unit, an operation detection unit for detecting an operation on the operation surface, and a display object to be operated is displayed on the display unit, and the operation object is displayed in response to the operation on the operation surface. A control unit that controls the display position of the display object, and the control unit moves the display position of the display object in response to a change in the operation position on the operation surface detected by the operation detection unit, When the operation detection unit detects an operation corresponding to a preset end condition, the movement of the display position of the display object is stopped.
ADVANTAGE OF THE INVENTION According to this invention, when moving the display position of a display object according to operation with respect to an operation surface, the moving amount | distance of a display position can be controlled with a high freedom degree. For example, if the operation position on the operation surface stops and is set so as not to satisfy the termination condition, the display object can be moved larger than the movement amount of the operation position. In this case, the operability restriction due to the size of the operation surface can be relaxed. Furthermore, the position or timing at which the movement of the display position of the display object is stopped can be easily controlled.

Further, the present invention is characterized in that, in the display device, the control unit sets the movement direction of the display position of the display object to a direction corresponding to the movement direction of the operation position on the operation surface.
According to the present invention, the operator can move the display object to be operated in the intended direction, and the operability can be improved.

In the display device according to the aspect of the invention, the control unit performs a process of obtaining a moving direction of the display position of the display object after the operation detection unit detects an operation on the operation surface, and then performs the display. The display position of the object is moved.
According to the present invention, the display object to be operated can be moved in an appropriate direction.

In the display device according to the aspect of the invention, the control unit may perform the process of obtaining a moving direction of the display position of the display object after the operation detection unit detects an operation on the operation surface. The display object is displayed at a display position corresponding to the operation position.
According to the present invention, it is possible to appropriately control the display position of the display object even during the process of obtaining the moving direction of the display object. For this reason, the operator can operate without a sense of incongruity. In addition, the display object can be moved in response to an operation with a small amount of movement of the operation position for a short time.

In the display device according to the present invention, the control unit sets the movement speed of the display position of the display object to a speed corresponding to the movement amount or movement speed of the operation position on the operation surface. And
According to the present invention, since the operator can easily control the moving speed of the display object, the operability can be further improved.

In the display device according to the aspect of the invention, when the operation position on the operation surface detected by the operation detection unit is changed while the display position of the display object is moved, the control unit It is characterized in that at least one of the moving direction and moving speed of the display position is changed.
According to the present invention, after the movement of the display position of the display object is started, at least one of the moving direction and the moving speed of the display position is changed according to the operation. For this reason, even after the movement of the display position of the display object is started by the operation of the operator, the operator can adjust the moving direction and moving speed of the display object.

In the display device according to the aspect of the invention, the control unit may detect an operation position when the operation detection unit detects contact of the operation body with the operation surface, and the operation detection unit may detect the operation surface from the operation surface. It is determined that the termination condition is satisfied when it is detected that the operating body is separated and the contact is released.
According to the present invention, since the operator can easily instruct the timing to start and stop the movement of the display position of the display object, the operability can be further improved.

In the display device according to the aspect of the invention, the control unit may detect an operation position when the operation detection unit detects contact of the operation body with the operation surface, and the operation detection unit may detect the operation surface from the operation surface. It is characterized in that it is determined that the end condition is satisfied when a predetermined time has elapsed since it was detected that the operating body has left and the contact has been released.
According to the present invention, it is possible to continue the movement of the display position of the display object even after the operation tool is separated from the operation surface. Therefore, the display position of the display object can be moved larger than the amount by which the operation position moves on the operation surface, and the restriction on the size of the operation surface can be avoided.

In the display device according to the aspect of the invention, the control unit may detect the operation within a predetermined time after the operation detection unit detects that the operation body has moved away from the operation surface and contact has been released. When the unit detects contact with the operation surface, the moving direction and the moving speed of the display position of the display object are changed corresponding to the operation position of the new contact detected by the operation detection unit. It is characterized by this.
According to the present invention, the movement direction or movement speed can be controlled for the movement of the display position of the display object even after the operation body is separated from the operation surface. For this reason, the mode of movement of the display position of the display object can be controlled with a high degree of freedom.

Further, the present invention provides the display device including an operation device having the operation surface, further including an attitude detection unit that detects an attitude of the operation device or a change in the attitude, and the control unit displays a display position of the display object. While moving, at least one of the moving direction and moving speed of the display position of the display object is changed in response to a change in the posture or posture of the operation device detected by the posture detection unit. And
According to the present invention, the moving direction and moving speed of the display position of the display object can be controlled by changing the posture of the operation device while the display position of the display object moves. For this reason, the mode of movement of the display position of the display object can be controlled with a high degree of freedom by a simple operation.

Further, the present invention is characterized in that, in the display device, an area or size that the operation detection unit detects an operation on the operation surface is smaller than an area or size of a display area on which the display unit displays an image. And
According to the present invention, it is possible to arbitrarily move a display object using an operation surface smaller than the display area, and to improve operability. Further, there is an advantage that the degree of freedom of the device configuration is increased.

Further, the present invention is characterized in that, in the display device, the shape of the operation surface is different from a display area where the display unit displays an image.
According to the present invention, it is possible to arbitrarily move a display object using an operation surface having a shape different from that of the display area, and to improve operability. Further, there is an advantage that the degree of freedom of the device configuration is increased.

In order to achieve the above object, the display device of the present invention includes a display unit, an operation detection unit that detects an operation on the operation surface, and an input to the virtual input plane according to the operation detected by the operation detection unit. A controller that generates data, and the controller detects input data on the virtual input plane in response to the operation when the operation detector detects an operation in which the operation position moves on the operation surface. Generation of the input data is started, and generation of the input data is stopped when the operation detection unit detects an operation corresponding to a preset end condition.
According to the present invention, it is possible to generate input data for a virtual input plane beyond the limitation of the size of the operation surface. For example, if the display object is moved based on the input data, the amount of movement of the display object can be controlled with a high degree of freedom. Therefore, it is possible to relax the operability restriction caused by the size of the operation surface.

In the display device according to the present invention, the input data generated by the control unit includes an input position in the virtual input plane, a moving direction of the input position in the virtual input plane, and an input in the virtual input plane. It includes at least one of the moving speed of the position.
According to the present invention, an input position on the virtual input plane, a moving direction of the input position, or a moving speed of the input position can be input by an operation on the operation surface.

In the display device according to the aspect of the invention, the control unit may at least one of a moving direction of the input position in the virtual input plane and a moving speed of the input position in the virtual input plane included in the input data. Is set according to the movement amount or movement speed of the operation position detected by the operation detection unit.
According to the present invention, the input position on the virtual input plane, the moving direction of the input position, or the moving speed of the input position can be controlled by operating the operation surface.

Further, in the display device according to the present invention, the operation detection unit detects a contact position as an operation position according to an operation in which an operating body contacts the operation surface, and the control unit includes the operation detection unit, When it is detected that the contact with the operation surface is released, it is determined that the termination condition is satisfied, and generation of the input data is stopped.
According to the present invention, the operator can easily instruct the timing for starting and stopping the generation of the input data, so that the operability can be further improved.

In the display device according to the aspect of the invention, the control unit may determine that the end condition is satisfied when a predetermined time has elapsed after the operation detection unit detects release of contact with the operation surface. The generation of the input data is stopped.
According to the present invention, it is possible to continue generating the input data even after the operating body is separated from the operation surface. For example, when the display position of the display object is moved corresponding to the input data, the movement of the display position can be continued even after the contact with the operation surface is released. For this reason, input data can be input without being limited by the amount by which the operation position moves on the operation surface, and the restriction on the size of the operation surface can be avoided.

According to the present invention, in the display device, the control unit includes a three-dimensional input including a direction perpendicular to the virtual input plane and the virtual input plane according to an operation detected by the operation detection unit. Data can be generated, and when the operation detection unit detects operations at a plurality of operation positions on the operation surface, the input data including a component in a direction perpendicular to the virtual input plane is generated. And
According to the present invention, three-dimensional input can be performed by an operation on the operation surface.

According to the present invention, the display device includes an operation device having the operation surface, and includes an attitude detection unit that detects an attitude of the operation device or a change in the attitude, and the control unit is detected by the attitude detection unit. The correspondence between the operation surface and the virtual input plane is changed in response to the attitude of the operation device or a change in attitude.
According to the present invention, the direction of input data can be changed by an operation of tilting an operation device having an operation surface.

According to the present invention, in the display device, the virtual input plane is associated with a display area in the display unit, and the control unit displays a display object to be operated on the display unit, and the input data Correspondingly, the display position of the display object is controlled.
According to the present invention, the display position of the display object can be moved according to the operation on the operation surface, and the amount of movement of the display position can be controlled with a high degree of freedom.

In order to achieve the above object, the display device control method of the present invention controls a display device having a display unit to detect an operation on the operation surface, and causes the display unit to display a display object to be operated. The display position of the display object is controlled in response to an operation on the operation surface, the display position of the display object is moved in response to a change in the operation position on the operation surface, and is set in advance on the operation surface. When the operation corresponding to the end condition is detected, the movement of the display position of the display object is stopped.
ADVANTAGE OF THE INVENTION According to this invention, when moving the display position of a display object according to operation with respect to an operation surface, the moving amount | distance of a display position can be controlled with a high freedom degree. For example, if the operation position on the operation surface stops and is set so as not to satisfy the termination condition, the display object can be moved larger than the movement amount of the operation position. In this case, the operability restriction due to the size of the operation surface can be relaxed. Furthermore, the position or timing at which the movement of the display position of the display object is stopped can be easily controlled.

In order to achieve the above object, the present invention is a program executable by a computer that controls a display device having a display unit, the computer including an operation detection unit that detects an operation on an operation surface, and the display. A display object to be operated is displayed on the screen, a display position of the display object is controlled in response to an operation on the operation surface, and a moving direction of the display object is obtained in response to a change in the operation position on the operation surface. The display position of the display object is moved in response to a change in the operation position on the operation surface detected by the operation detection unit, and the operation detection unit detects an operation corresponding to a preset end condition And configured as a program that functions as a control unit that stops the movement of the display position of the display object. Kill.
When the computer executes this program, the amount of movement of the display position can be controlled with a high degree of freedom when the display object at the operation target position displayed by the computer is moved according to the operation on the operation surface. For example, if the operation position on the operation surface stops and is set so as not to satisfy the termination condition, the display object can be moved larger than the movement amount of the operation position. In this case, the operability restriction due to the size of the operation surface can be relaxed. Furthermore, the position or timing at which the movement of the display position of the display object is stopped can be easily controlled.

The present invention can also be realized in the form of a storage medium storing the above-described program, a server device that distributes the program, a transmission medium that transmits the program, a data signal in which the program is embodied in a carrier wave, and the like. . The storage medium may be a magnetic or optical storage medium or a semiconductor memory device, and other types of storage media may be used. This storage medium may be any of a portable storage medium such as a memory card, a storage medium fixedly provided in the apparatus, and a storage medium provided in an apparatus connected to the apparatus via a communication line. The specific mounting form is also arbitrary.
The program can be implemented as a single application program that operates on the operating system in a device that operates with the operating system. Further, the present invention is not limited to a single application program, and may be implemented as a plurality of functions among an operating system, a device driver, and an application program. For example, a device driver program that controls an operation device having an operation surface and / or a program module that receives an operation of the operation device in an operating system may cooperate to realize the program. Moreover, the structure which implement | achieves the said program of this invention with a some application program may be sufficient, and the form of a specific program is arbitrary.

Explanatory drawing which shows the external appearance structure of HMD of 1st Embodiment. The functional block diagram of each part which comprises HMD of 1st Embodiment. It is explanatory drawing which shows operation | movement of HMD corresponding to operation of a trackpad, and shows the example of operation of a trackpad. Explanatory drawing which shows the example of the movement of the pointer corresponding to operation of a trackpad. Explanatory drawing which shows another example of the movement of the pointer with respect to operation of a trackpad. The schematic diagram which shows a response | compatibility with a trackpad, a virtual input plane, and a displayable area. The flowchart which shows operation | movement of HMD of 1st Embodiment. Explanatory drawing which shows the example which changes the display of ring type UI with respect to operation of a trackpad. The flowchart which shows operation | movement of HMD of 2nd Embodiment. Explanatory drawing which shows the example of operation in 3rd Embodiment. The flowchart which shows operation | movement of HMD of 3rd Embodiment. The flowchart which shows operation | movement of HMD of 4th Embodiment. The flowchart which shows operation | movement of HMD of 5th Embodiment. The flowchart which shows operation | movement of HMD of 5th Embodiment. Explanatory drawing which shows the example of operation in 5th Embodiment. The figure which shows the setting state of the operation area | region in the trackpad in 6th Embodiment. The flowchart which shows operation | movement of HMD of 6th Embodiment. The flowchart which shows operation | movement of HMD of 7th Embodiment. The flowchart which shows operation | movement of HMD of 8th Embodiment. The schematic diagram which shows a response | compatibility of a trackpad and a virtual input plane. The flowchart which shows operation | movement of HMD of 9th Embodiment. The figure which shows the modification of embodiment of this invention. The figure which shows the modification of embodiment of this invention. The figure which shows the modification of embodiment of this invention. The figure which shows the modification of embodiment of this invention.

[First Embodiment]
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is an explanatory diagram showing an external configuration of an HMD 100 according to the first embodiment to which the present invention is applied. The HMD 100 is a display device mounted on the head, and is also referred to as a head mounted display (Head Mounted Display). The HMD 100 of the present embodiment is an optically transmissive display device that allows a user to visually recognize a virtual image and at the same time directly view an outside scene. In this specification, a virtual image visually recognized by the user with the HMD 100 is also referred to as a “display image” for convenience. Moreover, emitting image light generated based on image data is also referred to as “displaying an image”.

  The HMD 100 includes an image display unit 20 (display unit) that allows the user to visually recognize a virtual image when mounted on the user's head, and a control device 10 that controls the image display unit 20. The control device 10 also functions as a controller for the user to operate the HMD 100. The control device 10 functions as an operation device.

  The image display unit 20 is a mounting body that is mounted on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a camera 61. And a microphone 63. The right optical image display unit 26 and the left optical image display unit 28 are arranged so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 20, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrow of the user when the user wears the image display unit 20.

  The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple of glasses.

  The right display drive unit 22 and the left display drive unit 24 are disposed on the side facing the user's head when the user wears the image display unit 20. The right display drive unit 22 and the left display drive unit 24 are collectively referred to simply as “display drive unit”, and the right optical image display unit 26 and the left optical image display unit 28 are simply referred to as “optical image display unit”. Also called.

  The display driving units 22 and 24 include liquid crystal displays 241 and 242 (hereinafter referred to as “LCDs 241 and 242”), projection optical systems 251 and 252 (see FIG. 2). Details of the configuration of the display driving units 22 and 24 will be described later. The optical image display units 26 and 28 as optical members include light guide plates 261 and 262 (see FIG. 2). The light guide plates 261 and 262 are formed of a light transmissive resin or the like, and guide the image light output from the display driving units 22 and 24 to the user's eyes. In the present embodiment, a case will be described in which at least the right optical image display unit 26 and the left optical image display unit 28 having such light transmittance that a user wearing the HMD 100 can visually recognize an outside scenery are used.

The camera 61 is disposed at the end ER that is the other end of the right optical image display unit 26. The camera 61 captures an outside scene that is an external scenery in a direction opposite to the user's eye side, and acquires an outside scene image. The camera 61 of this embodiment shown in FIG. 1 is a monocular camera, but may be a stereo camera.
The shooting direction of the camera 61, that is, the angle of view is the front side direction of the HMD 100, in other words, the direction of shooting at least a part of the outside scene in the user's viewing direction when the HMD 100 is worn. In addition, although the angle of view of the camera 61 can be set as appropriate, the imaging range of the camera 61 is a range including the outside world that the user can visually recognize through the right optical image display unit 26 and the left optical image display unit 28. It is preferable. Furthermore, it is more preferable that the imaging range of the camera 61 is set so that the entire field of view of the user through the right optical image display unit 26 and the left optical image display unit 28 can be captured.

  The image display unit 20 further includes a connection unit 40 for connecting the image display unit 20 to the control device 10. The connection unit 40 includes a main body cord 48 connected to the control device 10, a right cord 42, a left cord 44, and a connecting member 46. The right cord 42 and the left cord 44 are codes in which the main body cord 48 is branched into two. The right cord 42 is inserted into the casing of the right holding unit 21 from the distal end AP in the extending direction of the right holding unit 21 and connected to the right display driving unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the distal end AP in the extending direction of the left holding unit 23 and connected to the left display driving unit 24.

  The connecting member 46 is provided at a branch point between the main body cord 48, the right cord 42 and the left cord 44, and has a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30. A microphone 63 is provided in the vicinity of the earphone plug 30. The earphone plug 30 to the microphone 63 are combined into one cord, and the cord branches from the microphone 63 and is connected to each of the right earphone 32 and the left earphone 34.

  For example, as shown in FIG. 1, the microphone 63 is arranged such that the sound collection unit of the microphone 63 faces the user's line of sight, collects sound, and outputs a sound signal to the sound processing unit 190. For example, the microphone 63 may be a monaural microphone or a stereo microphone, may be a directional microphone, or may be an omnidirectional microphone.

  It is also possible to combine the right code 42 and the left code 44 into one code. Specifically, the lead wire inside the right cord 42 is drawn into the left holding portion 23 side through the inside of the main body of the image display unit 20, and is covered with a resin together with the lead wire inside the left cord 44 to be combined into one cord. Also good.

  The image display unit 20 and the control device 10 transmit various signals via the connection unit 40. A connector (not shown) that fits each other is provided at each of the end of the main body cord 48 opposite to the connecting member 46 and the control device 10. By fitting / releasing the connector of the main body cord 48 and the connector of the control device 10, the control device 10 and the image display unit 20 are connected or disconnected. For the right cord 42, the left cord 44, and the main body cord 48, for example, a metal cable or an optical fiber can be adopted.

  The control device 10 is a device for controlling the HMD 100. The control device 10 includes switches including an enter key 11, a lighting unit 12, a display switching key 13, a luminance switching key 15, a direction key 16, a menu key 17, and a power switch 18. In addition, the control device 10 includes a track pad 14 (operation surface) on which a user performs a touch operation with an operation body such as a finger.

  The determination key 11 detects a pressing operation and outputs a signal for determining the content operated by the control device 10. The lighting unit 12 notifies the operation state of the HMD 100 according to the light emission state. The operation state of the HMD 100 includes, for example, power ON / OFF. For example, an LED (Light Emitting Diode) is used as the lighting unit 12. The display switching key 13 detects a pressing operation and outputs a signal for switching the display mode of the content video between 3D and 2D, for example.

  The track pad 14 detects the operation of the user's finger on the operation surface of the track pad 14 and outputs a signal corresponding to the detected content. As the track pad 14, various track pads such as a capacitance detection type, a pressure detection type, and an optical type can be adopted. When the track pad 14 is a pressure detection type or an optical type, a variety of operating bodies for operating the track pad 14 such as a user's finger, a pen, and a pen-type dedicated indicator can be used. When the track pad 14 is a capacitance detection type, a part of a human body such as a finger, a capacitance stylus pen, or the like can be used as the operation body. Moreover, it is also possible to use what the user operates with a body other than the hand as the operating body. For example, the track pad 14 may be configured so that the user can operate to touch his / her feet, elbows, shoulders, waist, and the like. In this case, the position where the user's foot, elbow, shoulder, waist, etc. touched the track pad 14 is detected as the operation position, and if the contact position moves, the movement of the operation position is detected. This configuration can be realized by arranging the track pad 14 at a position where the user can easily touch his / her feet, elbows, shoulders, waist, and the like. The size of the track pad 14 may be a size suitable for the user to contact his / her legs, elbows, shoulders, waist, and the like. Further, a sheet-like or plate-like device having the track pad 14 may be provided separately from the control device 10 so that the device and the control device 10 are connected.

  The luminance switching key 15 detects a pressing operation and outputs a signal for increasing or decreasing the luminance of the image display unit 20. The direction key 16 detects a pressing operation on a key corresponding to the up / down / left / right direction, and outputs a signal corresponding to the detected content. The power switch 18 is a switch for switching the power-on state of the HMD 100, and is configured to be capable of a slide operation, for example.

FIG. 2 is a functional block diagram of each part constituting the HMD 100.
As shown in FIG. 2, the HMD 100 is connected to the external device OA via the interface 125. The interface 125 is an interface for connecting various external devices OA that are content supply sources to the control device 10. As the interface 125, for example, an interface corresponding to a wired connection such as a USB interface, a micro USB interface, or a memory card interface can be used.
The external device OA is used as an image supply device that supplies an image to the HMD 100. For example, a personal computer (PC), a mobile phone terminal, a game terminal, or the like is used.

  The control device 10 of the HMD 100 includes a control unit 140, an operation unit 111, an input information acquisition unit 110 (operation detection unit), a storage unit 120, a transmission unit (Tx) 51, and a transmission unit (Tx) 52. doing.

  The operation unit 111 detects an operation by the user. The operation unit 111 includes the respective operators of the determination key 11, the display switching key 13, the track pad 14, the luminance switching key 15, the direction key 16, the menu key 17, and the power switch 18 illustrated in FIG.

  The input information acquisition unit 110 detects the operation of each operation element of the operation unit 111 and outputs operation data indicating the operation content to the control unit 140. Specifically, the operation data includes data specifying the operated operator and data indicating the type of operation of the operator. For example, it is assumed that the enter key 11, the display switching key 13, the luminance switching key 15, the menu key 17, or the power switch 18 is operated. In this case, the input information acquisition unit 110 outputs operation data including data indicating the operated key or switch and data indicating the operation “present” to the control unit 140. For example, since the direction key 16 can be operated in four directions, when the input information acquisition unit 110 detects the operation of the direction key 16, the input information acquisition unit 110 sends the data indicating the direction key 16 and the data indicating the operation direction to the control unit 140. Output.

  When an operation on the track pad 14 is detected, the input information acquisition unit 110 detects a position (operation position) at which the touch operation is performed on the track pad 14. The input information acquisition unit 110 outputs operation data including data indicating the operation position and data indicating that the operated operator is the track pad 14 to the control unit 140. This operation data includes, for example, data indicating the operation position as a relative position in the operation area of the track pad 14. The specific mode of the operation data is arbitrary.

  The storage unit 120 is a nonvolatile storage device, and stores various computer programs. The storage unit 120 may store image data to be displayed on the image display unit 20 of the HMD 100. For example, the storage unit 120 stores setting data 121 including setting values related to the operation of the HMD 100 and content data 123 including character and image data that the control unit 140 displays on the image display unit 20.

  In addition, a three-axis sensor 113, a GPS 115, a communication unit 117, and a voice recognition unit 114 are connected to the control unit 140. The triaxial sensor 113 (attitude detection unit) is a triaxial acceleration sensor, and the control unit 140 can acquire the detection value of the triaxial sensor 113. The GPS 115 includes an antenna (not shown), receives a GPS (Global Positioning System) signal, and obtains the current position of the control device 10. The GPS 115 outputs the current position and the current time obtained based on the GPS signal to the control unit 140. Further, the GPS 115 may have a function of acquiring the current time based on information included in the GPS signal and correcting the time counted by the control unit 140 of the control device 10.

The communication unit 117 executes wireless data communication conforming to a wireless communication standard such as a wireless LAN (WiFi (registered trademark)) or Miracast (registered trademark). The communication unit 117 can also perform wireless data communication conforming to a short-range wireless communication standard such as Bluetooth (registered trademark), Bluetooth Low Energy, RFID, or Felica (registered trademark).
When the external device OA is wirelessly connected to the communication unit 117, the control unit 140 acquires content data from the communication unit 117 and performs control for displaying an image on the image display unit 20. On the other hand, when the external device OA is wired to the interface 125, the control unit 140 acquires content data from the interface 125 and performs control for displaying an image on the image display unit 20. Therefore, the communication unit 117 and the interface 125 are hereinafter collectively referred to as a data acquisition unit DA.
The data acquisition unit DA acquires content data from the external device OA. The data acquisition unit DA acquires image data displayed by the HMD 100 from the external device OA.

  The voice recognition unit 114 extracts features from digital voice data collected by the microphone 63 and converted into digital data by a voice processing unit 190, which will be described later, and models it. The voice recognition unit 114 extracts and models the features of the voice, recognizes the voices of a plurality of people separately, and recognizes the person who speaks for each voice, or converts the voice into text. Perform text conversion to convert. The voice recognition unit 114 may be configured to identify the language type of the voice data in the voice recognition process.

  The control unit 140 includes a CPU, a ROM, a RAM (not shown), and the like as hardware. The control unit 140 reads out and executes a computer program stored in the storage unit 120, thereby executing functions of each unit including the operating system (OS) 150. Specifically, the control unit 140 functions as an image processing unit 160, a display control unit 170, an operation detection unit 183, a GUI control unit 185, and an audio processing unit 190.

The image processing unit 160 outputs image data (Data in the figure) of an image to be displayed such as a vertical synchronization signal VSync, a horizontal synchronization signal HSync, and a clock signal PCLK for displaying content.
The image data of the content displayed by the processing of the image processing unit 160 may be image data generated by the processing of the control unit 140 in addition to being received via the interface 125 and the communication unit 117. For example, image data can be generated and displayed in response to the operation of the operation unit 111 during execution of the game application program.
Note that the image processing unit 160 may perform image processing such as resolution conversion processing, various tone correction processing such as adjustment of luminance and saturation, and keystone correction processing on the image data as necessary. Good.

  The image processing unit 160 transmits the generated clock signal PCLK, vertical synchronization signal VSync, horizontal synchronization signal HSync, and image data Data stored in the DRAM in the storage unit 120 via the transmission units 51 and 52, respectively. . The image data Data transmitted via the transmission unit 51 is also referred to as “right eye image data”, and the image data Data transmitted via the transmission unit 52 is also referred to as “left eye image data”. The transmission units 51 and 52 function as a transceiver for serial transmission between the control device 10 and the image display unit 20.

The display control unit 170 generates control signals for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the display control unit 170 individually controls driving ON / OFF of the right LCD 241 by the right LCD control unit 211 and driving ON / OFF of the right backlight 221 by the right backlight control unit 201 based on the control signal. . Further, the display control unit 170 individually controls ON / OFF driving of the left LCD 242 by the left LCD control unit 212 and ON / OFF driving of the left backlight 222 by the left backlight control unit 202.
Thereby, the display control unit 170 controls the generation and emission of image light by the right display driving unit 22 and the left display driving unit 24, respectively. For example, the display control unit 170 causes the right display driving unit 22 and the left display driving unit 24 to generate image light, or causes only one to generate image light. In addition, the display control unit 170 can prevent both the right display drive unit 22 and the left display drive unit 24 from generating image light.
The display control unit 170 transmits a control signal for the right LCD control unit 211 and a control signal for the left LCD control unit 212 via the transmission unit 51 and the transmission unit 52, respectively. In addition, the display control unit 170 transmits a control signal for the right backlight control unit 201 to the right backlight control unit 201, and a control signal for the left backlight control unit 202 is transmitted by the left backlight control unit 202.

The display control unit 170 displays the AR content by the image display unit 20. The AR content includes characters and images that are displayed in correspondence with the position where the object is viewed while the user is looking at the object through the image display unit 20. For example, the AR content is displayed so as to overlap or avoid the object in the outside scene that can be seen through the image display unit 20, that is, the object existing in the real space. The display control unit 170 provides information about the object by performing AR display that displays an image, characters, or the like at a position corresponding to the object, or the appearance of the object seen through the image display unit 20 Change the way.
The display position of the AR content may be a position overlapping the target object or may be around the target object. The object may be an object, and may be a non-movable object such as a building wall, or may be a natural object.

  The AR content is displayed based on the content data 123 stored in the storage unit 120 or data generated by the processing of the control unit 140. These data can include image data and text data.

  The display control unit 170 detects the position where the user visually recognizes the target object, and determines the display position of the AR content so as to correspond to the detected position. Although the method for detecting the position at which the user visually recognizes the object is arbitrary, in this embodiment, the display control unit 170 detects the object located in the user's field of view from the captured image data of the camera 61. In the process of extracting or detecting the image of the target object by analyzing the captured image data, feature amount data relating to the shape, color, size, etc. of the target object is used. These data can be included in the content data 123. The display control unit 170 detects the image of the target object from the captured image data of the camera 61 and specifies the position of the target object in the imaging range of the camera 61 based on the positional relationship between the detected target image and the entire captured image. To do. Then, the display control unit 170 determines the display position of the AR content corresponding to the position of the target object based on the positional relationship between the imaging range of the camera 61 and the display area of the image display unit 20.

  The audio processing unit 190 acquires an audio signal included in the content, amplifies the acquired audio signal, and a speaker (not shown) in the right earphone 32 and a speaker in the left earphone 34 (not shown) connected to the connecting member 46. (Not shown). For example, when the Dolby (registered trademark) system is adopted, processing on the audio signal is performed, and different sounds with different frequencies or the like are output from the right earphone 32 and the left earphone 34, respectively.

  The image display unit 20 includes an interface 25, a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, and a left light guide plate 262 as a left optical image display unit 28. A camera 61, a vibration sensor 65, and a nine-axis sensor 66.

  The vibration sensor 65 is configured using an acceleration sensor and is disposed inside the image display unit 20. For example, the vibration sensor 65 is built in the vicinity of the end ER of the right optical image display unit 26 in the right holding unit 21. When the user performs an operation of hitting the end ER (knock operation), the vibration sensor 65 detects vibration caused by this operation and outputs the detection result to the control unit 140. Based on the detection result of the vibration sensor 65, the control unit 140 detects a knocking operation by the user.

  The 9-axis sensor 66 is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). Since the 9-axis sensor 66 is provided in the image display unit 20, the control unit 140 is used based on the detection value of the 9-axis sensor 66 when the image display unit 20 is worn on the user's head. The movement of the person's head can be detected. Since the orientation of the image display unit 20 is known from the detected movement of the user's head, the control unit 140 can estimate the user's line-of-sight direction.

The interface 25 includes a connector to which the right cord 42 and the left cord 44 are connected. The interface 25 outputs the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data Data transmitted from the transmission unit 51 to the corresponding reception units (Rx) 53 and 54. Further, the interface 25 outputs a control signal transmitted from the display control unit 170 to the corresponding reception units 53 and 54, the right backlight control unit 201, or the left backlight control unit 202.
The interface 25 is an interface of the camera 61, the vibration sensor 65, and the 9-axis sensor 66. The detection result of vibration by the vibration sensor 65 and the detection result of acceleration (three axes), angular velocity (three axes), and geomagnetism (three axes) by the nine-axis sensor 66 are sent to the control unit 140 of the control device 10 via the interface 25. Sent.

  The right display driving unit 22 includes a receiving unit 53, a right backlight (BL) control unit 201 and a right backlight (BL) 221 that function as light sources, a right LCD control unit 211 and a right LCD 241 that function as display elements, A right projection optical system 251. The right backlight control unit 201 and the right backlight 221 function as a light source. The right LCD control unit 211 and the right LCD 241 function as display elements. The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are also collectively referred to as “image light generation unit”.

  The receiving unit 53 functions as a receiver for serial transmission between the control device 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is a light emitter such as an LED or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right eye image data Data input via the reception unit 53. The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

  The right projection optical system 251 is configured by a collimating lens that converts the image light emitted from the right LCD 241 into a light beam in a parallel state. The right light guide plate 261 as the right optical image display unit 26 guides the image light output from the right projection optical system 251 to the right eye RE of the user while reflecting the image light along a predetermined optical path.

  The left display drive unit 24 has the same configuration as the right display drive unit 22. The left display driving unit 24 includes a receiving unit 54, a left backlight (BL) control unit 202 and a left backlight (BL) 222 that function as light sources, a left LCD control unit 212 and a left LCD 242 that function as display elements, A left projection optical system 252. The left backlight control unit 202 and the left backlight 222 function as a light source. The left LCD control unit 212 and the left LCD 242 function as display elements. The left projection optical system 252 is configured by a collimating lens that converts the image light emitted from the left LCD 242 into a light beam in a parallel state. The left light guide plate 262 as the left optical image display unit 28 guides the image light output from the left projection optical system 252 to the left eye LE of the user while reflecting the image light along a predetermined optical path.

The control unit 140 can display a GUI (Graphical User Interface) in addition to the AR content by the image display unit 20. The GUI control unit 185 controls the display of the GUI by the image display unit 20, and displays images such as markers, pointers (including so-called mouse pointers), icons, etc., which are GUI operation indices (GUI operation position indices). Displays GUI objects containing characters. The GUI object corresponds to the display object of the present invention.
The GUI control unit 185 updates the display state of the GUI object in response to the operation of the operation unit 111. In the present embodiment, the GUI control unit 185 displays a pointer corresponding to the operation position operated on the track pad 14. The display position of the pointer in the display area of the image display unit 20 and the operation position on the track pad 14 are associated in advance. When the coordinates of the operation position of the track pad 14 are input from the operation detection unit 183, the GUI control unit 185 displays a pointer at the display position corresponding to the input operation position, or displays the pointer being displayed. Change the position.

The operation detection unit 183 detects an operation performed on each operation element of the operation unit 111 based on the operation data input from the input information acquisition unit 110, and a process executed by the control unit 140 in response to the operation To decide.
The operation detection unit 183 analyzes the operation data, and when the operated operator is the enter key 11, the display switching key 13, the brightness switching key 15, the menu key 17, or the power switch 18, the operation detection unit 183 responds to these operations. Select the process to be performed. For the operation of the decision key 11, a decision process for a menu screen or the like displayed under the control of the GUI control unit 185 is selected. When the display switching key 13 is operated, a process for switching the screen to be displayed is selected. For example, the operation detection unit 183 selects processing for sequentially switching a plurality of display states including a state of displaying a screen and a state of displaying the content data 123 under the control of the GUI control unit 185. When the luminance switching key 15 is operated, a process for adjusting the display luminance in the image display unit 20 is selected. Specifically, the display control unit 170 controls the image processing unit 160 to select a process for changing the luminance of the image data output to the right display driving unit 22 and the left display driving unit 24. Alternatively, a process in which the display control unit 170 changes the luminance of the right backlight control unit 201 and the left backlight control unit 202 is selected.

  When the direction key 16 is operated, the operation detection unit 183 selects processing for a menu screen or the like displayed under the control of the GUI control unit 185 corresponding to the operation direction of the direction key 16. When the power switch 18 is operated, the operation detection unit 183 selects a shutdown process for turning off the power of the HMD 100 or a startup sequence for turning on the power of the HMD 100.

When the track pad 14 is operated, the operation detection unit 183 obtains coordinates at which the GUI control unit 185 performs display processing corresponding to the coordinates of the operation position of the track pad 14.
In the present embodiment, the GUI control unit 185 displays a display object corresponding to the operation position of the track pad 14. Also, the GUI control unit 185 changes the display position of the display object in response to the operation of the track pad 14. Accordingly, when a user (user, operator) who operates the control device 10 performs an operation (touch operation) to contact the track pad 14, the display position of the display object displayed on the image display unit 20 moves.

  FIG. 3 is an explanatory diagram showing the operation of the HMD 100 corresponding to the operation of the track pad 14, and shows an operation example of the track pad 14.

The operation with respect to the track pad 14 includes an operation of moving the track pad 14 while keeping a finger or the like in contact with the track pad 14 as shown in FIG. The operation illustrated in FIG. 3 is an operation in which the user's hand H contacts the track pad 14 at the operation position A1, and the hand H moves from the operation position A1 to the operation position A2 while maintaining the contact state. In this example, the hand H is used as the operating tool. As indicated by reference characters X and Y in FIG. 3, in the present embodiment, XY orthogonal coordinates having the upper left corner of the track pad 14 as a reference position (for example, the origin) are set. When the input information acquisition unit 110 detects contact of the hand H with the track pad 14, the input information acquisition unit 110 generates and outputs operation data indicating an operation position that is a position where the hand H is in contact. The operation detection unit 183 acquires or specifies the X coordinate and Y coordinate of the operation position based on the operation data input from the operation unit 111.
Here, the operation data generated and output by the input information acquisition unit 110 may include the X coordinate and the Y coordinate of the operation position. Further, the input information acquisition unit 110 may generate and output an analog detection value or digital data for specifying the operation position on the track pad 14. In this case, the operation detection unit 183 performs processing for obtaining the X coordinate and the Y coordinate of the operation position based on the analog detection value output from the input information acquisition unit 110 or digital data.

The operation detection unit 183 generates input data based on the coordinates of the operation position. The input data is data obtained by converting the position input by operating the track pad 14 into a position on the input plane of the operating system 150.
In the present embodiment, the operating system 150 sets a virtual input plane in order to accept a two-dimensional position input operation. The operation detection unit 183 generates input data indicating a position on the input plane of the operating system 150 (hereinafter referred to as an input position) based on the operation data output from the input information acquisition unit 110. This virtual input plane is referred to as a virtual input plane IP in the following description. The operation detection unit 183 outputs input data indicating the input position on the virtual input plane IP to the GUI control unit 185. The virtual input plane IP is not limited to that defined in the operating system 150. For example, a configuration in which a coordinate system is defined for an application program running on the operating system 150 or corresponding to a programming language that describes the application program is conceivable. In this case, the present invention can also be applied to a configuration in which the virtual input plane IP corresponding to the defined coordinate system can be set and the control unit 140 generates and outputs the input data corresponding to these.
The virtual input plane IP can be used not only when receiving a two-dimensional position input. For example, a configuration in which the control device 10 including the operating system 150 receives a three-dimensional position input is conceivable. In this configuration, the control device 10 can detect a two-dimensional position input in the virtual input plane IP and a position input having, for example, a direction perpendicular to the virtual input plane IP as a coordinate axis with reference to the virtual input plane IP. It may be a configuration. In this case, the control device 10 can execute a process by receiving a three-dimensional position input with reference to the virtual input plane IP.

The movement of the operating body (for example, hand H) when operating the track pad 14 is called a gesture. The gesture includes a movement that touches the track pad 14, a movement that moves while the operating body is in contact as shown in FIG. 3, and a movement that taps (tap) the track pad 14. In the present embodiment, the movement in which the operating body moves while being in contact with the track pad 14 as shown in FIG. 3 is referred to as a slide gesture.
When the track pad 14 is a multi-touch compatible type capable of detecting simultaneous operations on a plurality of positions, a plurality of operating bodies come into contact with the track pad 14 and a gesture of moving the contact positions of the plurality of operating bodies is performed. Can do.

FIG. 4 is an explanatory diagram showing an example of movement of the pointer P corresponding to the operation of the track pad 14. FIG. 5 is an explanatory diagram showing an example of movement of the pointer P corresponding to the operation of the track pad 14.
The GUI control unit 185 displays the pointer P as shown in FIG. 4 corresponding to the position specified by the input data generated by the operation detection unit 183. FIG. 4 shows a visual field VR of a user wearing the image display unit 20, and a displayable area where the right display drive unit 22 and the left display drive unit 24 can display an image is indicated by a symbol D. In the present embodiment, since the same display is performed on both the right display drive unit 22 and the left display drive unit 24 of the image display unit 20, the right display drive unit 22 and the left display drive unit 24 are distinguished in FIG. do not do. The same applies to FIGS. 5, 6, and 8.

  The pointer P is displayed in the displayable area D by the GUI control unit 185 controlling the display control unit 170 and the image processing unit 160. The pointer P is an index that is displayed for GUI operation and indicates an operation target position in the display area. The pointer P is an example of a display object. The shape of the pointer P may not be circular, and the shape and size of the pointer P such as a geometric figure such as a quadrangle or a triangle, a symbol, an arrow, etc. are not limited.

  In the example of FIG. 3, while the position of the hand H moves from the operation position A1 to the operation position A2, the input information acquisition unit 110 detects the operation position on the track pad 14 at a preset sampling cycle. Therefore, the input information acquisition unit 110 outputs operation data indicating the operation position on the track pad 14 in accordance with the sampling period. For example, every time the input information acquisition unit 110 outputs operation data, the operation detection unit 183 generates input data corresponding to the operation data and outputs the input data to the GUI control unit 185. The GUI control unit 185 obtains the display position of the pointer P corresponding to the input position indicated by the input data, and displays the pointer P at the obtained display position. As a result, in response to the operation in which the hand H moves from the operation position A1 to the operation position A2 in FIG. 3, the GUI control unit 185 operates from the display position B1 corresponding to the operation position A1, as shown in FIG. The pointer P is moved to the display position B2 corresponding to the position A2. In the example of FIG. 4, the GUI control unit 185 repeatedly executes a process of displaying the pointer P at a plurality of positions between the display position B1 and the display position B2.

  In the normal process for the input operation, the coordinates of the operation position on the track pad 14, the coordinates of the input position on the virtual input plane IP, and the coordinates of the display position in the displayable area D are associated in advance. The GUI control unit 185 obtains the display positions on the right LCD 241 and the left LCD 242 corresponding to the coordinates on the displayable area D, and displays the image of the pointer P at the display positions. Thereby, the display position of the pointer P moves corresponding to the operation.

As described with reference to FIG. 1, the control device 10 is a device carried by the user, and since it is not easy to increase the size of the track pad 14, the display device D of the displayable region D that the user can visually recognize in the visual field VR is used. The track pad 14 is smaller than the size.
Therefore, in the present embodiment, the correspondence between the operation position on the track pad 14 and the display position of the displayable area D is changed by the processing of the operation detection unit 183.

  In a normal process for an input operation, the operation detection unit 183 generates input data indicating an input position corresponding to the operation position actually detected on the track pad 14 and passes it to the GUI control unit 185. Therefore, in response to the operation from the operation position A1 to the operation position A2 shown in FIG. 3, the GUI control unit 185 displays the pointer P from the display position B1 to the display position B2 in FIG.

  The operation detection unit 183 can perform an input process different from the normal process in response to the slide gesture. In this input process, the operation detection unit 183 generates input data indicating input positions corresponding to the coordinates included in the operation data, and also generates input data indicating input positions not corresponding to the coordinates included in the operation data. Specifically, the operation detection unit 183 generates input data so that the input position moves on the virtual input plane IP in a state where the operation position on the track pad 14 does not move, and outputs the input data to the GUI control unit 185. In this case, the operation detection unit 183 continuously generates input data a plurality of times, and sets the input position in each input data to a different position. In this case, the position at which the GUI control unit 185 displays the pointer P moves according to the input position on the virtual input plane IP. The operation of the GUI control unit 185 is the same whether it is a normal process or an input process different from the normal process.

  In this case, in response to the operation of moving the hand H from the operation position A1 to the operation position A2 shown in FIG. 3, the GUI control unit 185 changes the display position of the pointer P from the display position B1 as shown in FIG. Move to display position B3. The distance between the display positions B1 and B3 is longer than the distance between the display positions B1 and B2 shown in FIG. That is, in the normal processing, the amount of movement of the display position of the pointer P corresponds to the amount of movement of the hand H on the track pad 14, but the HMD 100 is larger than the amount of movement of the hand H on the track pad 14. Can be moved.

FIG. 6 is a schematic diagram showing the correspondence between the track pad 14, the virtual input plane IP, and the displayable area D.
In the present embodiment, the virtual input plane IP that is virtually set by the operating system 150 is defined as a plane parallel to the track pad 14. For this reason, the orthogonal coordinates constituted by the X axis and the Y axis in the track pad 14 and the orthogonal coordinate system constituted by the X axis and the Y axis in the virtual input plane IP can be superimposed. The displayable area D is defined as a plane parallel to the virtual input plane IP. For this reason, the orthogonal coordinate system configured by the X axis and the Y axis in the virtual input plane IP and the orthogonal coordinate configured by the X axis and the Y axis in the displayable region D can be overlapped.

  The sizes of the track pad 14, the virtual input plane IP, and the displayable area D may be different. Further, the resolution of the coordinates of the operation position A on the track pad 14, the resolution of the coordinates of the input coordinate IA on the virtual input plane IP, and the resolution of the display position B in the displayable area D may be different.

  In this configuration, the operation detection unit 183 can convert the coordinates of the operation position on the track pad 14 into the coordinates of the input position on the virtual input plane IP by an arithmetic expression, a function, a matrix, or the like. Similarly, the GUI control unit 185 can convert the coordinates of the input position in the virtual input plane IP into the coordinates of the display position in the displayable area D. An arithmetic expression, a function, a matrix, a parameter, and the like required for this conversion are stored in advance in the storage unit 120 in a format included in the setting data 121, for example.

  As described above, when the hand H moves from the operation position A1 of the track pad 14 to the operation position A2, the operation detection unit 183 moves from the input position IA1 corresponding to the operation position A1 to the input position IA2 corresponding to the operation position A2. Until the input data is generated, the input position moves. In this case, the GUI control unit 185 displays the pointer P at the display position B1 corresponding to the input position IA1, and moves the display position of the pointer P to the display position B2 corresponding to the input position IA2. Here, the operation detection unit 183 can generate input data that moves from the input position IA2 to the input position IA3 while the hand H stops at the operation position A2. Actually, the operation detection unit 183 sets the input position IA1 corresponding to the operation position A1 as an initial position, determines the movement direction and movement amount of the input position from this initial position, and determines the determined movement direction and movement amount. The input data is continuously generated so as to correspond to. Here, the operation detection unit 183 may calculate the coordinates of the input position IA3 that is the end position of the movement of the input position after determining the moving direction and the moving amount of the input position. In this case, the GUI control unit 185 moves the display position of the pointer P from the display position B1 to the display position B3 corresponding to the input position IA3.

The operation detection unit 183 analyzes the operation data input from the input information acquisition unit 110 and determines the movement direction of the input position based on the movement direction of the operation position of the track pad 14. In addition, the operation detection unit 183 determines the movement amount of the input position based on at least one of the movement amount and movement speed of the operation position of the track pad 14 and the operation time of the track pad 14.
For example, the operation detection unit 183 may obtain the amount of movement of the input position after the amount of movement of the operation position is determined by stopping the operation position on the track pad 14. In this case, the operation detection unit 183 may generate the input data while maintaining the input position at the initial position until the operation position on the track pad 14 stops. Alternatively, input data may be generated by obtaining an input position corresponding to the operation position of the track pad 14 according to the sampling period of the input information acquisition unit 110.

  Further, for example, the operation detection unit 183 may determine the amount of movement of the input position based on the time from detection of the hand H touching the track pad 14 to detection of contact release. Good. In this case, the operation detection unit 183 may generate the input data while maintaining the input position at the initial position until the contact with the track pad 14 is released. Alternatively, input data may be generated by obtaining an input position corresponding to the operation position of the track pad 14 according to the sampling period of the input information acquisition unit 110.

  For example, the operation detection unit 183 obtains the movement speed of the operation position within a predetermined time after the input information acquisition unit 110 detects the operation of the track pad 14, and moves the input position corresponding to the obtained movement speed. The amount may be determined. This predetermined time is set in advance, and is included in the setting data 121 and stored in the storage unit 120, for example. In this case, the operation detection unit 183 may generate the input data while maintaining the input position corresponding to the initial position of the operation position of the track pad 14 within the predetermined time. Alternatively, input data may be generated by obtaining an input position corresponding to the operation position of the track pad 14 according to the sampling period of the input information acquisition unit 110.

Further, for example, the operation detection unit 183 obtains the movement direction of the operation position within a predetermined time after the input information acquisition unit 110 detects the operation of the track pad 14, and moves the input position corresponding to the obtained movement direction. The direction may be set. This predetermined time is set in advance, and is included in the setting data 121 and stored in the storage unit 120, for example.
In this case, the operation detection unit 183 does not limit the amount of movement of the input position, but moves the input position at a preset movement speed or a movement speed of the input position corresponding to the movement speed of the operation position. Input data may be generated. In this case, the operation detection unit 183 continues to generate and output input data for moving the input position at the set moving direction and moving speed until the contact with the track pad 14 is released. A cycle for generating and outputting input data may be a preset cycle, or may be in accordance with a sampling cycle of the input information acquisition unit 110. Then, the operation detection unit 183 stops the generation and output of the input data at the timing when the input information acquisition unit 110 detects the release of contact on the track pad 14.

In any of these cases, the operation detection unit 183 continuously performs the process of generating input data until the hand H that is the operation body leaves the track pad 14, so that the GUI control unit 185 can display the data. In the region D, the display position of the pointer P continues to move.
When the user intentionally performs a slide gesture, the pointer P can be moved larger than the actual movement amount of the operating body. Further, the pointer P can be stopped by separating the operating body from the track pad 14, and the stop position of the pointer P can be controlled with high accuracy.

FIG. 7 is a flowchart showing the operation of the HMD 100 and shows the operation of controlling the display position of the pointer P in response to the operation of the track pad 14.
When the control unit 140 detects a touch operation on the track pad 14 based on the operation data input from the input information acquisition unit 110 (step S11), the control unit 140 acquires the coordinates of the operation position (step S12). The control unit 140 determines whether or not the operation position has moved, that is, whether or not the operation position corresponds to a slide gesture (step S13). For example, when the coordinate of the operation position changes more than a threshold value within a preset time, it can be determined as a slide gesture, and the determination condition of the slide gesture can be included in the setting data 121 and stored. .

  When it determines with it not being a slide gesture (step S13; No), the control part 140 displays the pointer P corresponding to the operation position on the trackpad 14 (step S14). In step S14, the control unit 140 generates input data corresponding to the operation position, displays the pointer P at the position indicated by the input data, or moves the display position of the pointer P. After the process of step S14, the control unit 140 ends this process. After the processing is completed, the control unit 140 waits for a contact operation to the track pad 14.

  When it determines with operation to the trackpad 14 being a slide gesture (step S13; Yes), the control part 140 waits until an operation position stops (step S15). During this standby, the control unit 140 continues to detect the operation position based on the operation data input from the input information acquisition unit 110.

When the operation position is stopped (step S15; Yes), the control unit 140 calculates a movement parameter of the input data in order to generate input data corresponding to the slide gesture (step S16). The input data movement parameters include, for example, data indicating the start position of the input position movement, the movement direction of the input position, and the movement speed of the input position.
The “when the operation position is stopped” in step S15 includes not only the case where the operation position on the track pad 14 is stationary, but also the case where the operation position moves within a preset threshold range. The control unit 140 is both in the case where the coordinate of the operation position of the track pad 14 detected by the input information acquisition unit 110 in the sampling period does not change and the case where the coordinate change amount is within a preset threshold range. It is determined that the operation has stopped. In addition, since the input information acquisition unit 110 detects the coordinates of the operation position of the track pad 14 at the sampling period, the coordinates of the operation position detected by a predetermined number of consecutive detection operations are within the threshold range. The operation position may be determined to have stopped.

  For example, the control unit 140 sets the moving direction of the input position in the same direction as the moving direction of the slide gesture detected in steps S11 to S15. The direction of the slide gesture with respect to the track pad 14 is not limited to the X direction shown in FIG. 3, and may be the Y direction or an oblique direction, and is not limited as long as it moves within the plane of the track pad 14. The movement direction of the input position only needs to correspond to the direction of the slide gesture, and may be a direction in the virtual input plane IP. Accordingly, the movement direction of the input position determined by the operation detection unit 183 corresponding to the movement direction of the operation position of the track pad 14 may be a direction that does not coincide with the movement direction of the operation position.

  For example, the control unit 140 sets the movement speed of the input position to a speed corresponding to the movement amount (operation amount) of the slide gesture detected in steps S11 to S15. For example, the control unit 140 sets the movement speed so that the movement speed of the input position becomes higher as the movement amount of the slide gesture is larger. Further, the moving speed of the input position may be constant. In this case, the control unit 140 sets a preset default moving speed as the moving speed of the input position. The default moving speed is set in advance and is included in the setting data 121, for example.

The control unit 140 starts generating input data according to the movement parameter set in step S16, and starts a process of displaying the pointer P on the right display driving unit 22 and the left display driving unit 24 based on the generated input data (step). S17). After step S17, the input position moves in the input data generated by the control unit 140, and the display position of the pointer P displayed by the image display unit 20 moves. Thereafter, the control unit 140 continues the process of generating the input data at a predetermined cycle (for example, the sampling cycle of the input information acquisition unit 110). Further, the control unit 140 continues the process of moving the display position of the pointer P based on the generated input data.
While the display position of the pointer P moves, the control unit 140 monitors the release of contact with the track pad 14 (step S18). If it is determined that the contact with the track pad 14 is not released (step S18; No), the generation of input data and the movement of the display position of the pointer P are continued.
When the control unit 140 detects that the contact with the track pad 14 is released (step S18; Yes), it stops generating the input data, thereby stopping the movement of the pointer P (step S19). The process ends.

The operation shown in FIG. 7 is applicable not only to the movement of the pointer P but also to, for example, a ring-shaped UI.
FIG. 8 is an explanatory diagram showing an example in which the display of a ring-type UI (User Interface) is changed in response to the operation of the track pad 14.
The ring-type UI is an interface in which a plurality of screens or display objects such as images and characters are arranged in a ring shape, and a screen or display object at a specific position on the ring is displayed in a large size. In the displayable area D, an image simulating a state in which the whole of the plurality of display objects arranged in a ring shape is looked down is displayed. With this display, the user can see the arrangement of the display objects constituting the ring UI and the contents of the display objects at a specific position by the displayable area D.

  In the example of FIG. 8, six screens M1 to M6 are prepared as display screens, and the screens M1 to M6 are sequentially enlarged and displayed by the image display unit 20 in response to an operation in the left-right direction. Further, the screens M1 to M6 can be switched and displayed in order in the right direction and the left direction in the figure.

  The GUI control unit 185 can sequentially switch and display the screens M1 to M6 corresponding to the slide gesture of the track pad 14. That is, the GUI control unit 185 acquires the input data generated by the operation detection unit 183, rotates the ring UI in the direction corresponding to the change in the input position included in the input data, and displays the screens M1 to M6. Display in order. As illustrated in FIG. 6, the input data generated by the operation detection unit 183 is two-dimensional data indicating the input position with coordinates in the XY coordinate system. In this case, the GUI control unit 185 may calculate both the change in the X coordinate and the change in the Y coordinate of the input position, and obtain the number of screens to be moved and the moving direction in the ring UI. Further, the GUI control unit 185 may obtain the number of screens moving in the ring UI and the moving direction based on either the X coordinate or the Y coordinate of the input position.

  In the ring-type UI of FIG. 8, each of the screens M1 to M6 corresponds to a display object of the present invention, and the GUI control unit 185 displays the display positions of the respective screens M1 to M6 in response to the operation of the track pad 14. Move. 7 is applied, when a slide gesture of the track pad 14 is detected, the screens M1, M2, M3... Are sequentially switched, and the screen is switched until the contact with the track pad 14 is released. continue. Then, when the user releases the operating tool from the track pad 14, the screen switching is stopped. Thus, not only the movement of the display position of the pointer P but also the operability of the track pad 14 can be improved by applying the present invention. In addition, when the ring-type UI of FIG. 8 is employed, the GUI control unit 185 may set a screen displayed at a specific position among the screens M1 to M6 as an operation target. The specific position is, for example, the center of the display screen, which is larger than the other screens, and the screen displayed in front of the user is the operation target. In this case, the display position of the operation target screen is moved in accordance with the direction of the operation on the track pad 14, and the display positions of other screens are appropriately moved according to the change in the display position of the operation target screen.

  When it is determined that the operation on the track pad 14 is not a slide gesture (step S13; No), the control unit 140 according to the present embodiment generates input data indicating an input position corresponding to the operation position on the track pad 14. Thereby, the pointer P is displayed at the display position of the displayable area D corresponding to the operation position on the track pad 14 (step S14). As described above, in this embodiment, as an example, the coordinates of the operation position on the track pad 14 detected by the input information acquisition unit 110 are converted into the input position on the virtual input plane IP. That is, the track pad 14 functions as a device for inputting absolute coordinates. The configuration of the HMD 100 is not limited to this. For example, the input information acquisition unit 110 may output a change in the operation position on the track pad 14 to the control unit 140 as a detection value. In this case, the control unit 140 may generate input data indicating a change in the input position corresponding to the change amount of the operation position of the track pad 14. That is, the track pad 14 may function as an input device for inputting a relative position, like a mouse or the like.

  As described above, the HMD 100 according to the first embodiment to which the present invention is applied includes the image display unit 20 that allows a user to visually recognize an image and transmits an outside scene. The HMD 100 includes an input information acquisition unit 110 that detects an operation on the track pad 14 and a control unit 140 that displays a display object to be operated such as a pointer P on the image display unit 20. The control unit 140 moves the display position of the pointer P in response to the change in the operation position with respect to the track pad 14 detected by the input information acquisition unit 110. The control unit 140 stops the movement of the display position of the pointer P when the input information acquisition unit 110 detects an operation corresponding to a preset end condition. In other words, the control unit 140 moves the display position of the pointer P so that the pointer P that is a display object maintains inertia until the operation on the track pad 14 that is the operation surface is completed. For this reason, the pointer P can be moved larger than the movement amount of the operation position on the track pad 14, and the operability restriction due to the size of the track pad 14 can be relaxed. Furthermore, the position where the movement of the pointer P is stopped can be easily controlled, and the operability can be improved.

  In the present embodiment, the control unit 140 detects the operation position when the input information acquisition unit 110 detects contact with the track pad 14. Then, when the input information acquisition unit 110 detects that the operating body has moved away from the track pad 14 and the contact is released, the control unit 140 determines that the termination condition is satisfied, and stops generating input data. . Thereby, since the user can easily instruct the timing to start and stop the movement of the display position of the pointer P, the operability can be further improved.

Further, the control unit 140 sets the movement direction of the display position of the pointer P to a direction corresponding to the movement direction of the operation position on the track pad 14. Therefore, the user can move the pointer P in the intended direction, and the operability can be improved.
Further, since the control unit 140 detects the operation of the slide gesture with respect to the track pad 14 and then performs a process for obtaining the moving direction of the pointer P, the control unit 140 moves the pointer P, so that the pointer P is moved in an appropriate direction. Can do.

  In this embodiment, the control unit 140 detects the pointer P while the input information acquisition unit 110 performs the process of obtaining the moving direction of the display position of the pointer P in step S16 after detecting the operation on the track pad 14 in step S11. It may be moved. Here, the control unit 140 moves the pointer P to a display position corresponding to the operation position of the track pad 14. In this case, the user can perform an operation without a sense of incongruity by appropriately moving the pointer P even during the process of obtaining the moving direction of the pointer P. In addition, the pointer P can be moved in response to an operation for which the movement amount of the operation position is small for a short time. The movement of the pointer P in this case may be realized by control for moving the pointer P to the display position of the displayable area D corresponding to the operation position of the track pad 14. In addition, after the input information acquisition unit 110 detects an operation on the track pad 14, the control unit 140 may perform a process for obtaining the moving direction of the display position of the pointer P and then move the display position of the pointer P. . In this case, the pointer P can be moved in an appropriate direction.

  Further, the control unit 140 may obtain the moving speed of the pointer P according to the operation on the track pad 14 and move the pointer P in the moving direction at the obtained moving speed. For example, the control unit 140 may set the moving speed of the display position of the pointer P to a speed corresponding to the moving amount or moving speed of the operation position on the track pad 14. In this case, since the user can control the moving speed of the pointer P, the operability can be further improved.

  In addition, since the control unit 140 detects the movement of the operation position on the track pad 14 and moves the pointer P until the operation is completed, the track pad 14 is compared with, for example, an area where the image display unit 20 displays an image. Can be small. The track pad 14 only needs to have a size that allows the control unit 140 to detect the direction of movement of the operation position. In other words, the pointer P can be moved larger than the size of the track pad 14. Therefore, even if the track pad 14 is small in the configuration of the HMD 100, the pointer P can be arbitrarily moved, and the operability can be improved. Moreover, there exists an advantage that the freedom degree of a structure of HMD100 increases.

  In the HMD 100, the control unit 140 generates input data for the virtual input plane IP in accordance with an operation detected by the input information acquisition unit 110. When the input information acquisition unit 110 detects an operation of moving the operation position on the track pad 14, the control unit 140 starts generating input data on the virtual input plane IP in response to the operation. The control unit 140 stops the generation of input data when the input information acquisition unit 110 detects an operation corresponding to a preset end condition. As a result, the input data for the virtual input plane IP of the operating system 150 can be generated beyond the limitation of the size of the track pad 14. In the present embodiment, since the display position of the pointer P is moved based on the input data, the movement amount of the display position of the pointer P can be controlled with a high degree of freedom. Therefore, the operability restriction due to the size of the track pad 14 can be relaxed.

  The input data generated by the control unit 140 includes at least one of an input position in the virtual input plane IP, a moving direction of the input position in the virtual input plane IP, and a moving speed of the input position in the virtual input plane IP. It can be. Thereby, an input position on the virtual input plane IP, a moving direction of the input position, or a moving speed of the input position can be input by an operation on the track pad 14.

  In addition, the control unit 140 determines at least one of the movement direction of the input position and the movement speed of the input position included in the input data according to the movement amount or movement speed of the operation position detected by the input information acquisition unit 110. It may be set. In this case, the input position, the moving direction of the input position, or the moving speed of the input position can be controlled by operating the track pad 14.

  In addition, when the input information acquisition unit 110 detects that the contact with the track pad 14 has been released, the control unit 140 may determine that the end condition is satisfied and stop generating input data. According to this configuration, the user can easily instruct the timing for starting and stopping the generation of the input data, so that the operability can be further improved.

  As shown in FIG. 6, in the HMD 100, the virtual input plane IP and the displayable area D (display area) in the display unit 20 are associated with each other. The control unit 140 can control the display position of the pointer P that is a display object to be operated displayed on the image display unit 20 in accordance with the input data. Thereby, the display position of the pointer P can be moved according to the operation on the track pad 14, and the movement amount of the display position can be controlled with a high degree of freedom.

[Second Embodiment]
FIG. 9 is a flowchart showing the operation of the HMD 100 according to the second embodiment to which the present invention is applied.
In the second embodiment, the configuration of the HMD 100 is the same as that of the first embodiment. For this reason, illustration and description regarding the configuration of the HMD 100 are omitted.

Steps S11 to S19 of the operation shown in FIG. 9 are the same as those in the first embodiment.
In the second embodiment, the control unit 140 moves the display position of the pointer P when the operation position on the track pad 14 is changed after the generation of input data and the movement of the display position of the pointer P are started in step S17. Change the parameter.

  As in the first embodiment, the control unit 140 monitors the release of contact with the track pad 14 after starting the generation of input data (step S18). Here, while the contact with the track pad 14 is not released (step S18; No), the control unit 140 continues to generate the input data, and thereby the display position of the pointer P continues to move. During this time, the operation position on the track pad 14 is in a state that can be regarded as a stop after it is determined that the operation is stopped in step S15.

  When the contact with the track pad 14 is not released (step S18; No), the control unit 140 determines whether or not the operation position of the track pad 14 has moved in the reverse direction (step S21). The “reverse direction” refers to the direction of movement of the operation position of the track pad 14 detected in step S13 or the direction opposite to the movement parameter of the input position set in step S16. The “reverse direction” is not limited to the case where the components in the X-axis direction and the Y-axis direction are completely opposite directions. For example, a direction in which components in the X-axis direction and the Y-axis direction are within a predetermined range with respect to completely opposite directions can be included in the “reverse direction”. That is, in step S21, it may be determined that the operation position on the track pad 14 has moved in the reverse direction when it has moved within a range that can be regarded as the reverse direction. In addition, an operation having a direction difference of a predetermined angle or more with respect to the movement direction of the operation position of the track pad 14 detected in step S13 or the movement direction indicated by the movement parameter of the input position set in step S16 is performed in the reverse direction. The movement may be determined. In step S21, determination based on the movement amount of the operation position of the track pad 14 may be performed. For example, when the movement amount of the operation position is smaller than a preset threshold value, it may be configured not to determine the movement in the reverse direction.

  When it is determined that the operation position of the track pad 14 has moved in the reverse direction (step S21; Yes), the control unit 140 updates (changes) the movement parameter related to the movement speed of the input position (step S22). After step S22, the control unit 140 generates input data based on the updated movement parameter, and the display position of the pointer P moves. After step S22, the control unit 140 returns to step S18. When the operation position of the track pad 14 has not moved in the reverse direction (step S21; No), the control unit 140 returns to step S18.

  In step S22, for example, the moving speed of the input position can be changed in stages, and the moving speed can be reduced by one stage. In this case, each time the control unit 140 detects an operation for moving the operation position in the reverse direction, the moving speed of the input position is decreased by one step. Corresponding to the change in the moving speed of the input position in the input data, the moving speed of the pointer P changes stepwise. Thereby, the user can easily control the moving speed of the pointer P.

  Thus, the control unit 140 changes the moving speed of the pointer P according to the change in the operation position of the track pad 14 while the pointer P is moving, so that the user can change the moving speed of the pointer P more. Fine control.

  In the second embodiment, when the control unit 140 determines that the operation position by the operating body has moved in the reverse direction, the movement speed of the input position in the input data is reduced, and the pointer P that is a display object is moved. An example of changing the speed to a low speed has been described. In this case, after changing the moving speed of the input position, the moving speed of the input position may be restored when the operation position of the track pad 14 moves in the positive direction. In addition, without limiting the number of times, the movement speed of the input position is increased each time a forward movement of the operation position is detected, and the movement speed is decelerated to the input position every time a reverse movement of the operation position is detected. Also good. Further, in the configuration in which the input information acquisition unit 110 and the control unit 140 can detect the area of the area where the operating body contacts in the track pad 14, the moving speed of the input position may be changed by changing the contact area. For example, when the contact area changes small, the moving speed of the input position may be reduced. Further, when the contact area increases, the moving speed of the input position may be increased. In these cases, when the movement parameter is updated so as to change the movement speed of the input position, the movement speed of the pointer P changes. Therefore, the movement speed of the pointer P can be easily controlled by the user's operation. In addition, the specific mode of changing the speed of moving the input position according to the mode of operation on the track pad 14 can be arbitrarily changed.

Furthermore, as a modification of the second embodiment, the moving speed of the input position in the input data may be changed corresponding to an operation of moving the control device 10.
As shown in FIG. 2, the control device 10 includes a triaxial sensor 113 that is a triaxial acceleration sensor.
After starting the generation of input data in step S16, the operation detection unit 183 acquires the detection value of the triaxial sensor 113, for example, at a preset sampling cycle, and detects a change in the attitude of the control device 10. The HMD 100 is set to increase the moving speed of the pointer P when the control device 10 is moved in the first direction and to decrease the moving speed of the pointer P when the control apparatus 10 is moved in the second direction. Specifically, a detection value or a detection value range of the three-axis sensor 113 corresponding to the movement in the first direction is set in advance. The same applies to the second direction. The first direction and the second direction only need to be two directions that the operation detection unit 183 can distinguish based on the detection value of the three-axis sensor 113, and may be opposite directions. The first and second directions may be directions in which the control device 10 is moved linearly, or directions in which the control device 10 is rotated. The rotation direction of the control device 10 can be arbitrarily set, and may be rotation about the detection axis of the three-axis sensor 113 or rotation about an axis that is not parallel to the detection axis.

  Here, the operation detection unit 183 may update the movement parameter according to the detection value of the three-axis sensor 113 only when the operation position is stopped. That is, the operation detection unit 183 determines that the operation position is stopped (step S15; Yes) and determines that the contact with the track pad 14 is released (step S18; Yes). You may perform only. Further, the operation detection unit 183 may be configured to execute the above process while generating input data.

When the operation detection unit 183 determines that the operation in the first direction or the operation in the second direction has been performed based on the detection value of the three-axis sensor 113, the operation detection unit 183 performs the same processing as step S22, and moves the input position. Update movement parameters for.
In this configuration, the moving speed of the pointer P can be accelerated or decelerated by moving the control device 10 while the user performs an operation of bringing the operating body into contact with the track pad 14 and moving the pointer P.

  As yet another modification of the second embodiment, the moving speed of the input position in the input data may be changed corresponding to the contact area on the track pad 14. The input information acquisition unit 110 can detect the contact position when the operating tool comes into contact with the track pad 14 and can also detect the contact area.

  In this case, the input information acquisition unit 110 can be configured to detect a plurality of contact positions on the track pad 14. The operation detection unit 183 calculates an area in which the operation body has contacted, assuming that one operation body has contacted the plurality of contact positions detected by the input information acquisition unit 110. That is, the operation detection unit 183 calculates the contact area based on the operation data output from the input information acquisition unit 110. Further, the input information acquisition unit 110 may be configured to detect the contact area on the track pad 14. In this case, the input information acquisition unit 110 outputs data indicating the contact area on the track pad 14 in the operation data.

  After starting the generation of input data in step S <b> 16, the operation detection unit 183 acquires the contact area on the track pad 14 based on the operation data output from the input information acquisition unit 110. Here, the operation detection unit 183 changes the moving speed of the input position in the input data based on the acquired contact area. For example, the moving speed of the input position may be set in association with the size of the contact area acquired by the operation detection unit 183. In this case, the operation detection unit 183 updates the movement parameter of the input data related to the movement speed of the input position to a value corresponding to the acquired contact area. Further, for example, the operation detection unit 183 may update the movement parameter in accordance with the acquired change in the contact area. Specifically, the operation detection unit 183 calculates a change in the contact area on the track pad 14, and when the contact area increases beyond a preset threshold value, the movement speed of the input position by a preset speed. Update the movement parameters to increase In addition, the operation detection unit 183 updates the movement parameter so that the movement speed of the input position is decreased by a preset speed when the contact area on the track pad 14 is reduced beyond a preset threshold. .

  Here, the operation detection unit 183 may execute the process of updating the movement parameter in accordance with the contact area on the track pad 14 only while the operation position is stopped. That is, the operation detection unit 183 determines that the operation position is stopped (step S15; Yes) and determines that the contact with the track pad 14 is released (step S18; Yes). You may perform only. Further, the operation detection unit 183 may be configured to execute the above process while generating input data.

  In this configuration, while the user performs an operation of bringing the operating body into contact with the track pad 14 and moving the pointer P, the moving speed of the pointer P is adjusted by adjusting the area where the operating body contacts the track pad 14. Can accelerate or decelerate. For example, when an operation body having elasticity such as a user's finger or hand is used, the operation detection unit 183 adjusts the contact area by adjusting the contact area by adjusting the contact area by adjusting the contact area. The moving speed can be accelerated or decelerated.

[Third Embodiment]
FIG. 10 is an explanatory diagram illustrating an example of an operation of the HMD 100 according to the third embodiment. FIG. 11 is a flowchart showing the operation of the HMD 100 according to the third embodiment to which the present invention is applied. In the third embodiment, the configuration of the HMD 100 is the same as that of the first embodiment. For this reason, illustration and description regarding the configuration of the HMD 100 are omitted.
In the third embodiment, an example will be described in which the control unit 140 changes the moving speed and moving direction of the pointer P when the operation position of the track pad 14 moves after the movement of the pointer P is started. In the second embodiment, the operation when the operation position of the track pad 14 is operated in the reverse direction has been described. However, in the third embodiment, the operation position is not limited to the reverse direction, and the operation position is moved in another direction. Can handle the case.

  FIG. 10 shows the movement of the operation position on the track pad 14. Reference signs A11 to A15 indicate contact positions at which an operating body such as a user's hand H (FIG. 3) contacts the track pad 14, that is, an operation position. Reference symbols AT1 to AT4 indicate movement trajectories of the operation positions, that is, movement operations. Hereinafter, the operation of the HMD 100 will be described with reference to the flowcharts of FIGS. 10 and 11.

  The operations in steps S11 to S16 are the same as those in the first and second embodiments. For example, when the input information acquisition unit 110 detects the operation AT1 from the operation position A11 to the operation position A12 in FIG. 10, the control unit 140 calculates a movement parameter corresponding to the operation AT1 in step S16. The control unit 140 starts generating input data according to the movement parameter set in step S16, and starts a process of displaying the pointer P on the right display driving unit 22 and the left display driving unit 24 based on the generated input data (step). S31).

  After step S31, the input position is moved in the input data generated by the control unit 140, and the display position of the pointer P displayed by the image display unit 20 is moved. While the display position of the pointer P moves, the control unit 140 monitors the release of contact with the track pad 14 (step S32). When the control unit 140 detects that the contact with the track pad 14 is released (step S32; Yes), the control unit 140 stops generating the input data, thereby stopping the movement of the pointer P (step S19). End the process.

  On the other hand, when it is determined that the contact with the track pad 14 is not released during the movement of the pointer P (step S32; No), the control unit 140 determines whether or not the operation position on the track pad 14 has moved. (Step S33). In step S33, it is determined whether or not the operation position detected by the input information acquisition unit 110 has moved a predetermined distance or more from the position determined to be stopped in step S15. When it determines with the operation position not moving (step S33; No), the control part 140 returns to step S32.

When it determines with the operation position having moved (step S33; Yes), the control part 140 waits until the operation position stops (step S34). During this standby, the control unit 140 continues to detect the operation position based on the operation data input from the input information acquisition unit 110. When the operation position is stopped (step S34; Yes), the control unit 140 calculates the movement parameter of the input data corresponding to the movement of the operation position detected in step S33, and updates the movement parameter (step S35).
The control unit 140 returns to step S31 and starts generating input data using the updated movement parameter.

For example, when the operation AT1 shown in FIG. 10 is performed and the operation body is stopped at the operation position A12 and then the operation AT2 is performed in which the operation body moves to the operation position A13, the control unit 140 performs the operation in step S35. A movement parameter corresponding to AT2 is generated. In this case, the control unit 140 sets the movement direction and / or movement speed of the input position based on at least one of the movement direction, movement speed, and movement amount from the operation position A12 to the operation position A13. The amount of movement of the input position is determined by the time until the operation AT2 stops. Here, the control unit 140 may calculate the movement amount of the input position based on the movement speed or movement amount of the operation AT2.
Thus, when the operation position stops at the first stop position and then moves to the second stop position and stops, the operation detection unit 183 moves from the first stop position to the second stop position. Input data can be generated based on at least one of the moving direction, moving speed, and moving amount. For example, the first stop position corresponds to the operation position A11, and the second stop position corresponds to the operation position A12, A13, or A14. Further, when the operation position moves to another position after stopping at the operation positions A12, A13, A14, the operation positions A12, A13, A14 may be operated as the first stop position.

According to the third embodiment, when the operation AT3 that moves to the operation position A14 or the operation AT4 that moves to the operation position A15 is performed after the operating body stops at the operation position A12, the operations AT3 and AT4 are performed. Input data whose input position moves so as to correspond to is generated. By displaying a display object such as the pointer P based on the input data, the HMD 100 can move the display position of the pointer P at a direction and speed reflecting the operations AT2, AT3, and AT4.
Furthermore, since the processes of steps S33 to S35 can be repeatedly executed, for example, when the operating body moves from the operation position A13 to another position after the operation AT2, the pointer reflects this movement. The display position of P can be changed. In this case, it is possible to adopt a configuration in which the operation position is stopped and the number of movements thereafter is not limited.

Thus, the HMD 100 follows the change in the direction of movement of the operating body when the user performs an operation to change the direction in which the operating body moves while the operating body is in contact with the track pad 14. In addition, the pointer P can be moved.
In addition, in steps S15 and S34, if the time for determining that the operation position detected by the input information acquisition unit 110 is stopped is set short, the time for the user to intentionally stop the operating tool may be short. . The time determined in steps S15 and S34 may be at least as long as the input information acquisition unit 110 detects the operation position a plurality of times. For example, the time may be shortened to a time corresponding to two sampling cycles.

[Fourth Embodiment]
FIG. 12 is a flowchart showing the operation of the HMD 100 according to the fourth embodiment to which the present invention is applied. In the fourth embodiment, the configuration of the HMD 100 is the same as that of the first embodiment. For this reason, illustration and description regarding the configuration of the HMD 100 are omitted.

  In the fourth embodiment, when the contact operation on the track pad 14 is released in a short time, the HMD 100 sets the movement amount of the display position of the pointer P to a specified amount, and thereby the pointer P can be moved in a short operation. Enable to move the display position.

In FIG. 12, the operations in steps S12 to S14 are the same as those in the first to third embodiments described above.
When it is determined that the operation on the track pad 14 is a slide gesture (step S13; Yes), the control unit 140 starts a timer count (step S41). The timer can be realized as one function of the control unit 140 realized by a program. The controller 140 waits until the operation position stops (step S15). During this standby, the control unit 140 continues to detect the operation position based on the operation data input from the input information acquisition unit 110.

While the operation position does not stop, that is, while the slide gesture continues (step S15; No), the control unit 140 monitors the release of contact with the track pad 14 (step S42). When it is determined that the contact with the track pad 14 is not released (step S42; No), the control unit 140 returns to step S15 and monitors the operation position.
Here, when the control unit 140 detects that the contact operation of the track pad 14 is released (step S42; Yes), is the count value of the count started in step S41 equal to or larger than a preset setting value? It is determined whether or not (step S43). That is, it is determined whether the count value of the time from the start of the slide gesture until the contact with the track pad 14 is released is greater than or equal to the set value. The setting value is included in the setting data 121 and stored in the storage unit 120, for example.

  When the count value (timer counter value) is equal to or larger than the set value (step S43; Yes), the control unit 140 calculates a movement parameter of input data corresponding to the slide gesture (step S44). The movement parameter calculated in step S44 is, for example, data indicating the start position of the input position, the moving direction of the input position, and the moving speed of the input position, similarly to the movement parameter calculated in step S16 described later. Etc. may be included. It is desirable that the movement parameter calculated and generated in step S44 includes data indicating the movement amount. The data indicating the movement amount may be a preset default value of the movement amount, may be a value calculated based on the count value determined in step S43, or may be a movement of the operation position of the slide gesture. It may be a movement amount corresponding to the amount or movement speed. After calculating the movement parameter, the control unit 140 proceeds to step S48 described later. The control unit 140 may perform a process of stopping the count started in step S41 and resetting the count value before or after the process of step S44.

  When the count value is smaller than the set value (step S43; No), the control unit 140 calculates a movement parameter of input data based on a reference value (not shown) stored in advance in the storage unit 120 (step S45). . The movement parameter generated in step S45 may include, for example, data indicating the movement start position of the input position, the movement direction of the input position, and the movement speed of the input position, and includes data indicating the movement amount. At least one of the input position movement start position and the input position movement direction may be calculated based on the operation position of the slide gesture detected by the input information acquisition unit 110 or the movement direction of the operation position. The moving speed and moving amount of the input position may be set based on a reference value stored in advance in the storage unit 120. After setting the movement parameter, the control unit 140 proceeds to step S48 described later. The control unit 140 may perform a process of stopping the count started in step S41 and resetting the count value before or after the process of step S45.

  When the operation position is stopped (step S15; Yes), the control unit 140 calculates a movement parameter of the input data in order to generate input data corresponding to the slide gesture (step S16). As described above, the input data movement parameter generated in step S16 includes, for example, data indicating the input position movement start position, the input position movement direction, and the input position movement speed. The moving direction of the input position is set to the same direction as the moving direction of the slide gesture detected in steps S11 to S15, for example. The moving speed of the input position is set to, for example, a speed corresponding to the moving amount (operation amount) of the slide gesture or a default value.

  The control unit 140 starts generating input data according to the movement parameter set in step S16, and starts a process of displaying the pointer P on the right display driving unit 22 and the left display driving unit 24 based on the generated input data (step). S17). After step S17, the input position moves in the input data generated by the control unit 140, and the display position of the pointer P displayed by the image display unit 20 moves. Thereafter, the control unit 140 continues the process of generating the input data at a predetermined cycle (for example, the sampling cycle of the input information acquisition unit 110). Further, the control unit 140 continues the process of moving the display position of the pointer P based on the generated input data.

While the display position of the pointer P moves, the control unit 140 monitors the release of contact with the track pad 14 (step S18). If it is determined that the contact with the track pad 14 is not released (step S18; No), the generation of input data and the movement of the display position of the pointer P are continued.
When the control unit 140 detects that the contact with the track pad 14 is released (step S18; Yes), it is determined whether or not the count value of the count started in step S41 is greater than or equal to a preset set value. Determination is made (step S46). This setting value may be included in the setting data 121 and stored in the storage unit 120, for example, or may be a value common to the setting value in step S43.

  When the count value (timer counter value) is equal to or greater than the set value (step S46; Yes), the control unit 140 stops generating the input data, thereby stopping the movement of the pointer P (step S19). This process ends. The control unit 140 may perform a process of stopping the count started in step S41 and resetting the count value before or after the process of step S19.

  When the count value is smaller than the set value (step S46; No), the control unit 140 calculates a movement parameter of input data based on a reference value (not shown) stored in advance in the storage unit 120 (step S47). . In step S47, similarly to step S45, the control unit 140 sets a movement parameter based on a reference value stored in advance in the storage unit 120, and updates the movement parameter calculated in step S16. The control unit 140 may perform a process of stopping the count started in step S41 and resetting the count value before or after the process of step S47.

  The control unit 140 executes processing for generating input data according to the movement parameter, and thereby the display position of the pointer P is moved (step S48). The control unit 140 executes the process of generating the input data by an amount corresponding to the movement amount included in the movement parameter, stops the generation of the input data, and ends this process.

  According to the fourth embodiment, it is possible to move the pointer P by a preset amount of movement when the operating body is in contact with the track pad 14 for a short time. For this reason, the user can move the pointer P by performing an operation of bringing the operating body into contact with the track pad 14 for a short time. Therefore, the display of the image display unit 20 can be controlled by the operation of touching the track pad 14 for a short time in addition to the operation of maintaining the state in which the operating body is in contact with the track pad 14, and the operability can be improved.

[Fifth Embodiment]
FIG. 13 is a flowchart showing the operation of the HMD 100 according to the fifth embodiment to which the present invention is applied. FIG. 14 is a flowchart showing the operation of the HMD 100 according to the fifth embodiment to which the present invention is applied. In the fifth embodiment, the configuration of the HMD 100 is the same as that of the first embodiment. For this reason, illustration and description regarding the configuration of the HMD 100 are omitted.

  In the fifth embodiment, the HMD 100 moves the display position of the pointer P in accordance with the contact operation on the track pad 14 when the contact operation is performed again within a predetermined time after the contact operation is released. These contact operations are processed as continuous operations.

  The operations in steps S11 to S18 shown in FIG. 13 are the same as those in the first to third embodiments described above. In the fifth embodiment, the control unit 140 monitors the release of contact with the track pad 14 while the operation position of the track pad 14 is not stopped, that is, while the slide gesture continues (step S15; No). S61). If it is determined that the contact with the track pad 14 is not released (step S61; No), the control unit 140 returns to step S15 and monitors the operation position.

  When the control unit 140 detects that the contact operation on the track pad 14 has been released (step S61; Yes), the control unit 140 calculates the movement parameter of the input data corresponding to the slide gesture (step S62). The movement parameter calculated in step S62 is, for example, data indicating the start position of the movement of the input position, the movement direction of the input position, and the movement speed of the input position, similarly to the movement parameter calculated in step S16 described later. Etc. may be included. It is desirable that the movement parameter calculated and generated in step S62 includes data indicating the movement amount. The data indicating the movement amount may be a preset default value of the movement amount, or may be a movement amount corresponding to the movement amount or movement speed of the operation position of the slide gesture.

  After calculating the movement parameter, the control unit 140 executes a process of generating input data according to the movement parameter, thereby moving the display position of the pointer P (step S63). The control unit 140 executes processing for generating input data by an amount corresponding to the movement amount included in the movement parameter.

  After starting the generation of input data, the control unit 140 starts a timer count (step S64). The timer can be realized as one function of the control unit 140 realized by a program. The control unit 140 monitors a new contact with the track pad 14 (step S65), and determines whether or not the count value of the timer is equal to or larger than the set value while the contact with the track pad 14 is not detected (step S65; No). Determination is made (step S66). Here, when the count value is equal to or larger than the preset setting value (step S66; Yes), the control unit 140 generates the input data corresponding to the movement amount of the movement parameter, and then performs this processing. Exit.

When the count value of the timer does not reach the set value (step S66; No), the control unit 140 returns to step S65 and monitors the contact with the track pad 14.
When contact with the track pad 14 is detected (step S65; Yes), the control unit 140 proceeds to step S71 (FIG. 13) described later.

  Further, when the control unit 140 detects that the contact with the track pad 14 is released in step S18 (step S18; Yes), the control unit 140 starts counting of the timer (step S51). The control unit 140 monitors a new contact with the track pad 14 (step S52), and determines whether or not the count value of the timer is equal to or greater than the set value while no contact with the track pad 14 is detected (step S52; No). Determination is made (step S53). Here, when the count value is equal to or greater than the preset set value (step S53; Yes), the control unit 140 stops generating the movement parameter, and thereby stops the movement of the pointer P (step S19). ), This process is terminated.

  If the count value of the timer does not reach the set value (step S53; No), the control unit 140 returns to step S52 and monitors contact with the track pad 14. When contact with the track pad 14 is detected (step S52; Yes), the control unit 140 proceeds to step S71 (FIG. 13).

Steps S71 and after are operations corresponding to the case where the user performs an operation of bringing the operating body into contact with the track pad 14 and again brings the operating body into contact with the track pad 14 within a predetermined time after releasing the contact. .
The control unit 140 monitors whether or not the contact position of the track pad 14 detected in step S52 or S65, that is, the operation position is stopped (step S71). While the operation position does not stop (step S71; No), the control unit 140 monitors the release of contact with the track pad 14 (step S72). When it is determined that the contact with the track pad 14 has not been released (step S72; No), the control unit 140 returns to step S71 and monitors the operation position.

  When it is detected that the operation position is stopped (step S71; Yes), the control unit 140 calculates the movement parameter of the input data corresponding to the operation and updates the movement parameter (step S72). The movement parameter calculated in step S72 may include, for example, data indicating the start position of the input position, the movement direction of the input position, and the movement speed of the input position. After updating the movement parameter in step S72, the control unit 140 starts a process of generating input data based on the updated movement parameter.

  The controller 140 monitors the release of contact with the track pad 14 during the process of generating input data (step S75). When it is determined that the contact with the track pad 14 is not released (step S75; No), the generation of input data is continued, and the monitoring in step S75 is continued. When detecting that the contact with the track pad 14 has been released (step S75; Yes), the control unit 140 starts counting the timer (step S74).

The control unit 140 monitors a new contact with the track pad 14 (step S75), and determines whether the count value of the timer is equal to or greater than the set value while the contact with the track pad 14 is not detected (step S75; No). Determination is made (step S76). Here, when the count value is equal to or larger than the preset set value (step S76; Yes), the control unit 140 proceeds to step S54 and stops generating the movement parameter, and thereby moves the pointer P. It stops (step S54), and this process is complete | finished. The control unit 140 may perform a process of stopping the count started in step S74 and resetting the count value after the process of step S76.
When the count value of the timer does not reach the set value (step S76; No), the control unit 140 returns to step S75 and monitors the contact with the track pad 14.

  In Step S71, when the operation position is not stopped, that is, while the operation position continues to move (Step S71; No), the control unit 140 monitors the release of the contact with the track pad 14 (Step S71). S77). When it is determined that the contact with the track pad 14 is not released (step S77; No), the generation of input data is continued, and the monitoring in step S77 is continued.

  When the control unit 140 detects that the contact with the track pad 14 is released (step S77; Yes), the control unit 140 calculates the movement parameter of the input data corresponding to the slide gesture (step S78). The movement parameter calculated in step S78 is the same as the movement parameter calculated in step S62, for example. That is, it may include data indicating the start position of the input position, the moving direction of the input position, and the moving speed of the input position, and preferably includes data indicating the amount of movement. The data indicating the movement amount may be a preset default value of the movement amount, or may be a movement amount corresponding to the movement amount or movement speed of the operation position of the slide gesture. In step S78, the control unit 140 updates the movement parameter with the calculated movement parameter.

  The control unit 140 starts processing for generating input data based on the updated movement parameter, and thereby the display position of the pointer P moves (step S79). The control unit 140 starts counting the timer after starting the generation of input data (step S80). In step S80, the control unit 140 counts an elapsed time after detecting the release of contact in step S77. Therefore, the process of step S80 may be executed before step S78 or S79.

The control unit 140 monitors a new contact with the track pad 14 (step S81), and determines whether or not the count value of the timer is equal to or greater than the set value while the contact with the track pad 14 is not detected (step S81; No). Determination is made (step S82). Here, when the count value is equal to or larger than the preset setting value (step S82; Yes), the control unit 140 generates the input data corresponding to the movement amount of the movement parameter, and then performs this processing. Exit. When the count value of the timer does not reach the set value (step S82; No), the control unit 140 returns to step S81 and monitors contact with the track pad 14.
When contact with the track pad 14 is detected (step S81; Yes), the control unit 140 returns to step S71 (FIG. 13). When contact with the track pad 14 is detected (step S81; Yes), the control unit 140 may perform a process of stopping the count started in step S80 and resetting the count value.

In the fifth embodiment, the control unit 140 corresponds to an end condition that the operation is considered to have ended when a predetermined time has elapsed since it was detected that the operating body has moved away from the track pad 14 and contact has been released. Then, it is determined that the operation when the operation is released is executed. For example, the control unit 140 stops generating input data.
Therefore, the movement of the display position of the pointer P can be continued even after the operating body is separated from the track pad 14. Therefore, the display position of the pointer P can be moved larger than the amount by which the operation position moves on the track pad 14, and the restriction on the size of the track pad 14 can be avoided.

  In addition, when the control unit 140 detects contact with the track pad 14 within a predetermined time after the operation detection unit detects that the operating body has moved away from the track pad 14 and released the contact, a new contact is made. The movement parameter is updated according to the operation position. By updating the movement parameter, at least one of the movement direction and the movement speed of the display position of the pointer P changes. Accordingly, even after the operating tool is separated from the track pad 14, the user can control the movement of the display position of the pointer P by performing an operation of bringing the operating tool into contact with the track pad 14 again. For this reason, the mode of movement of the display position of the pointer P can be controlled with a high degree of freedom. For example, the control unit 140 performs a series of operations that are performed across the state in which the operation body is separated from the track pad 14 when the operation body is again in contact with the track pad 14 within a predetermined time. It can be detected as an operation.

  Furthermore, the operation in which the control unit 140 monitors the contact until the count value of the timer reaches the set value after the contact with the track pad 14 is released can be repeatedly executed.

FIG. 15 is an explanatory diagram illustrating an example of an operation on the track pad 14 according to the fifth embodiment. The example of FIG. 15 shows an example in which two operations AT11 and AT12 are performed on the track pad 14. Specifically, after the operation AT11 for moving the operating body from the operating position A21 to the operating position A22 is executed, the operating body leaves the track pad 14, and then the operating body contacts the track pad 14 at the operating position A23. This is an example of moving to the operation position A24.
Here, it is assumed that the time until the operating body leaves the track pad 14 at the operation position A22 and then contacts the track pad 14 at the operation position A23 is shorter than a predetermined time. The predetermined time corresponds to a set value (steps S53, S66, S76, S88) set for the count value of the timer counted by the control unit 140.

In this case, the control unit 140 generates a movement parameter based on the operation AT11 and executes processing for generating input data. For this reason, although not shown, the display position of the pointer P moves in a moving direction or a moving speed reflecting the operation AT11.
When detecting the operation AT12, the control unit 140 generates and updates a movement parameter corresponding to the operation AT12. Here, the control unit 140 determines the position of the operation position A22 that is the start position of the operation AT12, the position of the operation position A23 that is the end position, the movement direction of the operation position of the operation AT12, the movement speed of the operation position, the contact time, and the like. Corresponding movement parameters can be updated.

  Further, the control unit 140 may generate and update the movement parameter based on the result of combining the operation AT11 and the operation AT12. Specifically, the control unit 140 generates a vector indicating the movement direction and the movement amount of the operation position for the operation AT11. When detecting the operation AT12, the control unit 140 generates a vector indicating the movement direction and the movement amount of the operation position for the operation AT12. The control unit 140 combines the vector of the operation AT11 and the vector of the operation AT12 to generate a combined vector. The control unit 140 may generate a movement parameter corresponding to the composite vector. In this case, the operation AT11 and the operation AT12 are regarded as one operation, and input data can be generated corresponding to this operation. In this case, in the process of moving the display position of the pointer P based on the input data, the moving direction, moving speed, etc. of the pointer P can be gently changed according to the user's operation.

  Furthermore, the control unit 140 may generate input data corresponding to the operation AT12 after generating input data corresponding to the operation AT11. That is, a movement parameter corresponding to the operation AT11 may be generated, input data may be generated according to the movement parameter, and then the movement parameter may be updated corresponding to the operation AT12. That is, the control unit 140 can generate input data so that a plurality of operations are accumulated and reflected when a plurality of operations are performed across a non-operation period within a predetermined time. In this case, the display position of the pointer P can be moved by reflecting the operations AT11 and AT12 in order.

  Further, even after the operating tool leaves the track pad 14 at the operation position A22, the control unit 140 continues to generate input data, so the display position of the pointer P continues to move. For this reason, the pointer P can be moved even after the user moves the operating tool away from the track pad 14. Furthermore, if an operation of contacting the track pad 14 is performed within a predetermined time, the display position can be moved without stopping the pointer P.

  Further, when performing the various operations described above, the generation of input data may be stopped by an operation in a state where the operating body is not in contact with the track pad 14 and the input data is generated. The input data is generated and the pointer P moves within a predetermined time after the control unit 140 detects that the contact with the track pad 14 is released. For example, when an operation of tapping (hitting) the track pad 14 is performed while the pointer P is moving, the control unit 140 may stop or interrupt the generation of input data regardless of the elapsed time. Further, for example, when the operation of moving the control device 10 is detected by the three-axis sensor 113 while the pointer P is moving, the control unit 140 may stop or interrupt the generation of input data regardless of the elapsed time. . For example, when the control unit 140 detects an operation of tapping the image display unit 20 or an operation of moving the image display unit 20 in a predetermined direction by the 9-axis sensor 66, the control unit 140 stops generating input data regardless of the elapsed time. Or you may interrupt.

  In the fifth embodiment, when an operation detected within a predetermined time after the operating body leaves the track pad 14 satisfies a predetermined condition with respect to the previous operation, the control unit 140 is set in advance. It is good also as a structure which performs the process which carried out. The predetermined condition is, for example, a case where the moving direction of the operation position in the operation detected within a predetermined time after the operating body leaves the track pad 14 is opposite to the previous operation. The process executed by the control unit 140 is, for example, a process of reducing the moving speed of the input position in the moving parameter. The predetermined condition may be detecting an operation of moving the control device 10 by the three-axis sensor 113. Moreover, you may set a specific direction about the motion of the control apparatus 10 detected in this case. In this case, the control unit 140 may execute a process of accelerating or decelerating the moving speed of the input position in the moving parameter, for example, according to the movement of the control device 10 or the direction of the movement. Further, for example, a process of changing the movement direction of the input position in the movement parameter in accordance with the movement direction of the control device 10 may be executed.

[Sixth Embodiment]
FIG. 16 is a diagram showing a setting state of the operation area on the track pad 14 in the sixth embodiment to which the present invention is applied, and FIG. 17 is a flowchart showing the operation of the HMD 100. In the sixth embodiment, the configuration of the HMD 100 is the same as that of the first embodiment. For this reason, illustration and description regarding the configuration of the HMD 100 are omitted.

  As shown in FIG. 16, in the sixth embodiment, a plurality of operation areas C are set on the track pad 14. The operation area C is designated by coordinates on the track pad 14, and the coordinates of the boundary between each operation area C and another area are set in the setting data 121, for example.

  As described in the first and second embodiments, the operation area C1 is an area to be controlled so that the pointer P is moved even when the operation position of the track pad 14 does not move. On the other hand, the area other than the operation area C1, that is, the operation area C2, is an area that is controlled so that the display position of the pointer P in the displayable area D corresponds to the operation position of the track pad 14.

  The control unit 140 executes the process described in the first embodiment when a slide gesture is performed in the operation area C1. Further, in the operation area C2, even when a slide gesture is performed, a normal process for moving the pointer P to a position corresponding to the operation position of the track pad 14 is executed. This operation is shown in FIG.

Steps S11 to S19 of the operation shown in FIG. 17 are the same as those in the first embodiment.
In the sixth embodiment, when it is determined that the operation on the track pad 14 is a slide gesture (step S13; Yes), the control unit 140 of the HMD 100 determines the coordinates of the operation position (step S91). In step S31, for example, the coordinates of the operation position of the track pad 14 are compared with the coordinates set in the setting data 121 to determine whether or not the operation position is included in the operation area C1.

The control unit 140 determines whether or not to perform normal processing based on the determination result of step S31 (step S32). When the normal process is performed (step S92; No), the process proceeds to step S14, and the pointer P is moved to the display position of the displayable area D corresponding to the operation position on the track pad 14 (step S14). Exit.
Moreover, when not performing a normal process (step S92; No), the control part 140 transfers to step S15.

  According to the operation of the sixth embodiment, the process of moving the pointer P larger than the movement amount of the actual operation position of the track pad 14 can be limited to a specific area in the track pad 14. In the track pad 14, the area to be processed is not limited to the shape and size as in the operation area C1 illustrated in FIG. For example, the track pad 14 may be divided into a plurality of parts in the vertical and horizontal directions, and one of the areas may be the same area as the operation area C1. This is useful in that unnecessary movement of the pointer P can be avoided when the shape of the track pad 14 is long in the X and Y directions, or when the track pad 14 having a relatively large area can be mounted.

[Seventh Embodiment]
FIG. 18 is a flowchart showing the operation of the HMD 100 in the seventh embodiment to which the present invention is applied. In the seventh embodiment, the configuration of the HMD 100 is the same as that of the first embodiment. For this reason, illustration and description regarding the configuration of the HMD 100 are omitted.

Steps S11 to S19 of the operation shown in FIG. 18 are the same as those in the first embodiment.
In the seventh embodiment, the control unit 140 of the HMD 100 executes the process described in the first embodiment for an operation in which the operation position moves in a specific direction among the slide gestures on the track pad 14. . Even if a slide gesture is performed, if the movement direction of the operation position is not the set direction, a normal process of moving the pointer P to a position corresponding to the operation position of the track pad 14 is executed.

  That is, when the control unit 140 of the HMD 100 determines that the operation on the track pad 14 is a slide gesture (step S13; Yes), the control unit 140 determines the movement direction of the operation position (step S101).

The controller 140 determines whether or not to perform normal processing based on the movement direction determined in step S101 (step S102). When the normal process is performed (step S102; No), the process proceeds to step S14, and the pointer P is moved to the display position of the displayable area D corresponding to the operation position on the track pad 14 (step S14). Exit.
Moreover, when not performing a normal process (step S102; No), the control part 140 transfers to step S15.

  According to the operation of the seventh embodiment, the process of moving the pointer P larger than the movement amount of the actual operation position of the track pad 14 can be limited to the operation in a specific direction on the track pad 14. For example, a slide gesture in the Y direction may be the target of the above process, and a normal process may be performed on the slide gesture in the X direction. In this case, it is useful when the track pad 14 has a horizontally long shape that is long in the X direction. That is, the above-described processing can be performed only in the direction in which it is difficult to secure the movement amount of the operation position, such as when the shape of the track pad 14 is long in the X direction or the Y direction.

  In step S42, it is determined whether or not the direction determined in step S41 corresponds to the set direction. However, in this determination, the direction is not limited to the case where the direction is accurately moved, and is regarded as a specific direction. You may set the range that can. That is, a configuration may be adopted in which a negative range is determined in step S42 when a range of directions in which the above processing is performed is set in advance and the moving direction of the operation position is included in the above range.

[Eighth Embodiment]
FIG. 19 is a flowchart showing the operation of the HMD 100 in the eighth embodiment. FIG. 20 is a schematic diagram showing the correspondence between the track pad 14 and the virtual input plane IP in the eighth embodiment. In the eighth embodiment, the configuration of the HMD 100 is the same as that of the first embodiment. For this reason, illustration and description regarding the configuration of the HMD 100 are omitted.

  In the eighth embodiment, when an operation for contacting the track pad 14 at a plurality of positions is performed, the HMD 100 executes a process according to the contact operation at the plurality of positions.

The operations in steps S11 to S19 shown in FIG. 19 are the same as those in the first to third embodiments described above.
In the eighth embodiment, the control unit 140 acquires the operation position of the track pad 14 detected by the input information acquisition unit 110 (step S12), and determines whether there are a plurality of acquired operation positions (step S12). Step S111). When the operation position is one place (step S111; No), the control unit 140 proceeds to step S13.

  When there are a plurality of operation positions (step S111; Yes), the control unit 140 determines whether or not the operation position is moving (step S112). While the operation position does not move (Step S112; No), the control unit 140 stands by until it moves. During this standby, the control unit 140 continues to detect the operation position based on the operation data input from the input information acquisition unit 110. In step S112, the control unit 140 determines that the operation position has moved when at least one of the plurality of operation positions has moved.

  When it determines with the operation position having moved (step S112; Yes), the control part 140 determines whether the operation position stopped (step S113). While the operation position does not stop (step S113; No), it waits until it stops. During this standby, the control unit 140 continues to detect the operation position based on the operation data input from the input information acquisition unit 110.

  When the operation position is stopped (step S113; Yes), the control unit 140 calculates a movement parameter of the input data according to the number of operation positions and the movement of the operation position (step S114). The input data movement parameters include, for example, data indicating the start position of the input position movement, the movement direction of the input position, and the movement speed of the input position.

  The control unit 140 starts generating input data according to the movement parameter set in step S16, and starts a process of displaying the pointer P on the right display driving unit 22 and the left display driving unit 24 based on the generated input data (step). S115). As the input position of the input data generated by the control unit 140 moves, the display position of the pointer P displayed by the image display unit 20 moves. Thereafter, the control unit 140 continues the process of generating the input data at a predetermined cycle (for example, the sampling cycle of the input information acquisition unit 110). Further, the control unit 140 continues the process of moving the display position of the pointer P based on the generated input data.

  While the display position of the pointer P moves, the control unit 140 monitors the release of contact with the track pad 14 (step S116). In step S116, the control unit 140 determines whether or not all contact with the track pad 14 has been released. When it is determined that the contact with the track pad 14 is not released (step S116; No), the control unit 140 continues to monitor the contact position in step S116, and generates input data and moves the display position of the pointer P. Continue.

  When the control unit 140 detects that all the contacts with the track pad 14 have been released (step S116; Yes), the control unit 140 stops generating the input data, thereby stopping the movement of the pointer P (step S19). This process is terminated.

  In step S114, the control unit 140 may generate a movement parameter based on the direction, speed, and movement amount of each of the plurality of operation positions. In addition, the control unit 140 may obtain a change in the relative position of the plurality of operation positions, and generate a movement parameter based on the direction of change in the relative position, the speed of change, and the amount of change.

A specific operation example of the control unit 140 in step S114 is given below.
(1) The control unit 140 sets a non-moving side as a detection target when some of the plurality of operating positions move. In this case, when the operation position set as the detection target moves, the control unit 140 sets the movement parameter corresponding to the movement direction, the movement speed, the movement amount, and the like of the operation position to be detected as the process of step S16, for example. Generate in the same way. When executing the operation (1), the control unit 140 sets one operation position as the operation position to be detected. Therefore, when the input information acquisition unit 110 detects three or more operation positions, the process waits in step S112 until other operation positions excluding any one operation position move. Further, after setting the operation position to be detected, the control unit 140 stands by in step S112 until the operation position to be detected moves.
On the contrary, the control unit 140 may set the moved side as a detection target when some of the operation positions move. In this example, when the input information acquisition unit 110 detects three or more operation positions, the control unit 140 stands by until any one operation position moves in step S112. Further, after setting the operation position to be detected, the control unit 140 stands by in step S112 until the operation position to be detected moves.

  (2) The control unit 140 is an input data for enlarging / reducing an image displayed by the image display unit 20 when a plurality of operation positions are moved and the movement directions of the plurality of operation positions are parallel to each other. Generate a movement parameter to generate For example, this corresponds to a case where the user moves a plurality of fingers on the track pad 14 together. In this case, the control unit 140 may add data indicating input data for instructing to enlarge or reduce the display image as an attribute of the movement parameter. Further, the input data generated by the control unit 140 may be data for instructing enlargement / reduction of the display image, data for designating the display magnification or instructing increase / decrease of the display magnification. Also good. The display control unit 170 updates the display magnification of the display image (for example, the pointer P) displayed by the right display driving unit 22 and the left display driving unit 24 according to the generated input data.

(3) When the plurality of operation positions are moved and the movement directions of the plurality of operation positions are not parallel, the control unit 140 performs the Z-axis direction of the virtual input plane IP in which the XY coordinates are set ( A process of generating input data for a direction perpendicular to the virtual input plane IP) may be performed.
In this case, the control device 10 including the operating system 150 is configured to accept a three-dimensional position input. The control device 10 can detect a two-dimensional position input in the virtual input plane IP and a position input having, for example, a direction perpendicular to the virtual input plane IP as a coordinate axis with reference to the virtual input plane IP. That is, the process can be executed by receiving an XY coordinate in the plane of the virtual input plane IP and a position input in the Z-axis direction perpendicular to the virtual input plane IP. That is, the control device 10 has a virtual input space including the virtual input plane IP, and can process input data that determines the position and direction in the virtual input space with reference to the virtual input plane IP.

In FIG. 20, the operation body contacts the operation position A31 and the operation position A33 on the track pad 14, and the operation AT31 moves from the operation position A31 to the operation position A32. The operation AT32 moves from the operation position A33 to the operation position A34. An example in which is executed. By the operations AT31 and AT32, the two operation positions on the track pad 14 move so as to approach each other.
The control unit 140 generates a movement parameter for generating input data in which the input position moves in the Z-axis direction on the virtual input plane IP in response to the two operation positions approaching by the operations AT31 and AT32. In the example of FIG. 20, input data is generated in which the input position moves in the positive direction of the Z axis with respect to the virtual input plane IP. Further, when the control unit 140 detects an operation in which the distance between the two operation positions is increased on the track pad 14, the control unit 140 generates input data in which the input position moves in the negative direction of the Z axis with respect to the virtual input plane IP. May be. In this case, the base point of the input position that moves in the Z-axis direction may be the input position IA11 corresponding to the center or the center of gravity of the plurality of operation positions.

  The example of (3) can be combined with the case where a plurality of operation positions move as described in (2). In this case, when the operation position moves in parallel, input data for enlarging / reducing the display image is generated, and when the distance between a plurality of operation positions changes, the input position moves in the Z-axis direction on the virtual input plane IP. Movement parameters may be set to generate data. In addition, when a plurality of operation positions move in parallel, input data for moving the input position in the Z-axis direction on the virtual input plane IP is generated corresponding to the movement direction and movement amount of the plurality of operation positions. May be set. In addition, when a plurality of operation positions move horizontally, input data for enlarging / reducing the display image is generated, and when a plurality of operation positions move non-horizontally, that is, move so that the mutual distance changes, the Z axis The movement parameter may be set so as to generate input data in which the input position moves in the direction.

  (4) When three or more operation positions move at the same time, the control unit 140 corresponds to the movement of the three or more operation positions in the Z-axis direction of the virtual input plane IP in which the XY coordinates are set ( A process of generating input data for a direction perpendicular to the virtual input plane IP) may be performed. In this case, the control unit 140 executes arithmetic processing such as obtaining an average for at least one of the movement direction, the movement speed, and the movement amount of the plurality of operation positions, and the movement direction, movement speed, or movement amount of the operation position. The representative value of is obtained. The control unit 140 may generate a movement parameter based on the representative value. In this case, the control unit 140 can generate input data whose input position moves in the positive or negative direction of the Z axis with respect to the virtual input plane IP.

  The control unit 140 can execute at least one of the operations (1), (2), (3), and (4) described above, or an operation that combines two or more, in step S114. Accordingly, in response to various operations on the track pad 14, a complicated input operation can be performed by a contact operation on the track pad 14.

[Ninth Embodiment]
FIG. 21 is a flowchart showing the operation of the HMD 100 in the ninth embodiment. In the ninth embodiment, the configuration of the HMD 100 is the same as that of the first embodiment. For this reason, illustration and description regarding the configuration of the HMD 100 are omitted.

  In the ninth embodiment, the HMD 100 generates input data in response to a contact operation on the track pad 14 and an operation of moving the control device 10 or changing the posture of the control device 10.

The operations in steps S11 to S19 shown in FIG. 21 are the same as those in the first to third embodiments described above.
When it is detected that the operation position is stopped (step S15), the control unit 140 detects the tilt of the control device 10 and / or the movement of the control device 10 based on the detection value of the three-axis sensor 113 ( Step S121). After the input information acquisition unit 110 detects a touch operation in step S11, the control unit 140 acquires the detection value of the three-axis sensor 113, for example, at a set sampling period, and based on the accumulated value, the control device 10 You may specify the posture. Further, after detecting that the operation position has moved in step S13, the control unit 140 acquires the detection value of the three-axis sensor 113, for example, at a set sampling period, and based on the accumulated value, the control device 10 You may specify the posture. Moreover, when the control apparatus 10 has a geomagnetic sensor (not shown), the attitude of the control apparatus 10 may be detected based on the detection value of the geomagnetic sensor. Further, the control unit 140 may acquire the detection value of the three-axis sensor 113 in step S121 and detect the attitude of the control device 10 based on the acquired detection value.

  The control unit 140 calculates the movement parameter corresponding to the tilt of the control device 10 detected in step S121 and / or the movement of the control device 10 and the operation position detected in steps S12 to S15 (step S122). . In step S122, the control unit 140 may include data indicating the start position of the input position, the direction of movement of the input position, and the speed of movement of the input position. The movement parameter may include data indicating the movement amount. The movement parameter may be data for generating input data including movement of an input position in a direction perpendicular to the virtual input plane IP.

  In step S122, the control unit 140 may generate a movement parameter based on, for example, the locus of the operation position on the track pad 14, the movement speed, the movement direction, the movement amount, and the like. Furthermore, the control unit 140 may correct or correct the generated movement parameter based on the tilt (posture) of the control device 10 and / or the movement of the control device 10 detected in step S121. As a specific example, when it is detected in step S121 that the control device 10 is tilted in a preset direction, the control unit 140 updates data indicating the movement speed of the input position among the movement parameters. Also good. In this case, the control unit 140 may update the movement parameter so as to increase the movement speed of the input position when the control device 10 is tilted in a predetermined direction. Further, when the control device 10 is tilted in the direction opposite to the predetermined direction, the movement parameter may be updated so as to reduce the movement speed of the input position. Alternatively, the control unit 140 may update the movement parameter so as to increase the movement speed of the input position when the control device 10 moves in a predetermined direction. Further, when the control device 10 moves in a direction opposite to the predetermined direction, the movement parameter may be updated so as to reduce the movement speed of the input position.

  Further, the control unit 140 may change the relative position between the track pad 14 and the virtual input plane IP in accordance with the inclination of the control device 10. For example, in the initial state, as shown in FIG. 6, the track pad 14 and the virtual input plane IP are parallel, and the XY axis of the track pad 14 and the XY axis of the virtual input plane IP are parallel. Set as The control unit 140 may change the inclination of the virtual input plane IP with respect to the track pad 14 according to the inclination of the control device 10 detected in step S121.

  As described above, the HMD 100 includes the three-axis sensor 113 that detects the posture of the control device 10 having the track pad 14 or a change in the posture. While the display position of the pointer P moves, the control unit 140 generates input data corresponding to the posture of the control device 10 detected by the three-axis sensor 113 or a change in the posture. Thereby, the control unit 140 changes at least one of the moving direction and the moving speed of the display position of the pointer P. For this reason, the mode of movement of the display position of the pointer P, which is a display object, can be controlled with a high degree of freedom by a simple operation.

[Other variations]
Hereinafter, modified examples of the above-described embodiments will be described with reference to FIGS. 22 to 25.
FIG. 22 is a diagram showing a circular track pad 14 a as a modification of the track pad 14. The track pad 14 a functions as an operation surface that detects a contact operation, like the track pad 14. When the HMD 100 includes the track pad 14a, the input information acquisition unit 110 detects an operation on the track pad 14a as, for example, coordinates in the orthogonal coordinate system indicated by reference characters X and Y in FIG. The track pad 14a cannot move the operation position greatly in the X-axis direction and the Y-axis direction, but by applying the first to ninth embodiments, the movement amount of the operation position in the track pad 14 The pointer P can be moved. Similar to the track pad 14, the track pad 14 a only needs to have a size that allows the control unit 140 to detect the direction of movement of the operation position. Therefore, the track pad 14 a is different from the display area of the image display unit 20 as shown in FIG. It may be a shape.
Therefore, the pointer P can be moved greatly using various track pads such as the circular track pad 14a shown in FIG. 22 or an elongated shape, and high operability can be realized.

FIG. 23 shows a modification in which the track pad 14 b is arranged on the side surface of the image display unit 20 instead of the track pad 14 (FIG. 1) provided in the main body of the control device 10. Similar to the track pad 14, the track pad 14 b functions as an operation surface that detects a contact operation.
In the example of FIG. 23, the slide gesture can be executed by an operation in which the user rubs the side surface of the image display unit 20 with a finger. Also in this case, it is not easy to increase the area of the track pad 14b, but high operability can be realized by applying the first to ninth embodiments.

FIG. 24 shows an example in which a track pad 14 c capable of detecting a contact operation is arranged on the surface of a ring type device 71 in the same manner as the track pad 14. Similarly to the track pad 14, the track pad 14c functions as an operation surface that detects a contact operation.
The track pad 14 c may be formed on part or all of the surface of the device 71, and the appearance of the track pad 14 c may be integrated with the surface of the device 71. The track pad 14 c only needs to be able to detect contact with the surface of the device 71, and may have a configuration in which a circuit for detecting contact or the like is embedded or accommodated in the device 71. That is, the surface of the device 71 may not be configured integrally with a circuit for detecting an operation or the like, and may not configure a part of the circuit.
For example, the device 71 can perform a wireless communication with the communication unit 117 of the control device 10, and the control unit 140 can detect an operation position on the track pad 14c.
In this case, by applying the first to ninth embodiments, the pointer P can be freely moved by a contact operation on the small ring type device 71.

FIG. 25 shows an example in which a track pad 14 d capable of detecting a contact operation like the track pad 14 is arranged on the side surface of the pen-type device 72. Similarly to the track pad 14, the track pad 14 d functions as an operation surface that detects a contact operation. The track pad 14 d may be formed on part or all of the surface of the device 72, and the appearance of the track pad 14 c may be integrated with the surface of the device 72. The track pad 14 d only needs to be able to detect contact with the surface of the device 72, and may have a configuration in which a circuit or the like for detecting contact is embedded or accommodated in the device 72. That is, the surface of the device 72 may not be configured integrally with a circuit or the like for detecting an operation, and may not configure a part of the circuit.
For example, the device 72 can perform a wireless communication with the communication unit 117 of the control device 10, and the control unit 140 can detect an operation position on the track pad 14d. Further, the device 72 may have a configuration that can actually be used as a writing instrument. By applying the first to ninth embodiments to the configuration using the device 72, the pointer P can be freely moved by a contact operation on the small pen-type device 71.

  The operations on the track pads 14c and 14d are a touch (contact) operation with fingers and the like, and an operation to move while maintaining the contact of the touched fingers. When detecting contact with the track pads 14c and 14d, the control unit 140 determines that the operation is not completed while the contact is maintained, that is, until the contact is released. In other words, when the contact of the fingers with the track pads 14c and 14d is released, it is determined that the operation is finished. Therefore, when the user performs an operation of bringing the fingers into contact with the track pads 14c and 14d and moving the fingers away from the track pads 14c and 14d, the pointer P, which is a display object, and the screens M1 to M1 are displayed. The display position of M6 moves.

As another example of a configuration in which the surface of a three-dimensional object is used as an operation surface, for example, a configuration in which a surface of a handle of an automobile is used as an operation surface and a contact operation on the surface of the handle is detected can be employed.
In addition, the operation surface such as the track pads 14, 14a, 14b, 14c, and 14d described in the above embodiments may be divided into a plurality of regions, and each region may be used as the operation surface. In this case, the control unit 140 may set a movement parameter and generate input data corresponding to movement of one or a plurality of operation positions in each region. In addition, the control unit 140 may generate input data by setting a movement parameter corresponding to the movement of the operation position detected on the plurality of operation surfaces.

The present invention is not limited to the configurations of the above-described embodiments and modifications, and can be implemented in various modes without departing from the scope of the invention.
For example, in the above embodiment, instead of the image display unit 20, another type of image display unit such as an image display unit worn like a hat may be employed. What is necessary is just to provide the display part which displays an image corresponding to a user's left eye, and the display part which displays an image corresponding to a user's right eye. The display device of the present invention may be configured as a head mounted display mounted on a vehicle such as an automobile or an airplane. Moreover, for example, it may be configured as a head-mounted display built in a body protective device such as a helmet, or may be a head-up display (HUD) used for a windshield of an automobile.

  In each of the above embodiments and modifications, the pointer P or the screens M1 to M6 in the ring type UI shown in FIG. 8 is exemplified as the display object. The present invention is not limited to this, and the present invention can be applied as a display object as long as the display position can be moved according to an operation on the operation surface. For example, the present invention may be applied by using a display object displayed in content such as an application program or a game executed by the control device 10 as a display object.

  Furthermore, in the above embodiment, the image display unit 20 and the control device 10 are separated and connected by way of the connection unit 40, but the control device 10 and the image display unit 20 are integrated. It is also possible to be configured to be mounted on the user's head.

  For example, as a configuration for generating image light in the image display unit 20, a configuration including an organic EL (Organic Electro-Luminescence) display and an organic EL control unit may be used. Further, as a configuration for generating image light, LCOS (Liquid crystal on silicon, LCoS is a registered trademark), a digital micromirror device, or the like can be used. Further, for example, the present invention can be applied to a laser retinal projection type head mounted display. That is, the image generation unit includes a laser light source and an optical system that guides the laser light source to the user's eye, makes the laser light enter the user's eye, scans the retina, and forms an image on the retina. A configuration that allows the user to visually recognize an image may be employed. When a laser retina projection type head-mounted display is employed, the “image light emitting area in the image light generation unit” can be defined as an image area recognized by the user's eyes.

  As an optical system that guides image light to the user's eyes, an optical member that transmits external light that is incident from the outside toward the apparatus and that enters the user's eyes together with the image light can be employed. Moreover, you may use the optical member which is located ahead of a user's eyes and overlaps a part or all of a user's visual field. Further, a scanning optical system that scans a laser beam or the like to obtain image light may be employed. Further, the optical member is not limited to guiding the image light inside the optical member, and may have only a function of guiding the image light by refracting and / or reflecting it toward the user's eyes.

  Further, the present invention can be applied to a display device that employs a scanning optical system using a MEMS mirror and uses MEMS display technology. That is, as an image display element, a signal light forming unit, a scanning optical system having a MEMS mirror that scans light emitted from the signal light forming unit, and an optical member on which a virtual image is formed by light scanned by the scanning optical system May be provided. In this configuration, the light emitted from the signal light forming unit is reflected by the MEMS mirror, enters the optical member, is guided through the optical member, and reaches the virtual image forming surface. When the MEMS mirror scans the light, a virtual image is formed on the virtual image forming surface, and the user recognizes the virtual image with the eyes, and the image is recognized. The optical component in this case may be one that guides light through a plurality of reflections, such as the right light guide plate 261 and the left light guide plate 262 of the above embodiment, and may use a half mirror surface. .

The display device of the present invention is not limited to a head-mounted display device, and can be applied to various display devices such as a flat panel display and a projector. The display device of the present invention may be any device that allows an image to be visually recognized by external light and image light. For example, a configuration in which an image by image light is visually recognized by an optical member that transmits external light is exemplified. Specifically, the above-described head-mounted display includes an optical member that transmits external light, and a light-transmitting plane or curved surface that is fixedly or movably installed at a position away from the user ( The present invention can also be applied to a display device that projects image light onto glass, transparent plastic, or the like. As an example, a configuration of a display device that projects image light onto a window glass of a vehicle and allows a user who is on board or a user outside the vehicle to visually recognize the scenery inside and outside the vehicle together with an image by the image light. It is done. In addition, for example, image light is projected onto a transparent, semi-transparent, or colored transparent display surface that is fixedly installed such as a window glass of a building, and is displayed together with an image by the image light to a user around the display surface. A configuration of a display device that visually recognizes a scene through the surface can be given.
The present invention is not limited to a display device that visually recognizes an image together with external light, but can be applied to various display devices. For example, the present invention can also be applied to a display device that displays an image in a state where an outside scene cannot be visually recognized. Specifically, the present invention is applied to a display device that displays a captured image of the camera 61, an image or CG generated based on the captured image, video based on video data stored in advance or video data input from the outside, and the like. Can be applied. This type of display device can include a so-called closed display device in which an outside scene cannot be visually recognized. Of course, a display device that displays video data or an analog video signal input from the outside without performing processing such as AR display, MR display, or VR display is also included as an application target of the present invention.

In addition, at least a part of each functional block shown in FIG. 2 may be realized by hardware, or may be realized by cooperation of hardware and software, as shown in FIG. It is not limited to a configuration in which independent hardware resources are arranged. The program executed by the control unit 140 may be stored in the storage unit 120 or the storage device in the control device 10, or the program stored in an external device is acquired via the communication unit 117 or the interface 125. It is good also as a structure to execute.
In addition, the present invention is realized in the form of a storage medium storing a program executed by the control device 10, a server device that distributes the program, a transmission medium that transmits the program, a data signal that implements the program in a carrier wave, and the like. You can also The storage unit 120 as a storage medium may be any one of a magnetic or optical storage medium or a semiconductor memory device, or other types of storage media. This storage medium may be a portable storage medium such as a memory card, and other specific mounting forms are also arbitrary.

  The program can be implemented as a single application program that runs on the operating system 150 of the control device 10. Further, the program may be implemented as a function of a device driver that operates as a part of the operating system 150, the operating system 150, or in cooperation with the operating system 150. Or you may implement as a function of the application program performed with these. For example, you may implement as a function of the device driver program which controls trackpad 14, 14a, 14b, 14c, 14d. In addition, the operating system 150 may be implemented as a function of a program module that accepts the operation of the track pads 14, 14a, 14b, 14c, and 14d. Moreover, the structure which implement | achieves the said program with a some application program may be sufficient, and the form of a specific program is arbitrary.

Further, the function of the program executed by the control unit 140, that is, each processing unit (for example, the image processing unit 160, the display control unit 170, the operation detection unit 183, the GUI control unit 185, the audio processing unit 190, or the like provided in the control unit 140, or Other generation units, determination units, identification units, etc.) are configured using ASIC (Application Specific Integrated Circuit) or SoC (System on a Chip) designed to realize the function. May be. Moreover, you may implement | achieve by programmable devices, such as FPGA (Field-Programmable Gate Array).
Moreover, only the operation part 111 among the structures formed in the control apparatus 10 may be formed as a single user interface (UI). Further, the configuration formed in the control device 10 may be formed in the image display unit 20 in an overlapping manner. For example, the control unit 140 illustrated in FIG. 2 may be formed in both the control device 10 and the image display unit 20, or the control unit 140 formed in the control device 10 and the CPU formed in the image display unit 20. The functions performed by and may be configured separately.

  DESCRIPTION OF SYMBOLS 10 ... Control apparatus (operation device) 14, 14a, 14b, 14c, 14d ... Trackpad (operation surface), 20 ... Image display part (display part), 22 ... Right display drive part, 24 ... Left display drive part, DESCRIPTION OF SYMBOLS 61 ... Camera, 100 ... HMD (display apparatus), 110 ... Input information acquisition part (operation detection part), 111 ... Operation part, 113 ... Three axis sensor (attitude detection part), 120 ... Memory | storage part, 121 ... Setting data, 123 ... Content data 140 ... Control unit 160 ... Image processing unit 170 ... Display control unit 182 ... Operation detection unit 185 ... GUI control unit 190 ... Audio processing unit 261 ... Right light guide plate 262 ... Left guide Light plate, P ... pointer (display object).

Claims (20)

A display unit;
An operation detection unit for detecting an operation on the operation surface;
A control unit that displays a display object to be operated on the display unit and controls a display position of the display object in response to an operation on the operation surface,
The control unit moves the display position of the display object in response to a change in the operation position on the operation surface detected by the operation detection unit, and the operation detection unit operates in accordance with a preset end condition. Stopping the movement of the display position of the display object when detecting
A display device. The control unit sets the movement direction of the display position of the display object to a direction corresponding to the movement direction of the operation position on the operation surface;
The display device according to claim 1. The control unit is configured to display the display at a display position corresponding to the operation position on the operation surface while performing a process of obtaining a moving direction of the display position of the display object after the operation detection unit detects an operation on the operation surface. Displaying objects,
The display device according to claim 2. The control unit sets the movement speed of the display position of the display object to a speed corresponding to the movement amount or movement speed of the operation position on the operation surface;
The display device according to claim 1, wherein: When the operation position on the operation surface detected by the operation detection unit is changed while the display position of the display object is moved, the control unit is at least one of a moving direction and a moving speed of the display position of the display object. Changing things,
The display device according to claim 1, wherein: The control unit detects an operation position when the operation detection unit detects contact of the operation body with the operation surface, and the operation detection unit is released from the operation surface when the operation body is separated from the operation surface. It is determined that the termination condition is met,
The display device according to claim 1, wherein: The control unit detects an operation position when the operation detection unit detects contact of the operation body with the operation surface, and the operation detection unit is released from the operation surface when the operation body is separated from the operation surface. It is determined that the end condition is satisfied when a predetermined time has elapsed since the detection of
The display device according to claim 1, wherein: When the operation detection unit detects contact with the operation surface within a predetermined time after the operation detection unit detects that the operation body has moved away from the operation surface and contact has been released. And changing at least one of the moving direction and the moving speed of the display position of the display object corresponding to the operation position of the new contact detected by the operation detection unit,
The display device according to claim 1, wherein: An operation device having the operation surface;
A posture detection unit for detecting a change in posture or posture of the operation device;
The control unit moves and moves the display position of the display object in response to a change in posture or posture of the operation device detected by the posture detection unit while the display position of the display object moves. Changing at least one of the
The display device according to claim 1, wherein: The area or size that the operation detection unit detects an operation on the operation surface is smaller than the area or size of a display area on which the display unit displays an image,
The display device according to claim 1, wherein: The shape of the operation surface is different from the display area where the display unit displays an image,
The display device according to claim 1, wherein: A display unit;
An operation detection unit for detecting an operation on the operation surface;
A control unit that generates input data for the virtual input plane according to the operation detected by the operation detection unit,
The control unit starts generation of input data in the virtual input plane in response to the operation when the operation detection unit detects an operation in which an operation position moves on the operation surface, and the operation detection unit Stopping the generation of the input data when an operation corresponding to a preset end condition is detected;
A display device. The input data generated by the control unit includes at least one of an input position in the virtual input plane, a moving direction of the input position in the virtual input plane, and a moving speed of the input position in the virtual input plane. ,
The display device according to claim 12. The control unit detects at least one of a moving direction of the input position in the virtual input plane and a moving speed of the input position in the virtual input plane included in the input data. Set according to the moving amount or moving speed of the operation position,
The display device according to claim 13. The operation detection unit detects a contact position as an operation position according to an operation in which an operation body contacts the operation surface,
The control unit determines that the termination condition is satisfied when the operation detection unit detects that the contact with the operation surface is released, and stops generating the input data;
The display device according to claim 12, wherein: The control unit determines that the end condition is satisfied and stops generating the input data when a predetermined time has elapsed since the operation detection unit detected release of contact with the operation surface.
The display device according to claim 12, wherein: The control unit can generate three-dimensional input data including a direction perpendicular to the virtual input plane and the virtual input plane according to an operation detected by the operation detection unit, and the operation detection unit Generating the input data including a component in a direction perpendicular to the virtual input plane when detecting operations at a plurality of operation positions on the operation surface;
The display device according to claim 12, wherein: An operation device having the operation surface;
A posture detection unit for detecting a change in posture or posture of the operation device;
The control unit changes the correspondence between the operation surface and the virtual input plane in response to a change in posture or posture of the operation device detected by the posture detection unit,
The display device according to claim 12, wherein: The virtual input plane is associated with the display area in the display unit,
The control unit displays a display object to be operated on the display unit, and controls a display position of the display object corresponding to the input data;
The display device according to claim 12, wherein: Controlling a display device having a display unit;
Detects operations on the operation surface,
Displaying a display object to be operated on the display unit, and controlling a display position of the display object in response to an operation on the operation surface;
The display position of the display object is changed when the display position of the display object is moved in response to a change in the operation position on the operation surface and an operation corresponding to a preset end condition on the operation surface is detected. Stopping the movement,
A control method of a display device characterized by the above.

Images